Data processing method and device performing same

Abstract

Disclosed is a data processing method. According to an embodiment, the method for processing path data generated on the basis of a driving log may comprise the steps of: detecting turning data from the path data; upsampling the turning data; constructing the upsampled turning data as a sub-dataset to process the path data; and generating a driving path of a vehicle on the basis of the processed path data.

Description

Data processing method and device for performing the same

[0001] The following disclosure relates to a data processing method and an apparatus for performing the same.

[0002] An end-to-end model is a type of machine learning model that derives output data by performing operations directly on input data without intermediate processing steps or separate components. Based on information obtained from sensors of an autonomous driving vehicle (ADV), an end-to-end model can generate a path for the vehicle to drive (e.g., a target trajectory).

[0003] The background technology described above is possessed or acquired by the inventor in the process of deriving the content of the disclosure of the present application, and cannot necessarily be considered as prior art disclosed to the general public prior to the filing of this application.

[0004] One embodiment may provide a technique for processing data for training a model.

[0005] One embodiment can provide a network structure capable of deriving a driving path by fusing various data.

[0006] However, technical challenges are not limited to the technical challenges described above, and other technical challenges may exist.

[0007] A method for processing path data generated based on a driving log according to one embodiment may include, from the path data, an operation of detecting turning data, an operation of upsampling the turning data, an operation of processing the path data by constructing the upsampled turning data into a subset, and an operation of generating a driving path of a vehicle based on the processed path data.

[0008] The above path data may include a plurality of vectors partitioned into predetermined time units. The detection operation may include an operation of extracting a pair of vectors from the plurality of vectors and, based on the pair of vectors, an operation of determining whether a path section corresponding to the pair of vectors is a turning section.

[0009] The operation of determining whether the above is direction change data may include the operation of performing a vector product operation on the vector pair, the operation of determining whether the path segment corresponding to the vector pair is a direction change segment based on the result of the vector product operation, and the operation of determining the part of the path data corresponding to the vector pair as the direction change data in response to the fact that the path segment is a direction change segment.

[0010] The operation of generating the driving path may include the operation of obtaining the driving path by inputting the sensing data obtained from the vehicle, the processed path data, and the driving trajectory of the vehicle into a model.

[0011] The above-mentioned acquisition operation may include the operation of inputting the sensing data, the processed path data, and the driving trajectory to a corresponding first mixer respectively, and the operation of inputting the output of the first mixer to a second mixer to acquire the driving path.

[0012] The above route data may be generated based on navigation information generated based on the driving log, the driving status of the vehicle, and collected navigation history.

[0013] The above driving record may be a record of the vehicle's driving recorded chronologically on an SD (standard definition) map.

[0014] The above navigation information may be generated by inputting the vehicle's starting point and destination, extracted from the driving record, into the navigation route search algorithm.

[0015] The above model may be an end-to-end model included in a vehicle's autonomous driving system.

[0016] According to one embodiment, a computer-readable recording medium storing one or more computer programs may include instructions for performing the method in a processor.

[0017] A device for processing path data generated based on a driving record according to one embodiment may include at least one processor and a memory for storing instructions. Based on the instructions being executed individually or collectively by the at least one processor, the device may detect turning data from the path data, upsample the turning data, construct the upsampled turning data into a sub-dataset to process the path data, and generate a driving path of a vehicle based on the processed path data.

[0018] The path data may include a plurality of vectors partitioned into predetermined time units. Based on the instructions being executed individually or collectively by the at least one processor, the device may be able to extract vector pairs from the plurality of vectors and, based on the vector pairs, determine whether a path section corresponding to the vector pairs is a turning section.

[0019] Based on the above instructions being executed individually or collectively by at least one processor, the device may be configured to perform a vector product operation on the vector pair, determine whether the path segment corresponding to the vector pair is a direction change segment based on the result of the vector product operation, and, in response to the path segment being a direction change segment, determine the portion of the path data corresponding to the vector pair as the direction change data.

[0020] Based on the above instructions being executed individually or collectively by at least one processor, the sensing data obtained from the vehicle, the processed path data, and the driving trajectory of the vehicle can be input into a model to obtain the driving path.

[0021] Based on the above instructions being executed individually or collectively by at least one processor, the device may input the sensing data, the processed path data, and the driving trajectory respectively to a corresponding first mixer, and input the output of the first mixer to a second mixer to obtain the driving path.

[0022] The above route data may be generated based on navigation information generated based on the driving log and collected navigation history.

[0023] The above driving record may be a record of the vehicle's driving recorded chronologically on an SD (standard definition) map.

[0024] The above navigation information may be generated by inputting the vehicle's starting point and destination, extracted from the driving record, into the navigation route search algorithm.

[0025] The above model may be an end-to-end model included in a vehicle's autonomous driving system.

[0026] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0027] FIG. 1 is a drawing for explaining an autonomous driving system according to one embodiment.

[0028] FIG. 2 is a diagram illustrating an end-to-end based path generation framework according to one embodiment.

[0029] FIG. 3a is a diagram illustrating a dataset generation and verification process according to one embodiment.

[0030] FIG. 3b is a diagram illustrating a learning process according to one embodiment.

[0031] FIG. 3c is a diagram illustrating an inference process according to one embodiment.

[0032] FIG. 4 is a diagram illustrating a dataset generation device according to one embodiment.

[0033] FIG. 5 is a diagram illustrating a data processing process according to one embodiment.

[0034] FIG. 6 is a diagram illustrating the network structure of a model according to one embodiment.

[0035] FIG. 7 illustrates an example of a dataset according to one embodiment.

[0036] FIGS. 8A and FIGS. 8B are drawings for illustrating an example of a dataset according to one embodiment.

[0037] FIGS. 9a to 9c are drawings for illustrating a predicted path according to one embodiment.

[0038] FIG. 10 is a diagram illustrating a dataset verification process according to one embodiment.

[0039] FIGS. 11a and FIGS. 11b are drawings for explaining verification results according to one embodiment.

[0040] FIG. 12 is a flowchart illustrating a method according to one embodiment.

[0041] FIG. 13 is a schematic block diagram of an electronic device according to one embodiment.

[0042] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0043] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0044] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0045] Singular expressions include plural expressions unless the context clearly indicates otherwise. In this document, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may each include any one of the items listed together with the corresponding phrase, or all possible combinations thereof. In this specification, terms such as “comprising” or “having” are intended to designate the existence of the described feature, number, step, action, component, part, or combination thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0046] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0047] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0048] As used in this document, the term "part" refers to software or hardware components, such as FPGAs or ASICs, and the "part" performs certain roles. However, the meaning of "part" is not limited to software or hardware. The "part" may be configured to reside in an addressable storage medium or configured to operate one or more processors. For example, the "part" may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." Furthermore, the components and "parts" may be implemented to operate one or more CPUs within a device or secure multimedia card. Additionally, '~part' may include one or more processors.

[0049] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0050]

[0051] FIG. 1 is a drawing for explaining an autonomous driving system according to one embodiment.

[0052] Referring to FIG. 1, according to one embodiment, an autonomous driving system (e.g., autonomous driving system (10)) may be a system that generates a driving path and controls the driving of a vehicle (e.g., vehicle (110)) based on sensing data about the surroundings of a vehicle (e.g., vehicle (110)). The autonomous driving system (10) may include an autonomous driving device (100), a vehicle (110), and a server (130). The autonomous driving system (10) may generate a driving path of the vehicle (110) based on an end-to-end model. The end-to-end model may be a model that performs motion planning by directly processing input data without complex intermediate steps. The end-to-end model may be included in the autonomous driving device (100) and / or the server (130) to generate a driving path of the vehicle (110). The autonomous driving device (100) is mounted inside the vehicle (110) and can control the driving of the vehicle (110). For example, the autonomous driving device (100) may be a software module implemented on a processor (not shown) of the vehicle (110). Some components of the autonomous driving device (100) may be located outside the vehicle (110) (e.g., a server (130)). For example, a module that generates a driving path of the autonomous driving device (100) may be implemented on the server (130) to generate a driving path and transmit the generated driving path to the vehicle (110). The vehicle (110) means a vehicle that carries people and / or goods and may include a vehicle such as an automobile. The vehicle (110) may be an autonomous vehicle.

[0053] The autonomous driving device (100) can obtain information about the surroundings of the vehicle (110) (e.g., scene images) and driving information of the vehicle (110) (e.g., driving records) from a plurality of sensors mounted on the vehicle (110). For example, the autonomous driving device (100) can obtain information such as the surroundings of the vehicle (110) and driving information of the vehicle (110) from various sensors such as an image sensor, a LiDAR sensor, a radar sensor, an event sensor, an illuminance sensor, a GPS device, and an acceleration sensor. The autonomous driving device (100) can generate a driving path of the vehicle (110) based on information about the surroundings of the vehicle (110). For example, the autonomous driving device (100) can generate a driving path of the vehicle (110) based on information such as sensing data about the surroundings of the vehicle (110), the destination of the vehicle (110), and the location of the vehicle (110).

[0054] The autonomous driving device (100) may include a model (e.g., an end-to-end model). The model may be trained using a dataset generated based on the driving records of the vehicle (110). The training of the model may be performed internally within the autonomous driving device (100), or the autonomous driving device (100) may use a model trained in a separate device. The autonomous driving device (100) may generate a driving path for the vehicle (110) using the trained model.

[0055] The autonomous driving device (100), vehicle (110), and server (130) can communicate using a network (not shown). For example, the network may include a Local Area Network (LAN), a Wide Area Network (WAN), a Value Added Network (VAN), a mobile radio communication network, a satellite communication network, and combinations thereof. The network is a comprehensive data communication network that enables the autonomous driving device (100) and the server (130) to communicate smoothly with each other, and may include wired internet, wireless internet, and mobile wireless communication networks. Additionally, the wireless communication network may include, for example, Wi-Fi, Bluetooth, Bluetooth Low Energy, Zigbee, Wi-Fi Direct (WFD), Ultra-Wideband (UWB), Infrared Data Association (IrDA), Near Field Communication (NFC), but is not limited thereto.

[0056]

[0057] FIG. 2 is a diagram illustrating an end-to-end based path generation framework according to one embodiment.

[0058] Referring to FIG. 2, according to one embodiment, an end-to-end based path generation framework may include a dataset generation and verification process (210), a training process (220), and an inference process (230). An autonomous driving system (e.g., the autonomous driving system (10) of FIG. 1) can generate a driving path for a vehicle (e.g., the vehicle (110) of FIG. 1) through the end-to-end based path generation framework.

[0059] The dataset generation and verification process (210) may be a process for generating a dataset to train a model (e.g., an end-to-end model) and verifying the generated dataset. The generation of the dataset may be performed by a dataset generation device (e.g., the dataset generation device (300) of FIG. 3a). The operation of the dataset generation device (300) generating the dataset will be described in detail later with reference to FIG. 4. The verification of the dataset may be performed by a dataset verification device (e.g., the dataset verification device (330) of FIG. 3a). The verification of the dataset performed by the dataset verification device (330) will be described in detail later with reference to FIG. 10 to FIG. 11b.

[0060] The data generation device (300) and the dataset verification device (330) may be a learning device (not shown) or a device separate from the learning device (not shown). For example, the dataset generation and verification process (210) and the learning process (220) may be performed on different devices. The learning device (not shown) may receive the dataset generated through the dataset generation and verification process (210) and perform the learning process (220).

[0061] The learning process (220) may be a process of training a model (e.g., an end-to-end model) to generate an optimal driving path. The learning process (220) may be performed in a learning device (not shown). The learning device (not shown) may be an autonomous driving device (100) or a device separate from the autonomous driving device (100). For example, the training of the model may be performed in a separate learning device (not shown), and the autonomous driving device (100) may generate a driving path using the trained model.

[0062] The inference process (230) may be a process for generating a driving path of a vehicle (e.g., vehicle (110)) using a learned model. In the inference process (230), an autonomous driving device (e.g., the autonomous driving device (100) of FIG. 1) may generate a driving path based on data input from a plurality of sensors of the vehicle (110). The inference process (230) will be described in detail later with reference to FIG. 6.

[0063]

[0064] FIG. 3a is a diagram illustrating a dataset generation and verification process according to one embodiment.

[0065] Referring to FIG. 3a, according to one embodiment, a dataset generating device (300) can generate a dataset for training a model (e.g., an end-to-end model) that generates a driving path of a vehicle (e.g., the vehicle (110) of FIG. 1). The model may be an end-to-end model included in an autonomous driving system (e.g., the autonomous driving system (10) of FIG. 1). The dataset generating device (300) can transmit the generated dataset to a dataset verification device (330). The dataset verification device (330) can verify whether the dataset was properly generated. For example, the dataset verification device (330) can verify whether navigation information generated retrospectively based on driving records was properly generated.

[0066] The dataset generation device (300) and the dataset verification device (330) may be implemented as the same device or as different devices. For example, the dataset generation device (300) and the dataset verification device (330) may be implemented as software modules running on a single device (e.g., the autonomous driving device (100) and server (130) of FIG. 1) or as separate devices.

[0067]

[0068] FIG. 3b is a diagram illustrating a learning process according to one embodiment.

[0069] Referring to FIG. 3b, according to one embodiment, a learning process (e.g., the learning process (220) of FIG. 2) may be performed by a learning device (not shown). The learning process (220) may be a process of training a model so that a desired output (e.g., a driving path) can be generated based on input data (e.g., sensing data). The model (350) may be an end-to-end model. The learning device (not shown) may generate a learned model (370) by training the model (350) using a dataset generated by a dataset generating device (e.g., the dataset generating device (300) of FIG. 3a). For example, the learning device (200) may generate a learned model (370) by calculating the difference between the predicted value and the actual value (ground truth) using a loss function and adjusting the weights of the model (350) with an optimization algorithm based on the loss value.

[0070]

[0071] FIG. 3c is a diagram illustrating an inference process according to one embodiment.

[0072] Referring to FIG. 3c, according to one embodiment, an inference process (e.g., the inference process (230) of FIG. 2) may be a process of inputting input data (e.g., sensing data) into a trained model (370) to generate a driving path in real time. The trained model (370) can generate a driving path in real time using the sensing data in an actual driving situation of a vehicle (e.g., the vehicle (110) of FIG. 1).

[0073]

[0074] FIG. 4 is a diagram illustrating a dataset generation device according to one embodiment.

[0075] Referring to FIG. 4, according to one embodiment, a dataset generating device (400) (e.g., the dataset generating device (300) of FIG. 3a) can generate a dataset for training of a model (e.g., an end-to-end model). The dataset generating device (400) can collect a navigation history (430). The navigation history (430) may include information about a starting point, a destination, and a driving route from the starting point to the destination. The navigation history (430) may be a record (e.g., a trajectory) of a vehicle driving from a starting point (e.g., the vehicle's starting location) to a destination using a navigation device. For example, the navigation history (430) may be a driving log containing the driving route from the vehicle's starting point to the destination. The navigation usage record (430) may be collected from at least one of a navigation device and a navigation server (e.g., the server (130) of FIG. 1) installed in a vehicle (e.g., vehicle (110)). For example, information regarding the starting point, destination, and driving route from the starting point to the destination included in the navigation usage record (430) may be collected from at least one of a navigation device and a navigation server installed in the vehicle (110).

[0076] The driving record (410) may be a driving record that does not include navigation information. For example, the driving record (410) may include the driving trajectory of the vehicle (110) driven without using a navigation device. The driving record (410) may be a record of the vehicle's driving in a time-series format on a standard definition (SD) map. An SD map may be a general map containing static geographical information. Compared to a high-definition (HD) map, an SD map contains less information and can use fewer resources. A dataset generating device (400) may generate navigation information based on the driving record (410). A navigation information generating module (420) may be a module that generates navigation information retrospectively from a record of the vehicle driving without using a navigation device (e.g., driving record (410)).

[0077] The navigation information generation module (420) can extract the vehicle's starting point and destination from the driving record (410). For example, the navigation information generation module (420) can extract the point where the vehicle started driving as the starting point and the point where the vehicle ended driving as the destination from the driving record (410). The navigation information generation module (420) can input the extracted starting point and destination into a navigation path search algorithm to generate navigation information including a path to the destination. For example, the navigation information generation module (420) can input the extracted vehicle's starting point and destination into a path search algorithm to generate navigation information retrospectively.

[0078] The dataset generation device (400) can generate a dataset based on navigation information generated retrospectively from navigation usage records (430) and driving records (410). The dataset generation device (400) can generate a dataset by using navigation information and SD map information together. The dataset generation device (400) can encode navigation information generated retrospectively from navigation usage records (430) and driving records (410) into various types of datasets using an encoder (470). The dataset generation device (400) can obtain route data based on navigation usage records (430) and navigation information. The encoder (470) can encode route data based on navigation usage records (430) and navigation information into SD map-based geometry information.

[0079] The dataset generation device (400) can extract turn-by-turn (TBT) information from the navigation usage record (430) and navigation information. The dataset generation device (400) can generate a dataset by encoding route data based on the navigation usage record (430) and navigation information together with the TBT information. The dataset generated by the dataset generation device (400) will be described in detail later with reference to FIGS. 7 to 8b.

[0080]

[0081] FIG. 5 is a diagram illustrating a data processing process according to one embodiment.

[0082] Referring to FIG. 5, according to one embodiment, a dataset generated by a dataset generating device (e.g., the dataset generating device (300) of FIG. 3) may include path data encoded with geometric information based on an SD map. For example, the path data may be at least part of a dataset generated based on navigation information based on a global SD map. The path data may include a plurality of vectors (e.g., a plurality of segments) partitioned into predetermined time units. If the time unit of the path data is too long, the path data may become complex, and if it is too short, sufficient context information may not be provided, so it may be necessary to set an appropriate interval. For example, it should be noted that the optimal time unit of the path data that includes sufficient context information without making the data complex may be 12 seconds, but is not limited thereto.

[0083] Since driving situations consist mostly of straight sections with relatively few turn sections, path data may primarily contain samples related to straight-line travel. Consequently, data imbalance may affect model training. It may be necessary to process the path data before using the dataset for model training.

[0084] A processing unit (not shown) can detect turning data from path data. A processing unit (not shown) can extract vector pairs from multiple vectors constituting the path data. For example, a processing unit (not shown) can iteratively extract vector pairs from multiple vectors. Based on the vector pairs, a processing unit (not shown) can determine whether a path section corresponding to a vector pair is a turning section. A processing unit (not shown) can perform a vector product operation on the vector pairs. For example, a processing unit (not shown) can perform a dot product or a cross product on vector (510) and vector (520). Based on the result of the vector product operation, a processing unit (not shown) can determine whether a path section corresponding to a vector pair is a turning section. For example, if the value of the cross product between vector (510) and vector (520) is equal to or greater than 0 degrees, and the arc cosine value between vector (510) and vector (520) is greater than 80 degrees and less than 100 degrees, the path section corresponding to vector (510) and vector (520) may be determined to be a left turn section. For example, if the value of the cross product between vector (510) and vector (520) is less than 0 degrees, and the arc cosine value between vector (510) and vector (520) is greater than 80 degrees and less than 100 degrees, the path section corresponding to vector (510) and vector (520) may be determined to be a right turn section. A processing device (not shown) can determine the part of the path data corresponding to the vector pair as direction change data in response to the determination that the path section corresponding to the vector pair is a direction change section (e.g., left turn section, right turn section).

[0085] A processing unit (not shown) can upsample direction change data to construct a subset. For example, the processing unit (not shown) can construct a balanced dataset by forming a mini-batch using the detected direction change data. The processing unit (not shown) can upsample the direction change data. The processing unit (not shown) can process path data by constructing the upsampled direction change data into a subset (e.g., a mini-batch). The processed path data can be used to generate a driving path for a vehicle. The process of generating a driving path will be explained in detail later with reference to FIG. 6. The path data processing unit (not shown) may be a learning unit (not shown) or a separate unit.

[0086]

[0087] FIG. 6 is a diagram illustrating the network structure of a model according to one embodiment.

[0088] Referring to FIG. 6, according to one embodiment, FIG. 6 may illustrate the network structure of a model. The model may be the learned model (370) described with reference to FIG. 3b. The model may obtain sensing data regarding the surrounding conditions of the vehicle (e.g., the vehicle (110) of FIG. 1) from a sensor (e.g., sensor (601)) mounted on the vehicle (e.g., the vehicle (110) of FIG. 1). The sensing data may be various types of sensing data (e.g., multi-modality sensing data) that can be obtained from the sensor (e.g., sensor (601)) mounted on the vehicle (110). The model may receive the sensing data, path data (603) expressed in SD geometric information, and a driving trajectory (605) to generate a driving path. The driving trajectory (605) may include a record of the path traveled by the vehicle (110) and information regarding the state of the vehicle (110) (e.g., the state of the vehicle (110) in a driving situation, such as speed, acceleration, and steering). The sensing data, path data (603) expressed in SD geometric information, and driving trajectory (605) may be processed by element-wise projection and converted into tokens. An autonomous driving device (e.g., the autonomous driving device (100) of FIG. 1) may input the sensing data, processed path data (e.g., path data (603)), and driving trajectory (605) respectively into a corresponding first mixer. For example, the sensing data, path data (603), and driving trajectory (605) converted into tokens may be input into mixers (611) to (615), respectively. The first mixer (e.g., mixer (611) to mixer (615)) may be an intra mixer. The mixer (611) to mixer (615) may convert data of different dimensions (e.g., sensing data, path data (603), and driving trajectory (605)) into the same dimension.For example, mixers (611) to (615) can encode data of different dimensions (e.g., sensing data, path data (603), and driving trajectory (605)) so that they can be converted into the same dimension. The output of the first mixer (e.g., mixer (611) to (615)) can be input to the second mixer (e.g., mixer (630)). For example, the sensing data, path data, and driving trajectory, which are input to the mixers (611) to (615) respectively and converted into the same dimension, can be input to the mixer (630). The second mixer (e.g., mixer (630)) may be an inter-mixer that performs interaction between different types of data. The second mixer (e.g., mixer (630)) can generate an accurate driving path based on various types of data by performing encoding between different types of data. The model (e.g., the trained model (370) of FIG. 3b) can generate a driving path in which interactions between data (e.g., fusion) are performed with better performance by inputting data of different dimensions into corresponding mixers and processing them primarily.

[0089]

[0090] FIG. 7 illustrates an example of a dataset according to one embodiment.

[0091] Referring to FIG. 7, according to one embodiment, FIG. 7 may represent a dataset in which path data obtained based on navigation information is encoded into SD map-based geometric information. The dataset may be generated based on navigation information generated retrospectively from a driving record. The dataset may consist of a plurality of frames at predetermined time intervals. A dataset generating device (e.g., the dataset generating device (300) of FIG. 3a) may obtain path data based on navigation information generated retrospectively from a driving record (e.g., the driving record (410) of FIG. 4) and a navigation usage record (e.g., the navigation usage record (430) of FIG. 4). The dataset generating device (300) may generate a dataset by encoding the path data into SD map-based geometric information. The path data may include a plurality of vectors of predetermined time units. The dataset generating device (300) may request SD data regarding information about surrounding conditions (e.g., scene images) or driving information (e.g., driving trajectory) obtained from a vehicle (e.g., vehicle (110) of FIG. 1). The dataset generating device (300) may interpolate each vector at intervals of a predetermined distance (e.g., 1 meter). The dataset generating device (300) may interpolate data by finding the point closest to the location of the ego-vehicle (e.g., vehicle (110)) in each frame included in the dataset. For example, the dataset generating device (300) may determine the time that becomes the interpolation interval based on the speed limit at that location. The dataset may consist of nA+nB+1 frames, which is the sum of nA points corresponding to a past time (e.g., A seconds), 1 point corresponding to the current time, and nB points corresponding to a future time (e.g., B seconds in the future). Note that the interpolation interval may be set to 1 / n, but is not limited thereto.

[0092]

[0093] FIGS. 8A and FIGS. 8B are drawings for illustrating an example of a dataset according to one embodiment.

[0094] Referring to FIGS. 8a and 8b, according to one embodiment, a dataset generating device (e.g., the dataset generating device (300) of FIG. 3) can extract turn-by-turn (TBT) information from navigation usage records and navigation information. The TBT information may represent information regarding turn-by-turns in a driving path from a starting point to a destination. The dataset generating device (300) may encode path data and TBT information to generate a dataset. For example, TBT information (811) to TBT information (815) may be data in which turn-by-turn information at points (801) to (805) is encoded. The TBT information (e.g., TBT information (811) to TBT information (815)) may include an offset, which is the distance from the starting point to each turn-by-turn point, a type of turn, a turn location expressed in x and y coordinates, and confidence of the information. FIG. 8b may be another example of a dataset generated by encoding TBT information. Graphs (830) to (850) may each represent the distance from the starting point, the speed of the vehicle at the time corresponding to the frame, and the yaw angle of the vehicle. The dataset may include TBT information extracted from navigation usage records and navigation information.

[0095]

[0096] FIGS. 9a to 9c are drawings for illustrating a predicted path according to one embodiment.

[0097] Referring to FIGS. 9a through 9c, according to one embodiment, a model (e.g., the learned model (370) of FIG. 3b) can predict and output a driving path of a vehicle (e.g., an image taken of the front of the vehicle (110)) based on an image (e.g., an image taken of the front of the vehicle (110)) obtained from a sensor (e.g., an image sensor) mounted on the vehicle (e.g., the vehicle (110) of FIG. 1). For example, an autonomous driving device (e.g., the autonomous driving device (100) of FIG. 1) can generate a right-turn path based on an image taken of a situation where the vehicle (110) needs to make a right turn. The image (910) may be an image taken of a situation where the vehicle (110) needs to make a right turn with the generated right-turn path overlaid on it. Since the driving path is determined based on the destination of the vehicle (110), different driving paths can be generated based on the same image (e.g., the image shown in FIG. 9b and FIG. 9c). For example, the autonomous driving device (100) can generate a left-turn path based on an image of the front of the vehicle (110) shown in FIG. 9b when the vehicle (110) needs to turn left to reach a destination. The image (920) may be an image of the front of the vehicle (110) with the generated left-turn path overlaid. For example, the autonomous driving device (100) can generate a straight-line path based on an image of the front of the vehicle (110) shown in FIG. 9c when the vehicle needs to go straight to reach a destination. The image (930) may be an image of the front of the vehicle (110) with the generated straight-line path overlaid. Images (910) to (930) may be an image of the front of the vehicle with a driving path (e.g., a driving path generated by the autonomous driving device (100)) overlaid on an image of the front of the vehicle. The driving path may be the same as the predicted path generated by the model (370). The predicted path may be generated by the model (370) using the SD navigation path as input.

[0098]

[0099] FIG. 10 is a diagram illustrating a dataset verification process according to one embodiment.

[0100] Referring to FIG. 10, according to one embodiment, the dataset verification process may include a verification image generation process (1010) and a navigation information verification process (1030). A dataset verification device (e.g., the dataset verification device (330) of FIG. 3a) can verify whether the dataset generated by the dataset generation device (e.g., the dataset generation device (300) of FIG. 3a) is appropriate. For example, the dataset verification device (330) can verify whether the navigation information generated retrospectively from a driving record (e.g., the driving record (410) of FIG. 4) by the navigation information generation module (e.g., the navigation information generation module (420) of FIG. 4) of the dataset generation device (300) is appropriately generated.

[0101] In the verification image generation process (1010), the dataset verification device (330) may generate multiple frames containing driving records and navigation information. Each of the multiple frames may include a driving trajectory at a point in time corresponding to each of the multiple frames in the driving records and a path segment corresponding to the driving trajectory in the navigation information. The navigation information may be generated by inputting the vehicle's starting point and destination extracted from the driving records into a navigation path search algorithm. The generated verification images will be described in detail later with reference to FIGS. 11a and FIGS. 11b. The dataset verification device (330) may extract a representative frame from the generated multiple frames. Since navigation information does not change in real time, verifying all of the multiple frames may be inefficient, so the dataset verification device (330) may extract a representative frame. The dataset verification device (330) can verify navigation information based on the difference between the driving record and navigation information included in the representative frame. The dataset verification device (330) can determine the navigation information as failure data in response to the difference between the driving record and navigation information being greater than a predetermined threshold. The failure data is organized into a separate list and may not be used for training a model that generates a driving path (e.g., the trained model (370) in FIG. 3b). The failure data may be used in the navigation information generation algorithm of the dataset generation device (300) after additional classification is performed. The dataset verification device (330) can determine the navigation information as success data in response to the difference between the driving record and navigation information being less than or equal to a predetermined threshold.

[0102]

[0103] FIGS. 11a and FIGS. 11b are drawings for explaining verification results according to one embodiment.

[0104] Referring to FIGS. 11a and 11b, according to one embodiment, a dataset verification device (e.g., the dataset verification device (330) of FIG. 3a) can extract a representative frame from a plurality of generated frames. The representative frame may include at least one frame among the initial frame, middle frame, and last frame of the plurality of frames. Since navigation information generated retrospectively from a driving record may have ambiguity at the starting and ending points of the driving, it may be a suitable target for verifying whether the navigation information was properly generated. The dataset generation device (330) can verify navigation information based on the difference between the generated navigation information (e.g., SD data (1105)) and the driving record (e.g., past trajectory (1101), future trajectory (1103)). For example, in FIG. 11a, the dataset verification device (330) can determine the navigation information (e.g., SD data (1105)) as successful data because the difference between the subsequently generated navigation information (e.g., SD data (1105)) and the driving record (e.g., past trajectory (1101), future trajectory (1103)) is below a predetermined threshold value. For example, in FIG. 11b, the dataset verification device (330) can determine the difference between the subsequently generated navigation information (e.g., SD data (1105)) and the driving record (e.g., past trajectory (1101), future trajectory (1103)) is below a predetermined threshold value. Since it is greater than the threshold value, navigation information (e.g., SD data (1105)) can be determined as failure data.

[0105]

[0106] FIG. 12 is a flowchart illustrating a method according to one embodiment.

[0107] Referring to FIG. 12, according to one embodiment, operations 1210 to 1270 may be operations performed by a processing device (not shown) described with reference to FIG. 1 to FIG. 11b.

[0108] According to one embodiment, operations 1210 to 1270 may be understood to be performed in a processor (e.g., processor (1330) of FIG. 13) of a processing device (not shown) (e.g., electronic device (1300) of FIG. 13) described with reference to FIG. 1 to 11b.

[0109] In operation 1210, the processing device can detect direction change data from the path data.

[0110] In operation 1230, the processing unit can upsample the direction change data.

[0111] In operation 1250, the processing device can process path data by constructing upsampled direction change data into a subset. The processing device can improve data imbalance by constructing upsampled direction change data into a subset. The processed path data can be input into an autonomous driving device (e.g., the autonomous driving device (100) of FIG. 1) and used to generate a driving path.

[0112] In operation 1270, the autonomous driving device (100) can generate a driving path of the vehicle based on processed path data.

[0113] Operations 1210 through 1270 may be performed sequentially, but are not limited thereto. For example, two or more operations may be performed in parallel.

[0114]

[0115] FIG. 13 is a schematic block diagram of an electronic device according to one embodiment.

[0116] Referring to FIG. 13, according to one embodiment, an electronic device (1300) (e.g., autonomous driving device (100) of FIG. 1, dataset generation device (300) of FIG. 3a, dataset verification device (330), processing device (not shown)) may include a memory (1310) and a processor (1330).

[0117] The memory (1310) can store instructions (or programs) executable by the processor (1330). For example, the instructions may include instructions for executing the operation of the processor (1330) and / or the operation of each component of the processor (1330).

[0118] The memory (1310) may include one or more computer-readable storage media. The memory (1310) may include non-volatile storage devices (e.g., magnetic hard disc, optical disc, floppy disc, flash memory, EPROM (electrically programmable memories), EEPROM (electrically erasable and programmable)).

[0119] The memory (1310) may be a non-transitory medium. The term "non-transitory" may indicate that the storage medium is not implemented by a carrier wave or a propagated signal. However, the term "non-transitory" should not be interpreted as meaning that the memory (1310) is immobile.

[0120] The processor (1330) can process data stored in memory (1310). The processor (1330) can execute computer-readable code (e.g., software) stored in memory (1310) and instructions triggered by the processor (1330).

[0121] The processor (1330) may be a data processing device implemented in hardware having a circuit having a physical structure for executing desired operations. For example, the desired operations may include code or instructions included in a program.

[0122] For example, a data processing device implemented in hardware may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

[0123] The processor (1330) can cause the electronic device (1300) to perform one or more operations by executing code and / or instructions stored in memory (1310). The operations performed by the electronic device (1300) may be substantially the same as the operations performed by the processing device (not shown) described with reference to FIGS. 1 through 13. Such redundant descriptions are omitted.

[0124]

[0125] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0126] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0127] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program instructions, data files, data structures, etc., either individually or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0128] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0129] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0130] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. A method for processing path data generated based on a driving log, An operation to detect turning data from the above path data; Upsampling the above direction change data; An operation to process the path data by constructing upsampled direction change data into a subset; and The operation of generating a vehicle's driving path based on processed path data A method including 2. In Paragraph 1, The above path data is, Multiple vectors divided into predetermined time units Includes, The above-mentioned detection operation is, The operation of extracting vector pairs from the plurality of vectors above; and An operation to determine whether the path section corresponding to the vector pair is a turning section based on the above vector pair. A method including 3. In Paragraph 2, The operation of determining whether the above-mentioned direction change data is, An operation to perform a vector multiplication operation on the above vector pair; An operation to determine whether the path segment corresponding to the vector pair is a direction change segment based on the result of the above vector product operation; and An operation of determining the portion corresponding to the vector pair in the path data as the direction change data in response to the fact that the above path segment is a direction change segment. A method including 4. In Paragraph 1, The operation of generating the above driving path is, The operation of obtaining the driving path by inputting the sensing data obtained from the vehicle, the processed path data, and the driving trajectory of the vehicle into a model. A method including 5. In Paragraph 4, The above-mentioned acquisition operation is, The operation of inputting the above sensing data, the above processed path data, and the above driving trajectory respectively to a corresponding first mixer; and The operation of inputting the output of the first mixer to the second mixer to obtain the driving path. A method including 6. In Paragraph 1, The above path data is, Navigation information generated based on the above driving log; Driving state of the above vehicle; and A method generated based on collected navigation history.

7. In Paragraph 1, The above driving record is, A method in which the driving of the above vehicle is recorded chronologically on an SD (standard definition) map.

8. In Paragraph 6, The above navigation information is, A method generated by inputting the starting point and destination of the vehicle extracted from the above driving record into a navigation route search algorithm.

9. In Paragraph 4, The above model is, A method that is an end-to-end model included in a vehicle's autonomous driving system.

10. A computer program stored on a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 through 9.

11. In a device for processing path data generated based on driving records, At least one processor; and memory that stores instructions Includes, Based on the above instructions being executed individually or collectively by the at least one processor, the device, From the above path data, turning data is detected, and Upsampling the above direction change data, and The above path data is processed by constructing upsampled direction change data as a sub-dataset, and A device that generates a driving path of a vehicle based on processed path data.

12. In Paragraph 11, The above path data is, Multiple vectors divided into predetermined time units Includes, Based on the above instructions being executed individually or collectively by the at least one processor, the device, Extract vector pairs from the above plurality of vectors, and A device for determining whether a path section corresponding to the vector pair is a turning section based on the vector pair.

13. In Paragraph 12, Based on the above instructions being executed individually or collectively by the at least one processor, the device, Perform a vector multiplication operation on the above vector pair, Based on the result of the above vector product operation, determine whether the path segment corresponding to the above vector pair is a direction change segment, and A device that determines the portion corresponding to the vector pair in the path data as the direction change data in response to the fact that the path section is a direction change section.

14. In Paragraph 11, Based on the above instructions being executed individually or collectively by the at least one processor, the device, A device for obtaining a driving path by inputting sensing data obtained from the vehicle, the processed path data, and the driving trajectory of the vehicle into a model.

15. In Paragraph 14, Based on the above instructions being executed individually or collectively by the at least one processor, the device, The above sensing data, the above processed path data, and the above driving trajectory are respectively input to the corresponding first mixer, and A device that inputs the output of the first mixer to the second mixer to obtain the driving path.

16. In Paragraph 11, The above path data is, Navigation information generated based on the above driving log; Driving state of the above vehicle; and A device that is generated based on collected navigation history.

17. In Paragraph 11, The above driving record is, A method in which the driving of the above vehicle is recorded chronologically on an SD (standard map).

18. In Paragraph 16, The above navigation information is, A device that is generated by inputting the starting point and destination of the vehicle extracted from the above driving record into a navigation route search algorithm.

19. In Paragraph 14, The above model is, A device that is an end-to-end model included in a vehicle's autonomous driving system.

Images