Method to generate an artificial lane marking for a vehicle in case of a missing or covered lane marking
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2026-01-28
- Publication Date
- 2026-08-13
Smart Images
Figure EP2026052145_13082026_PF_FP_ABST
Abstract
Description
[0001] 2023PF01588
[0002] 1
[0003] Method to generate an artificial lane marking in case of a missing or covered lane marking for a vehicle
[0004] The invention relates to a method to generate an artificial lane marking in case of a missing or covered lane marking on a road for a vehicle. Moreover, the invention relates to a control device, a vehicle and a computer program product to perform such a method.
[0005] A vehicle may comprise an advanced driver assistance system such as a lane keeping assistance system that requires information about boundaries of a lane of a road on which the vehicle is driving. Typically, the boundaries of the lane may be identified by lane markings painted on a surface of the road. However, there may be at least parts of roads where lane markings are missing, for example, because they have faded or have fallen in disrepair. Besides, there may be at least parts of roads where lane markings are at least temporarily covered, for example, with snow, leaves, fog and / or debris. Such missing or covered lane markings are difficult to detect using typical lane marking detection techniques, although precise information on lane markings are required by the advanced driver assistance system. Therefore, artificial lane markings should be predicted in case of missing or covered lane markings.
[0006] Document US 2018 / 0247138 A1 discloses a method and device to generate virtual lanes. The method includes determining validity of lane detection information extracted from an image in front of a vehicle and generating a virtual lane based on an object included in the image in response to a determination that the lane detection information is not valid.
[0007] Document US 2020 / 0064855 A1 discloses a method and apparatus for determining road lines. The method includes performing detection of two road lines of a road from a driving image, determining whether at least one road line is not detected, and if it is determined that the road line is not detected, determining at least one road line for the at least one undetected road line based on information about a drivable road area.
[0008] It is the object of the invention to predict lane markings in case of missing or covered lane markings on a road.
[0009] The independent claims solve the object.2023PF01588
[0010] 2
[0011] A first aspect of the invention relates to a method to generate an artificial lane marking in case of a missing or covered lane marking on a road. The method thus aims to predict lane markings if actual lane markings are missing or covered. The method is intended for a vehicle, meaning a vehicle may perform the method. In a preferred example, a control device of the vehicle performs the method. In this example, the method may be understood as a computer-implemented method.
[0012] A lane marking in the sense of invention may be understood as an object painted or otherwise placed on a surface of a road. The lane marking may mark a boundary or edge of a lane of the road. The vehicle may be located on the lane of the road so that two opposite lane markings may be expected, each of which is located on one side of the vehicle. The lane marking may be oriented parallel to a driving direction on the lane. The lane marking may be a continuous or a dashed line.
[0013] The missing or covered lane marking may be understood as a lane marking that is currently not visible in a way that a typical lane marking detection algorithm of the vehicle may identify it. For example, if a lane marking has faded or has fallen in disrepair over time, it is no longer visible on the surface of the road. The lane marking is thus missing. However, the missing lane marking is expected on the road because it is supposed to limit the lane spatially. In another example, an object may cover the lane marking, such as snow, leaves, fog and / or debris. The lane marking is thus covered so that it is at least temporarily not visible. However, the covered lane marking is expected on the road because it is actually located on the surface of the road to spatially limit the lane. In another example for a covered lane marking, the lane marking may be covered by a shadow and / or not visible because of a lightning condition.
[0014] The method comprises providing at least one color image and at least one infrared image. The color image and the infrared image each describe at least a part of a road in an environment of the vehicle. The color image and the infrared image may overlap partially or entirely. In a preferred example, multiple color images and multiple infrared images are provided. The control device may receive the at least one color image and the at least one infrared image and provide the received at least one color image and infrared image for further steps of the method. Providing in the sense of the invention thus may mean that2023PF01588
[0015] 3
[0016] the at least one color image and the at least the one infrared image are made available by the control device for further processing by the control device.
[0017] The at least one color image may be based on the RGB color model that uses three color channels for the colors red (R), green (G) and blue (B) to describe color. The at least one infrared image may describe a thermal signature of at least one objects by grayscale values that correspond to an intensity of infrared radiation emitted by the at least one object.
[0018] The method comprises applying a neural network module. The neural network module comprises at least one neural network. The neural network module is applied on the provided at least one color image and the provided at least one infrared image. Applying the neural network module may comprise extracting features from the at least one color image and features from the at least one infrared image individually and fusing the extracted features to determine a lane marking information. The lane marking information describes at least one lane marking on the at least part of the road and / or at least one missing or covered lane marking that is expected on the at least part of the road. This means that based on features derivable from each one of the provided input images, which are the at least one color image and the at least one infrared image, and features determined by combining the features that were derived from each one of the provided input images, the neural network module detects lane markings and / or predicts areas with missing or covered lane markings. The extracted features may be understood as extracted feature maps. This means that the neural network module may determine color feature maps for the at least one color image and infrared feature maps for the at least one infrared image in separate processing steps and fused the determined feature maps afterwards to determine the lane marking information. In an example, that lane marking information describes if a lane marking is or multiple lane markings are detected by the neural network module or if the lane marking is missing according to the neural network module.
[0019] Applying the neural network module may comprise, if the determined lane marking information describes the at least one missing or covered lane marking, predicting at least one location for the at least one missing or covered lane marking on the road. It is thus determined where in the at least one color image and / or the at least one infrared image, a lane marking should be located, which is however not located there because it is missing2023PF01588
[0020] 4
[0021] or covered. The location may be describes by at least one pixel position and / or a coordinate, in particular with respect to the vehicle.
[0022] Applying the neural network module may comprise generating an artificial lane marking for the predicted at least one location. The artificial lane marking is, for example, a part of an image or an image that describes the lane marking that is expected to be located on the road at the location. The artificial lane marking is hence no real or actual lane marking detected when applying the neural network module, but a virtual lane marking that is generated because the actual lane marking is missing or covered. The artificial lane marking is intended to replace the missing or covert lane marking in the at least one color image and / or in the at least one infrared image.
[0023] Furthermore, applying the neural network module may comprise determining an output image based on the provided at least one color image and / or infrared image. The determined output image describes the at least part of the road overlaid by the generated artificial lane marking at the predicted at least one location. The output image is thus a combination of the artificial lane marking and at least one of the input images of the neural network module, which are here the at least one color image and the at least one input image. In a preferred example, the output image is the provided at least one color image overlaid by the generated artificial lane marking. The output image is then a color image of the at least part of the road in the environment of the vehicle in which an area of the missing or covered lane marking is filled with the artificial lane marking. The output image may thus close a gap between lane marking parts of visible lane markings or replace entirely missing or covered lane markings, for example. It was hence possible to identify a missing or covered lane marking on a road and predict an output image in which this lane marking is not missing or covered.
[0024] After applying the neural network module, the method may comprise providing the determined output image. The output image may thus be used for further processing or analysis steps in the vehicle, for example, by the control device. The method hence allows to predict a lane marking in case of a missing or covered lane marking on a road. This is done by generate the artificial lane marking and determining the output image with the generated artificial lane marking.2023PF01588
[0025] 5
[0026] The advantage of the method is to rely on both the at least one color image and the at least one infrared image. During daytime, the color image describes a detailed image for accurate perception. At night or in low visibility conditions, for example in a tunnel, the infrared image ensures that it is possible to describe the road and its surroundings effectively due to the thermal signature of the road and its surroundings. For example, a lane marking that is not missing or covered may be easily detectable in an infrared image due to its heat signature that may differ from surrounding areas on the surface of the road. Therefore, using both the color image and the infrared image enhances an overall reliability for detecting lane markings and missing or covered lane markings across different lighting and environmental conditions.
[0027] An embodiment comprise that the neural network module comprises at least one lightweight convolutional neural network (CNN). Determining the lane marking information comprises applying the at least one lightweight CNN on the at least one color image and / or the at least one infrared image. In a preferred example, one lightweight CNN is applied on the at least one color image and the same lightweight CNN is applied on the at least one infrared image. In another example, the lightweight CNN applied on the at least one color image differs from the lightweight CNN applied on the at least one infrared image. The lightweight CNN may process, for example, the color image to extract features like edges, textures, colors and / or patterns that may support identifying lane markings and / or other objects in or around a road. The lightweight CNN may process, for example, the infrared image to extract features like shapes, thermal patterns and / or other features that are visible in infrared in order to identify lane markings and / or the other objects, in particular in the low light condition. The lightweight CNN does not process the at least one color image and / or the at least one infrared image together but separately.
[0028] The lightweight CNN may use multiple convolutional layers to perform operations on small patches of the at least one color image and / or the at least one infrared image individually. Hereby, features at different levels of abstraction may be extracted. The convolutional layers may be designed to at least detect patterns such as edges, gradients and / or textures. The convolutional layers of the lightweight CNN thus may use the color image and the infrared image to determine feature maps. The determined feature maps may highlight various aspects of the input images, in particular the aspects that are relevant for lane markings. Therefore, the lightweight CNN performs edge detection, texture and pattern recognition and thermal pattern recognition in a preferred example.2023PF01588
[0029] 6
[0030] Moreover, a multimodal fusion may be performed, meaning that the features extracted from the at least one color image and the features extracted from the at least one infrared image are combined, for example, by the lightweight CNN or a further lightweight CNN. The lightweight CNN may thus be configured to combine the features extracted from the color image and the features extracted from the infrared image. The fusion enhances the neural network module's ability to perceive the environment by leveraging the strength of both types of input images, meaning the color image and the infrared image. In a preferred embodiment, only one lightweight CNN is comprised by the neural network module to determine the lane marking information.
[0031] A lightweight CNN architecture is chosen to provide a deep neural network on a small device such as the control device of the vehicle. Known lightweight CNN architectures may be used. The lightweight CNN is particularly trained to detect lane markings and to predict where lane markings should be located and are thus expected. For example, lane markings are detected as objects in the color image and / or infrared image by using bounding boxes and class labels. Applying the lightweight CNN may alternatively or additionally comprise performing a semantic segmentation to classify every pixel of the at least one color image and / or the at least one infrared image at least to determine which pixel belongs to a lane marking. Depth information may also be derived for the color image and / or the infrared image to predicts three-dimensional structures in the environment. By fusing the extracted features, it is thus easily possible to combine visual information and thermal information.
[0032] During the training of the lightweight CNN, a training dataset comprising multiple color images and infrared images may be fed into a lightweight CNN model to train the lightweight CNN. The training procedure may emphasize lane marking detection and lane marking predictions.
[0033] A further embodiment comprise that the neural network module comprise at least one generative adversarial network (GAN). The GAN predicts the at least one location, generates the artificial lane marking and determines the output image. The GAN is thus used to fill in missing, damaged or obscured parts of the color image and / or infrared image where the lane markings are missing or covered.2023PF01588
[0034] 7
[0035] The GAN is a neural network architecture that comprises two neural networks, a generator and a discriminator. During a training process, the GAN learns to generate realistic outputs, meaning here output images with artificial lane markings, by pitting the generator against the discriminator in a minimum-maximum game. The generator may take random noise as input and try to generate images that look like real images of lane markings. The discriminator tries to distinguish between real images of lane markings that are provided by a training dataset on one hand and fake images generated by the generator on the other hand. Through this adversarial training process, the generator and the discriminator improve over time so that the generator learns to generate realistic images while the discriminator learns to distinguish between real and fake artificial lane markings. By doing so, the GAN may as well be trained on determining where to locate the artificial lane marking in the at least one color image and to determine the output image. After the GAN has been trained, it may be used to inpaint missing or covered lane marking parts in the color image and / or the infrared image.
[0036] Applying the GAN may comprise determining the area in the color image and / or infrared image where the missing or covered lane marking is expected. The determined area is then masked out and thus marked, for example.
[0037] After the training process, the generator may receive an incomplete image, meaning the provided at least one color image and / or infrared image, then determines individually where to locate the lane marking to complete the image. The generator then attempts to generate a realistic completion of the incomplete image, which is here the output image. An output of the generator is evaluated by the discriminator, which determines how realistic the completed image, meaning the output image, looks and may assign a percentage value that describes a confidence of the output image.
[0038] In other words, according to a preferred example the neural network module at least comprises a lightweight CNN and a GAN in order to determine the output image. The GAN may be understood as a virtual lane generation network, whereas the lightweight CNN is intended to check if lane markings are detected or are missing or covered.
[0039] A further embodiment comprises that the neural network module is applied on multiple consecutively provided color images and infrared images to determine the lane marking information. If the lane marking is detected in at least one of the color images and / or infrared2023PF01588
[0040] 8
[0041] images, meaning that the determined lane marking information describes that the lane marking is detected, the lane marking is tracked in the color images and / or infrared images provided afterwards. This means that in the consecutively provided color images and infrared images, continuing lane markings are tracked and thus followed through multiple color images and / or infrared images if possible. If the tracked lane marking disappears in at least one of the colored images and infrared images provided after detecting the lane marking for the first time, the lane marking information describes the missing or covered lane marking. The color images and / or infrared images provided afterwards thus describe the at least part of the road at a later point in time compared to the latest color image and / or infrared image in which the lane marking was tracked. Tracking lane markings and identifying sudden gaps in tracked lane markings is a fast and reliable way to determine the lane marking information.
[0042] A further embodiment comprises that the method comprises determining that the tracked lane marking disappeared by analyzing a distance between lane marking parts of the lane marking in consecutive color images and / or infrared images. The lane marking parts may be understood as segments or sub-parts of one lane marking. The distance between lane marking parts is particularly reasonable when considering lane markings in the shape of dashed lines on the surface of the road.
[0043] The method comprises checking if the distance is outside a distance range assigned to a present lane marking. The distance range assigned to a present lane marking means a value range for distances between individual lane marking parts that are considered to be typical for lane markings that are visual and thus present on the road. The distance range may be defined by a minimum distance value and a maximum distance value. This embodiment is based on the observation that missing or covered lane markings typically occur when lane markings are temporarily covered by obstacles such as other vehicles, shadows or road debris, or when the lane markings have worn out or faded. Such missing or covered lane markings are quickly and reliably detected under consideration of the distance between individual lane marking parts.
[0044] A further embodiment comprises that the distance range is determined under consideration of at least one previously analyzed distance between lane marking parts of the tracked lane marking. It is thus possible to use at least one previously detected distance between lane2023PF01588
[0045] 9
[0046] marking parts to device if a currently analyzed distance between lane marking parts indicates the missing or covered lane marking or not. If the distances between lane marking parts increases, for example, compared to previously analyzed distances for the same lane marking, a missing or covered lane marking may be assumed. Alternatively, the distance range may be a set range that may be predetermined and stored in a storage unit of the control device and / or the vehicle. This means that a dynamic or fixed threshold that limits the distance range may be considered to provide versatile possibilities to determine if the lane marking is missing or covered or not.
[0047] A further embodiment comprises that after determining that the tracked lane marking disappeared, the at least one location is predicted by interpolation. The Interpolation is done under consideration of the location of a last or latest detected lane marking part of the tracked lane marking. This location means the location of the lane marking in the latest color image and / or infrared image in which the lane marking was detected as such. The interpolation may as well or alternatively consider a predicted distance to a consecutive lane marking part. The predicted distance may be determined under consideration of at least one previously analyzed distance between lane marking parts of the tracked lane marking. This allows a dynamic adaptation to the lane marking. In other words, once gaps or occlusions are detected along the lane marking, the method comprises interpolating the positions of the lane marking parts based on the last known positions and the expected distance between lane markings. This allows to maintain a continuous representation of the lane markings even in the presence of missing parts or covered parts of the lane marking.
[0048] In a preferred embodiment, it is considered that the determined lane marking information describes the at least one missing or covered lane marking. In this case, an area of the at least one missing or covered lane marking is masked in the provided at least one color image and or / or infrared image. The area is considered when predicting the at least one location. This means that dynamic masking may be performed that may be updated in real time. Missing regions may be identified, which are here areas in the at least one color image and / or at least one infrared image in which the lane marking is expected but visible. The method may comprise determining such areas and mask them in the at least one color image and / or infrared image so that in these images it is clearly visible where the artificial lane marking may be positioned or may not be positioned.2023PF01588
[0049] 10
[0050] According to a further embodiment, determining the output image comprises applying at least one post-processing technique to integrate the artificial lane marking in the provided at least one color image and / or infrared image. Alternatively or additionally, applying the at least one post-processing technique may increase at least locally an image quality of the output image. The output image and / or the integration of the artificial lane marking in the provided at least one color image and / or infrared image may be improved by performing blending and / or smoothing as post-processing techniques. This may result in a seamless integration of the artificial lane marking in the color image and / or in the infrared image. To improve the image quality of at least the artificial lane marking or the output image, for example, color, shape and / or positioning of the artificial lane marking in the output image may be adapted. The post-processing technique may be a rule-based technique that is applied after applying the neural network module. Applying the post-processing techniques may thus not be part of the neural network module. Alternatively, the post-processing technique is applied by the neural network module. This helps to provide particular accurate output images that do not show that they were generated artificially.
[0051] Another embodiment considers that the determined lane marking information describes the at least one lane marking. In this situation, the method comprises operating at least one advanced drive assistance system of the vehicle under consideration of the determined lane marking information. The determined lane marking information may thus be fed to the advanced driver assistance system. The advanced driver assistance system is in a preferred example a lane keeping assistance system. In case the presence of a lane marking is detected in the color image and / or infrared image, it may be possible to provide this information to the advanced driver assistance system for further processing. The advanced driver assistance system may be configured to determine at least one control command for a drive system, a brake system and / or a steering system of the vehicle. In case the lane markings are detectable it is also possible to rely on the described method and use the determined lane marking information for the advanced driver assistance system.
[0052] Another embodiment comprises that if the determined lane marking information describes the at least one missing or covered lane marking, the method comprises operating at least one driver assistance system of the vehicle under consideration of the provided output image. The image with the artificial lane marking is then used by the advanced driver assistance system to, for example, operate the vehicle. The artificial lane marking is thus con-2023PF01588
[0053] 11
[0054] sidered as a real lane marking that is, for example, relevant for driving commands or steering commands of the vehicle. This is particularly useful and shows how the missing and covered lane marking may be repaired by the method.
[0055] A further embodiment comprises that at least one provided color image is captured by a color camera of the vehicle and / or the at least one provided infrared image is captured by an infrared camera of the vehicle. The color camera and the infrared camera may then transmit the captured images to the control device so that the control device may provide the at least one color image and the at least one infrared image. The color camera and the infrared camera may be located in a front area of the vehicle, for example at or in a bumper and / or an upper part of a windshield of the vehicle. Alternatively or additionally, the color camera and the infrared camera may be located on a side of the vehicle and / or in a rear area of the vehicle. Therefore, the at least one provided color image and / or the at least one provided infrared image may be captured and processed in real-time.
[0056] The neural network module may be trained in advance, meaning before performing the method, during a training procedure. During training, training data may be fed to the untrained neural network module. The training data may be comprised by a training dataset, such as a training data set used for autonomous driving research that comprises color images and infrared images and annotations for object detection. Each image of the training dataset may be annotated with a corresponding semantic label, creating a ground truth for training. Techniques like rotation, scaling, and flipping may be applied to increase the dataset size and improve model robustness. The used color images and / or infrared images may be captured at different light and / or weather conditions.
[0057] Another aspect of the invention relates to a control device for a vehicle. The control device is configured to perform the above-described method. The control device may perform the method.
[0058] The control device may be understood as a computing unit or as a data processing device with processing circuitry. The control device may therefore perform computing operations in order to process data and hence the computer-implemented method. The computing operations may also include indexed accesses to a data structure, for example a look-up table (LUT).2023PF01588
[0059] 12
[0060] In particular, the control device may comprise at least one computer, at least one microcontroller, and / or at least one integrated circuit, for example, at least one applicationspecific integrated circuit (ASIC), at least one field-programmable gate arrays (FPGA), and / or at least one system on a chip (SoC). The control device may comprise at least one processor, for example, at least one microprocessors, at least one central processing unit (CPU), at least one graphic processing unit (GPU), and / or at least one signal processor, in particular at least one digital signal processor (DSP). The control device may comprise a physical or a virtual cluster of computers or other of said units.
[0061] The control device may comprise at least one hardware and / or software interface and / or at least one storage unit or memory unit. The storage or memory unit may be implemented as a volatile data memory, for example a dynamic random access memory (DRAM), or a static random access memory (SRAM), or as a non-volatile data memory, for example a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory or flash EEPROM, a ferroelectric random access memory (FRAM), a magnetoresistive random access memory (MRAM), or a phase-change random access memory (PCRAM).
[0062] Another aspect of the invention relates to a vehicle that is configured to perform the above-described method. The vehicle performs the method. The vehicle is in a preferred example a motor vehicle, for example, a passenger car, a truck, a bus, a motorcycle and / or a moped.
[0063] A further aspect of the invention relates to a computer program product. The computer program product is a computer program. The computer program product comprises instructions which, when the program is executed by a computer, such as the control device, cause the computer to perform the above-described method.
[0064] The embodiments described in connection with the inventive method, both individually and in combination with each other, apply accordingly, when applicable, to the inventive control device, vehicle, and computer program product. The invention comprises combinations of the described embodiments.
[0065] The figures show:2023PF01588
[0066] 13
[0067] Fig. 1 a schematic representation of a road with missing or covered lane markings;
[0068] and
[0069] Fig. 2 a schematic representation of a method to generate an artificial lane marking in case of a missing or covered lane marking on a road.
[0070] Fig. 1 shows a vehicle 1 driving on a road 2 in a driving direction 9. The road 2 has two lanes. Edges or boundaries of the lanes are marked with lane markings 3. Here, outer edges of the road 2 are marked with two continuous lines as lane markings 3. The two lanes are separated from each other by a dashed center line as another lane marking 3.
[0071] The road 2 shows two examples for missing or covered lane markings 3. In an example A, the dashed center line painted on the surface of the road 2 has locally fainted so that there is a gap between individual lane marking parts of the lane parking 3 that separates the two lanes. In an example B, leaves 4 have been fallen from a tree 5 next to the road 2 so that the leaves 4 cover at least locally one of the lane markings 3. Example A thus shows a missing lane marking 3 and example B a covered lane marking 3. In an example C, a visible lane marking 3 is sketched, meaning a lane marking 3 that is neither missing nor covered.
[0072] The vehicle 1 may comprise a color camera 6 and an infrared camera 7 in a front area of the vehicle 1. Other or more positions of the color camera 6 and / or the infrared camera 7 are possible, for example at sides and / or in a rear area of the vehicle 1. In a preferred example, the color camera 6 and / or the infrared camera 7 are located at an upper end of a windshield of the vehicle 1 (not sketched here) or in or at a front bumper of the vehicle 1. The vehicle 1 may comprise a control device 8 to perform a method that is described in the following.
[0073] Fig. 2 shows steps of a method to generate an artificial lane marking 3 in case of a missing or covered lane marking 3 on the road 2. A step S1 of the method may be performed by the color camera 6 and the infrared camera 7. In step S1 , the color camera 6 may capture at least one color image 10 describing at least a part of the road 2 in an environment of the vehicle 1. Additionally, the infrared camera 7 may capture at least one infrared image 11 describing at least a part of the road 2 in an environment of the vehicle 1.2023PF01588
[0074] 14
[0075] A step S2 as well as following steps of the method may be performed by the control device 8 of the vehicle 1. Step S2 may comprise providing the at least one color image 10 and the at least one infrared image 11 . In a step S3, the method may comprise applying a neural network module 12 on the at least one color image 10 and the at least one infrared image 11. The neural network module 12 may comprise at least one artificial neural network.
[0076] Applying the neural network module 12 may comprise the following steps: It may comprise extracting features from the at least one color image 10 and the at least one infrared image 11 individually and using the extracted features to determine a lane marking information 15. The lane marking information 15 describes the at least one lane marking 3 on the road 2 and / or at least one missing or covered lane marking 3 that is expected on the road 2. These first steps of the neural network module 12 may be performed by at least one lightweight convolutional neural network (CNN) 13. The lightweight CNN 13 may be applied on the at least one color image 10 and on the at least one provided infrared image 11 individually and may fuse hereby extracted features. An output of the lightweight CNN 13 may be the lane marking information 15. To better illustrate the fusion of the extracted features from the two different types of image sources, meaning from the color image 10 and the infrared image 11 , a fusion step 14 is sketched.
[0077] Applying the neural network module 12 may comprise predicting at least one location for the missing or covered lane marking on the road 2 in case the determined lane marking information 15 describes the missing or covered lane marking 3. This is, for example, the case for the examples A and B. Besides, an artificial lane marking 3 for the predicted at least one location is generated and an output image 17 is determined based on the provided at least one color image 10 and / or infrared image 11 . The determined output image 17 describes the at least part of the road 2 overlaid by the generated artificial lane marking 3 at the predicted at least one location. Locating the lane marking 3, generating the artificial lane marking 3 and determining the output image 17 may be performed by at least one generative adversarial network (GAN) 16 that may be part of the neural network module 12.
[0078] The neural network module 12 may be applied on multiple consecutively provided color images 10 and infrared images 11 to determine the lane marking information 15. If the lane marking 3 is detected in at least one of the color images 10 and infrared images 11 ,2023PF01588
[0079] 15
[0080] the lane marking 3 is tracked in the color images 10 and / or the infrared images 11 provided afterwards, meaning after the color image 10 or infrared image 11 in which the lane marking 3 was first detected. If the tracked lane marking 3 disappears in at least one of the color images 10 and / or infrared images 11 provided after the tracked lane marking 3 was first detected, the lane marking information 15 describes the missing or covered lane marking 3.
[0081] The method may comprise determining that the tracked lane marking 3 has disappeared by analyzing a distance between lane marking parts of the lane marking 3 in consecutive color images 10 and / or infrared images 11 and checking if the distance is outside of a distance range assigned to the present lane marking 3. The present lane marking 3 may be understood as a visible lane marking 3. The distance range may be determined under consideration of at least one previously analyzed distance between lane marking parts of the tracked lane marking 3 or may be a set range. After determining that the tracked lane marking 3 disappeared, the at least one location may be predicted by interpolation under consideration of the location of a last detected lane marking part of the tracked lane marking 3 and a predicted distance to a consecutive lane marking part.
[0082] If the determined lane marking information 15 describes the missing or covered lane marking 3, an area of the missing or covered lane marking 3 may be masked in the provided at least one color image 10 and / or infrared image 11 . This area may be considered when predicting the at least one location and thus when applying the GAN 16, for example.
[0083] After determining the output image 17, at least one post-processing technique 19 may be applied on the output image 17 to integrate the artificial lane marking 3 in the provided at least one color image 10 and / or infrared image 11 on one hand, and / or to increase at least locally an image quality of the output image 17 on the other hand.
[0084] A step S4 may comprise providing the determined output image 17. In a step S5, the output image 17 may be provided to an advanced driver assistance system 18, such as a lane keeping assistance system. In an example, this takes place if the output image 17 is determined and thus in case the lane marking information 15 describes the at least one missing or covered lane marking 3. However, in case the lane marking information 15 describes the lane marking 3 and thus a visible lane marking 3, as it is the case in example2023PF01588
[0085] 16
[0086] C, the step S5 may comprise providing the determined lane marking information 15 to the advanced driver assistance system 18 of the vehicle 1 . Afterwards, for example, the advanced driver assistance system 18 may determine at least one control command for the vehicle 1 , for example, for a longitudinal and / or transversal guidance of the vehicle 1 , under consideration of the lane marking information 15 and / or the provided output image 17.
[0087] In summary, the invention shows a method based on artificial intelligence (Al) to percept the unlabeled road 2 for active safety in autonomous driving. Challenges are observed, where the road lane (lane markings 3) is not properly visible or available for driving autonomous vehicles 1 . The method considers three important aspects:
[0088] - Perception of unlabeled roads 2 due to unavailability of lane markings 3, faded lane markings 3, or occluded lane markings 3.
[0089] - Virtual lane marking construction on road surface.
[0090] - Accurate detection of unlabeled lane markings 3 and drawing the virtual lane (artificial lane marking 3) in night vision.
[0091] The lane keeping assistance system detects when the vehicle 1 deviates from a lane and automatically adjusts the steering to restore proper travel inside the lane without additional input from the driver. This system fails under the following conditions
[0092] - The system relies on painted lane markings 3 to operate. This system is not designed to work with markers that are faded, covered, in disrepair or overly complicated.
[0093] - If the road 2 is covered with snow, leaves, fog or debris, the lane keeping assist system may not be able to detect the lane markings 3 on the road 2.
[0094] A RGB front sensor and an infrared sensor are used to percept unlabeled roads in all driving scenarios like: road lane covered with snow, road lane covered with water, occluded with debris, faded lanes, and worn by tyres and so on.
[0095] Two advanced Al-based computer vision algorithms are developed to detect the unavailable or partially.
[0096] In autonomous driving lane detection for safety measures, detecting the unlabeled road lanes through our proposed Al-based algorithms can be a leading solution for better visibility and safety. Both infrared images 11 and color images 10, here in BEV (Bird Eye View) and / or front view, may be utilized in the data capturing process during night and day2023PF01588
[0097] 17
[0098] vision mode. The proposed advanced Al-based algorithms can be best suited for detection of not labelled road lanes (lane markings 3) along with generating virtual lanes (artificial lane markings 3) for smooth driving conditions in any level of advanced driver assistance system autonomy.
[0099] The invention relates to a method to generate an artificial lane marking 3 in case of a missing or covered lane marking 3 on a road 2 for a vehicle 1 , comprising: providing a color image 10 and an infrared image 11 ; and applying a neural network module 12 on the color image 10 and the infrared image 11. Applying the neural network module 12 com-prises: extracting features from the color image 10 and the infrared image 11 individually and fusing them to determine a lane marking information 15; in case the lane marking information 15 describes the missing or covered lane marking 3, predicting a location for the missing or covered lane marking 3 on the road 2; generating an artificial lane marking 3; and determining an output image 17 in which the road 2 is overlaid by the generated artifi-cial lane marking 3.
Claims
2023PF0158818Claims1. Method to generate an artificial lane marking (3) in case of a missing or covered lane marking (3) on a road (2) for a vehicle (1), comprising:- providing (S2) at least one color image (10) and at least one infrared image (11) each describing at least a part of a road (2) in an environment of the vehicle (1); and - applying (S3) a neural network module (12) comprising at least one neural network on the provided at least one image (10) and the provided at least one infrared image (11);wherein applying the neural network module (12) comprises:- extracting features from the at least one color image (10) and features from the at least one infrared image (11) individually and fusing the extracted features to determine a lane marking information (15) describing at least one lane marking (3) on the road (2) and / or at least one missing or covered lane marking (3) that is expected on the road (2);- if the determined lane marking information (15) describes the at least one missing or covered lane marking (3), predicting at least one location for the at least one missing or covered lane marking (3) on the road (2);- generating an artificial lane marking (3) for the predicted at least one location; - determining an output image (17) based on the provided at least one color image (10) and / or infrared image (11), wherein the determined output image (17) describes the at least part of the road (2) overlaid by the generated artificial lane marking (3) at the predicted at least one location;- providing (S4) the determined output image (17).
2. Method according to claim 1 , wherein the neural network module (12) comprises at least one lightweight convolutional neural network (13) and determining the lane marking information (15) comprises applying the at least one lightweight convolutional neural network (13) on the at least one provided color image (10) and / or the at least one provided infrared image (11 ).
3. Method according to any one of the preceding claims, wherein the neural network module (12) comprises at least one generative adversarial network (16) that predicts2023PF0158819the at least one location, generates the artificial lane marking (3) and determines the output image (17).
4. Method according to any one of the preceding claims, wherein the neural network module (12) is applied on multiple consecutively provided color images (10) and infrared images (11) to determine the lane marking information (15), wherein if the lane marking (3) is detected in at least one of the color images (10) and / or infrared images (11), the lane marking (3) is tracked in the color images (10) and / or infrared images (11) provided afterwards, wherein if the tracked lane marking (3) disappears in at least one of the color images (10) and / or infrared images (11) provided afterwards the lane marking information (15) describes the missing or covered lane marking (3).
5. Method according to claim 4, wherein the method comprises determining that the tracked lane marking (3) disappeared by analyzing a distance between lane marking parts in consecutive color images (10) and / or infrared images (11) and checking if the distance is outside a distance range assigned to a present lane marking (3).
6. Method according to claim 5, wherein the distance range is determined under consideration of at least one previously analyzed distance between lane marking parts of the tracked lane marking (3) or is a set range.
7. Method according to claim 5 or 6, wherein after determining that the tracked lane marking (3) disappeared, the at least one location is predicted by interpolation under consideration of a location of a last detected lane marking part of the tracked lane marking (3) and a predicted distance to a consecutive lane marking part.
8. Method according to any one of the preceding claims, wherein if the determined lane marking information (15) describes the at least one missing or covered lane marking (3), an area of the at least one missing or covered lane marking (3) is masked in the provided at least one color image (10) and / or infrared image (11), wherein the area is considered when predicting the at least one location.
9. Method according to any one of the preceding claims, wherein determining the output image (17) comprises applying at least one post-processing technique (19) to2023PF0158820integrate the artificial lane marking (3) in the provided at least one color image (10) and / or infrared image (11), and / or to increase at least locally an image quality of the output image (17).
10. Method according to any one of the preceding claims, wherein if the determined lane marking information (15) describes the at least one lane marking (3), the method comprises operating at least one advanced driver assistance system (18) of the vehicle (1) under consideration of the determined lane marking information (15).
11. Method according to any one of the preceding claims, wherein if the determined lane marking information (15) describes the at least one missing or covered lane marking (3), the method comprises operating at least one advanced driver assistance system (18) of the vehicle (1) under consideration of the provided output image (17).
12. Method according to any one of the preceding claims, wherein at least one provided color image (10) is captured by a color camera (6) of the vehicle (1) and / or the at least one provided infrared image (11) is captured by an infrared camera (7) of the vehicle (1) (S1).
13. Control device (8) for a vehicle (1 ) configured to perform a method according to any one of claims 1 to 11.
14. Vehicle (1) configured to perform a method according to any one of the claims 1 to 12.
15. Computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform a computer-implemented method according to any one of claims 1 to 11.