Lane line detection method and device, and computer-readable storage medium

Abstract

Provided in the present application are a lane line detection method and device, and a computer-readable storage medium. The method comprises: acquiring a long-focal image and a short-focal image of a vehicle in a traveling direction, wherein the field of view of the long-focal image is less than that of the short-focal image, and the viewing distance of the long-focal image is greater than that of the short-focal image; performing Bird's Eye View (BEV) processing on both the long-focal image and the short-focal image to obtain a long-focal lane line detection result corresponding to the long-focal image and a short-focal lane line detection result corresponding to the short-focal image in a BEV space; and fusing the long-focal lane line detection result and the short-focal lane line detection result to obtain a target lane line. By combining the advantages of the long-focal image and the short-focal image, both the near-range field of view and the far-range viewing range of a vehicle are taken into account, thereby improving lane line detection performance.

Description

Lane detection methods, equipment and computer-readable storage media Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a lane line detection method, device and computer-readable storage medium. Background Technology

[0002] Lane detection is one of the fundamental guarantees for autonomous driving. Real-time lane detection can provide a wealth of useful information for ADS (Automated Driving System), vehicle self-control, localization, and map building.

[0003] Current lane detection methods mainly include 2D and 3D image detection. However, due to the inherent perspective relationship in images, lane lines in the distance tend to cluster and become unclear, affecting the detection accuracy.

[0004] For 3D lane detection methods, when features are transformed to a BEV (Bird's Eye View) perspective, less information is available about distant lanes, and the image resolution is relatively low. This makes it difficult for the network to learn the features of distant lane lines, reducing the accuracy of predictions and resulting in larger biases.

[0005] For 2D lane detection methods, when predicting lane lines at a distance, the 2D image coordinates of the lane lines are first obtained, and then projected onto a flat ground plane using IPM (Inverse Perspective Mapping) to obtain 3D coordinates. Since distant lane lines are clustered and unclear in the image, the accuracy of the 2D image coordinates is low. After IPM projection, the error is further amplified, resulting in a large offset.

[0006] In summary, current methods for lane line detection, whether 2D or 3D, suffer from reduced performance when detecting lane lines at a distance, resulting in poor overall lane line detection performance. Summary of the Invention

[0007] In view of this, one of the objectives of this application is to provide a lane line detection method, device, and computer-readable storage medium to solve at least some of the problems in the related art.

[0008] In a first aspect, the lane line detection method provided according to the embodiments of this application includes:

[0009] Acquire a telephoto image and a short-focus image of the vehicle's driving direction, wherein the field of view of the telephoto image is smaller than that of the short-focus image, and the viewing distance of the telephoto image is greater than that of the short-focus image.

[0010] The telephoto and short-focus images are processed by bird's-eye view BEV processing to obtain the long-focus lane line detection results corresponding to the telephoto image and the short-focus lane line detection results corresponding to the short-focus image in BEV space.

[0011] The target lane line is obtained by fusing the long-focus lane line detection results and the short-focus lane line detection results.

[0012] In an optional embodiment, the process of performing bird's-eye view BEV processing on the telephoto and short-focus images to obtain the telephoto lane line detection results corresponding to the telephoto image and the short-focus lane line detection results corresponding to the short-focus image in BEV space is based on a BEV lane line detection method, which includes:

[0013] A lane line detection model is obtained, which includes a feature extraction module, a spatial transformation module, and a BEV lane line detection module;

[0014] The telephoto image and the short-focus image are input into the lane detection model, and the lane detection model performs the following steps on the input current image:

[0015] The feature extraction module extracts features from the current image to obtain the 2D features of the current image.

[0016] The spatial transformation module transforms the current image from a 2D perspective to a BEV perspective, thereby obtaining the 2D feature mapping of the current image to the BEV feature in the BEV space.

[0017] The BEV lane detection module performs lane detection based on the BEV features to obtain the lane detection results of the current image from the BEV perspective.

[0018] In an optional embodiment, the model further includes a forward detection module for assisting in supervising the feature extraction module.

[0019] In an optional embodiment, inputting the telephoto image and the short-focus image into the lane detection model includes:

[0020] Batch data is constructed using the telephoto image and the short-focus image;

[0021] The batch data is input into the lane detection model, which is used to process the input telephoto image and the short-focus image in parallel to obtain the telephoto lane detection result corresponding to the telephoto image and the short-focus lane detection result corresponding to the short-focus image.

[0022] In an optional embodiment, the spatial transformation module includes a telephoto feature transformation module and a short-focus feature transformation module. The step of transforming the current image from a 2D perspective to a BEV perspective using the spatial transformation module to obtain the 2D feature mapping of the current image to BEV features in the BEV space includes:

[0023] The telephoto feature conversion module converts the 2D features of the telephoto image to obtain BEV features mapped in the BEV space; or

[0024] The short-focus feature conversion module converts the 2D features of the short-focus image to obtain BEV features mapped in the BEV space.

[0025] In an optional embodiment, the space conversion module further includes a valve module; the method further includes:

[0026] The value output by the valve module is determined based on the data type of the 2D features of the image input to the spatial conversion module; the data type of the 2D features of the image includes telephoto and short-focus types.

[0027] Based on the value output by the valve module, the telephoto feature conversion module and the short-focus feature conversion module are switched so that the telephoto feature conversion module performs feature conversion on the 2D features of the telephoto image to obtain BEV features mapped in the BEV space; and the short-focus feature conversion module performs feature conversion on the 2D features of the short-focus image to obtain BEV features mapped in the BEV space.

[0028] In an optional embodiment, obtaining the BEV features mapped in the BEV space includes:

[0029] By combining the value output by the valve module and the BEV feature after the 2D feature of the telephoto image is transformed by the telephoto feature conversion module, the BEV feature of the telephoto image in the BEV space is determined.

[0030] or

[0031] By combining the values ​​output by the valve module and the BEV features obtained by the short-focus feature conversion module after performing feature conversion on the 2D features of the short-focus image, the BEV features of the short-focus image in the BEV space are determined.

[0032] In an optional embodiment, before obtaining the lane detection result from the BEV perspective by performing lane detection based on the BEV features using the BEV lane detection module, the method further includes:

[0033] The point cloud features are obtained after feature extraction of the point cloud data by the feature extraction module. The point cloud data is obtained by scanning the vehicle's driving direction by radar.

[0034] The point cloud features are superimposed with the BEV features to obtain the target BEV features;

[0035] The BEV lane detection module performs lane detection based on the BEV features to obtain lane detection results from the BEV perspective, including:

[0036] The BEV lane detection module performs lane detection based on the target BEV features to obtain lane detection results from the BEV perspective.

[0037] In an optional embodiment, the step of performing lane line detection based on the BEV features using the BEV lane line detection module to obtain the lane line detection result of the current image from the BEV perspective includes:

[0038] The BEV plane, based on the BEV features, is divided into N grids. Multiple parameter values ​​for each grid are predicted at the same spatial resolution based on multiple preset lane line feature parameters. The multiple lane line feature parameters are used to reflect the position and features of the lane lines from the BEV perspective.

[0039] By combining the values ​​of these multiple parameters, the coordinate information of the lane line point set, which serves as the lane line detection result, is obtained.

[0040] In an optional embodiment, the plurality of lane line feature parameters include at least one of confidence level, embedding for clustering, offset of the grid center from the lane in the driving direction, and average height of each grid, which are used to reflect the position and state information of the lane lines in the BEV plane.

[0041] In an optional embodiment, acquiring the telephoto and short-focus images of the vehicle's driving direction includes:

[0042] Obtain the original image of the vehicle's direction of travel;

[0043] Based on the relationship between the field of view and the viewing distance, the original image is transformed to obtain a telephoto image and a short-focus image.

[0044] In an optional embodiment, the lane detection model is trained based on image data collected from multiple vehicles, and the multiple image data are uniformly processed based on the intrinsic / extrinsic parameters of the current vehicle's camera.

[0045] In an optional embodiment, fusing the long-focus lane line detection results and the short-focus lane line detection results to obtain the target lane line includes:

[0046] Determine the point where the distance between the focal lengths of the telephoto image and the short focal length image intersects.

[0047] Determine the first point among all long-focus lane lines in the long-focus lane line detection results that is closest to the intersection with the distance; and determine the second point among all short-focus lane lines in the short-focus lane line detection results that is farthest from the intersection with the distance.

[0048] Determine the distance between the first point and the second point;

[0049] If the distance is less than a threshold, the long-focus lane line corresponding to the first point and the short-focus lane line corresponding to the second point will be merged into a single target lane line.

[0050] Secondly, embodiments of this application provide a storage medium for storing executable code, which is used to execute at runtime the lane detection method provided in the first aspect of embodiments of this application.

[0051] Thirdly, embodiments of this application provide an application for executing at runtime the lane line detection method provided in the first aspect of embodiments of this application.

[0052] Fourthly, embodiments of this application provide a lane line detection device, including: a processor, a memory, a communication interface, and a bus;

[0053] The processor, the memory, and the communication interface are connected via the bus and communicate with each other.

[0054] The memory stores executable program code;

[0055] The processor runs a program corresponding to the executable program code stored in the memory to execute the lane line detection method provided in the first aspect of the present application.

[0056] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0057] Firstly, images of the vehicle's driving direction are acquired for subsequent detection of target lane lines. These images include telephoto and short-focus images. Since short-focus images provide lane line information with a larger field of view in the vehicle's driving direction, while telephoto images provide lane line information with a longer viewing distance or field of view in the vehicle's driving direction, combining the advantages of both telephoto and short-focus images takes into account both the near-field and far-field range of the vehicle, improving the accuracy and comprehensiveness of lane line detection, especially enhancing the ability to detect lane lines at a distance.

[0058] Secondly, bird's-eye view BEV processing is performed on both telephoto and short-focus images to obtain lane line detection results for both the telephoto and short-focus images in BEV space. The target lane line is then detected by post-fusion of these two methods. This approach avoids diluting the features of the telephoto and short-focus images by combining them, instead complementing the advantages of the narrow, far-reaching telephoto and the wide, near-focus short-focus images, thus improving lane line detection accuracy.

[0059] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 is a flowchart of a lane line detection method according to an embodiment of this application. Figures 2(a) to (b) are schematic diagrams of short-focus and long-focus images in a lane line detection method according to an embodiment of this application. Figure 3 is another flowchart of a lane line detection method according to an embodiment of this application. Figure 4 is a source diagram of long-focus and short-focus images in a lane line detection method according to an embodiment of this application. Figure 5 is an architecture diagram of a lane line detection model in a lane line detection method according to an embodiment of this application. Figure 6 is a perspective conversion principle diagram of a spatial conversion module in a lane line detection method according to an embodiment of this application. Figure 7 is a flowchart of fusing lane line detection results in a lane line detection method according to an embodiment of this application. Figure 8 is a schematic diagram of merging long / short-focus lane lines in a lane line detection method according to an embodiment of this application. Figure 9 is a block diagram of a lane line detection device according to an embodiment of this application. Figure 10 is a schematic block diagram of a lane line detection device according to an embodiment of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0063] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Unless otherwise indicated, terms such as “front,” “rear,” “lower,” and / or “upper” are for illustrative purposes only and are not limited to a location or spatial orientation. Terms such as “connected” or “linked” are not limited to physical or mechanical connections and can include electrical connections, whether direct or indirect. In this application, “capable” can mean having the ability.

[0064] The various embodiments of this application will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the following embodiments and features in the implementation methods can be combined with each other.

[0065] Lane detection is one of the fundamental guarantees for autonomous driving. Real-time lane detection provides a wealth of useful information for Automated Driving Systems (ADS), vehicle self-control, localization, and map building. Current lane detection methods mainly include 2D image detection and 3D image detection methods.

[0066] For example, 2D lane detection methods directly detect lane lines in an image, obtaining the 2D coordinates of the lane line points in the image coordinate system. Then, they project these coordinates onto a flat ground plane using IPM (Inverse Perspective Mapping) based on camera intrinsic / extrinsic parameters, followed by curve fitting to obtain the BEV lane lines (i.e., lane lines in 3D coordinates from a top-down view). However, due to inclines and declines on real-world roads, this method can lead to inaccurate 3D results during actual driving. 3D lane perception directly learns the actual 3D coordinates of lane lines from the input image, providing more accurate 3D coordinates compared to the post-projection processing of 2D methods. However, the acquisition of BEV features in 3D lane detection methods heavily relies on camera intrinsic / extrinsic parameters. Therefore, 3D lane detection methods need to incorporate these parameters into the network or use input images with fixed intrinsic / extrinsic parameters to learn BEV features. Inaccurate camera intrinsic / extrinsic parameters directly affect the accuracy of BEV features, resulting in inaccurate 3D lane detection results.

[0067] However, due to the inherent perspective relationship in images, distant lane lines tend to cluster and become unclear. Therefore, both 2D and 3D lane line methods will have reduced performance in detecting distant lane lines.

[0068] For 3D lane detection methods, when features are transformed to a BEV (Bird's Eye View) perspective, less information is available for distant lanes, and the image resolution is relatively low. This makes it difficult for the network to learn the features of distant lanes, reducing the accuracy of the prediction results and leading to a larger offset. For 2D lane detection methods, when predicting distant lanes, the 2D image coordinates of the lanes are first obtained, and then projected onto a flat ground plane using an IPM (Integrated Power Module) to obtain the 3D coordinates. Because distant lanes are clustered and unclear in the image, the accuracy of the 2D image coordinates is low. After IPM projection, the error is further amplified, resulting in a larger offset.

[0069] In summary, current methods for detecting lane lines, whether 2D or 3D, tend to have lower detection accuracy for lane lines at a distance, resulting in poor overall lane line detection performance.

[0070] This application aims to address the problems existing in current lane detection technology by introducing new planning methods and strategies to improve the lane detection performance of vehicles, especially the detection performance of lane lines far from the vehicle.

[0071] The lane detection solution presented in this paper can be applied to any device or system in terminals, vehicles, standalone servers, server clusters, cloud computing, vehicle network platforms, IoT platforms, etc.

[0072] This application provides a first embodiment of a lane line detection method. Referring to FIG1, FIG1 is a schematic flowchart illustrating a lane line detection method according to an embodiment, which may include the following steps:

[0073] In step 100, a telephoto image and a short-focus image of the vehicle's driving direction are acquired, wherein the field of view of the telephoto image is smaller than that of the short-focus image, and the viewing distance of the telephoto image is greater than that of the short-focus image.

[0074] Understandably, in image acquisition scenarios, view distance is typically used to describe the farthest distance at which a lens can clearly capture an object. For telephoto images, the view distance is larger, allowing for the capture of more distant objects. For short-focus images, the view distance is smaller, primarily focusing on objects at closer distances.

[0075] Therefore, in autonomous driving scenarios, short-focus images of the vehicle's direction of travel have the characteristics of a large FOV (Field of View) and a short viewing distance. They can observe a wide range, but distant objects may not be clear enough. Referring to Figure 2(a), while the vehicle is in motion, the short-focus image can show a large area around the vehicle, but details at a distance are relatively blurry.

[0076] Telephoto images taken in the direction of vehicle travel have a smaller field of view (FOV) and a larger viewing distance, meaning they can capture distant objects, but the range of observation is relatively narrow. Referring to Figure 2(b), while the vehicle is in motion, the telephoto image can clearly show distant lane lines, but it may not be able to fully cover objects in a wider range nearby.

[0077] By simultaneously acquiring both telephoto and short-focus images, the advantages of both are combined. At a distance, the clear long-distance information provided by the telephoto image is used for distant lane detection; at close range, the wide field of view of the short-focus image is used to monitor various conditions around the vehicle, including nearby lane detection. This provides richer and more accurate visual information for lane detection, improving the vehicle's driving system's ability to detect lanes at different distances. It also solves the current problem of reduced detection performance for distant lanes due to the inherent perspective effect of near objects appearing larger and farther objects appearing smaller, resulting in clustered and unclear distant lane lines.

[0078] In step 200, the telephoto image and the short-focus image are processed by BEV (Bird's Eye View) to obtain the long-focus lane line detection results corresponding to the telephoto image and the short-focus lane line detection results corresponding to the short-focus image in BEV space.

[0079] BEV lane line information provides more intuitive lane line information from a top-down perspective, which is crucial for path planning and vehicle localization in scenarios such as autonomous driving. It allows the system to better understand the vehicle's surroundings and make more accurate decisions. Therefore, BEV processing is performed on the acquired telephoto and short-focus images. The resulting telephoto / short-focus lane line detection results include, but are not limited to, information such as the position, shape, and direction of the lane lines from the BEV perspective.

[0080] The BEV processing involves performing a series of image transformations and calculations on telephoto and short-focus images using a defined lane detection model. The goal is to convert the telephoto and short-focus images from the traditional camera perspective to the BEV perspective, enabling lane detection within the images. This process will be detailed in subsequent chapters.

[0081] In step 300, the long-focus lane line detection results and the short-focus lane line detection results are fused to obtain the target lane line.

[0082] The telephoto and short-focus images are detected separately, and the results of the telephoto and short-focus lane line detection are fused together. This not only complements the advantages of the telephoto's narrow and far-reaching features and the short-focus's wide and near-reaching features, but also avoids diluting the features of the telephoto and short-focus images and failing to fully utilize the advantages of the telephoto, thus obtaining a more accurate and complete target lane line.

[0083] In this embodiment, the problem of inaccurate lane line detection at a distance is solved by combining long-focus and short-focus image inputs. A post-fusion method is designed to obtain the final detection result by using the long-focus lane line detection result after BEV processing of the long-focus image and the short-focus lane line detection result after BEV processing of the short-focus image. This improves the accuracy and reliability of lane line detection, which is of great significance, especially for application scenarios such as autonomous driving that require accurate lane line information.

[0084] Referring to Figure 3, the above steps are described in detail below:

[0085] In step 100, a telephoto image and a short-focus image of the vehicle's driving direction are acquired, wherein the field of view of the telephoto image is smaller than that of the short-focus image, and the viewing distance of the telephoto image is greater than that of the short-focus image.

[0086] The sources of telephoto and short-focus images include the following two methods:

[0087] If the vehicle currently used for detecting lane lines has both telephoto and focal length lenses, then both lenses can be used directly to collect data, such as taking pictures of the lane lines in the direction of vehicle travel to obtain both telephoto and focal length images.

[0088] If the vehicle currently used for lane detection has only one type of lens, the image captured by that lens is processed to obtain both telephoto and short-focus images.

[0089] As an example, step 100 may specifically include:

[0090] In step 110, the original image of the vehicle's driving direction is acquired;

[0091] In step 120, based on the relationship between the field of view and the viewing distance, the original image is transformed to obtain a telephoto image and a short-focus image.

[0092] The original image is obtained by capturing lane lines in the direction of vehicle travel using a lens mounted on the vehicle. Based on the characteristics of telephoto and short-focal-length lenses in terms of field of view and viewing distance (telephoto: images captured by a lens with a longer focal length have a smaller field of view and a larger viewing distance, allowing for a wider field of view but with a narrower field of view. Short-focal-length: images captured by a lens with a shorter focal length have a larger field of view and a shorter viewing distance, allowing for a wider range of views but with less clarity at distant objects), the original image is processed to obtain telephoto and short-focal-length images.

[0093] Processing the original image can include, but is not limited to, resizing (i.e., image scaling) and cropping (image cropping) operations.

[0094] Referring to Figure 4, a cropping operation is performed on a large area of ​​the original image to select a specific region, resulting in an image with a wide field of view. This cropping operation yields a short-focus image. Alternatively, a cropping operation is performed on a distant local area of ​​the original image, resulting in an image that clearly presents distant scenes and highlights distant details. This cropping operation yields a long-focus image.

[0095] The resize operation is performed on the original image to change its resolution and size, thus meeting different needs.

[0096] The two methods described above can be used to acquire telephoto and short-focus images under different conditions, providing an accurate and comprehensive data foundation for subsequent image processing and analysis.

[0097] In step 200, the telephoto image and the short-focus image are processed by BEV to obtain the long-focus lane line detection results corresponding to the telephoto image and the short-focus lane line detection results corresponding to the short-focus image in BEV space.

[0098] This paper proposes a lane detection model that combines telephoto and short-focus images as inputs. This model couples telephoto and short-focus data, fusing features from both types of data. Through the synergistic effect of multiple modules and technologies, it achieves accurate lane detection and outputs 3D coordinates, thus addressing the problem of inaccurate lane detection at distant locations.

[0099] In some embodiments, the implementation of step 200 is based on a BEV lane detection method, which integrates image feature extraction, spatial transformation processing, BEV lane detection and other technologies, aiming to improve the accuracy and robustness of lane detection, and is especially suitable for autonomous driving scenarios.

[0100] The lane line detection method may specifically include the following steps 210 to 260.

[0101] In step 210, a lane line detection model is obtained, which includes a feature extraction module, a spatial transformation module, and a BEV lane line detection module.

[0102] First, a lane detection model is designed. Lane detection results for both telephoto and short-focus data are obtained and used as training data for the model. Referring to Figure 5, the overall model's network architecture consists of four parts: a data preprocessing module, a feature extraction module, a spatial transformation module, and a BEV lane detection module.

[0103] The first part is the data preprocessing module, which includes the virtual camera.

[0104] The lane detection model is trained based on image data collected from multiple vehicles, and the multiple image data are uniformly processed based on the intrinsic / extrinsic parameters of the current vehicle's camera.

[0105] It's understandable that during the initial training of the lane detection model, data from different vehicles or cameras is used. This data may have different camera parameters (including intrinsic and extrinsic parameters), resulting in different intrinsic / extrinsic parameters for telephoto and short-focus images. This significantly impacts the 3D lane detection results. To enable the model to better learn and adapt to different situations, a unified preprocessing method for camera intrinsic / extrinsic parameters is used for both the telephoto and short-focus images acquired for training. This ensures the model can handle images with different camera parameters during training, improving its generalization ability.

[0106] As an example, continuing with Figure 4, the preprocessing method for unifying camera intrinsic / extrinsic parameters involves establishing a virtual camera with fixed intrinsic / extrinsic parameters. This allows for the separate preprocessing of telephoto and short-focus images used for training, achieving rapid unification of intrinsic / extrinsic parameters for images acquired by different vehicles or cameras. It should be noted that this involves two types of parameters: virtual camera parameters for telephoto images and virtual camera parameters for short-focus images.

[0107] The method obtains massive training data (including telephoto and short-focus image data) and lane detection results corresponding to the telephoto and short-focus images. The training data and lane detection results are used to train a basic lane detection model to obtain a lane detection model that can perform bird's-eye view BEV processing on telephoto and short-focus images respectively, and obtain the long-focus lane detection results corresponding to the telephoto image and the short-focus lane detection results corresponding to the short-focus image in BEV space.

[0108] The second part is the feature extraction module, which includes the backbone network.

[0109] The backbone is a key architectural component of a deep neural network, responsible for extracting features from the input data to provide foundational support for subsequent tasks. In essence, it extracts 2D features of lane lines from the telephoto and short-focus images used for training.

[0110] As an example, a backbone can be, but is not limited to, residual networks (ResNet34), the Inception series of networks, the MobileNet series, DenseNet (Densely Connected Convolutional Networks), etc.

[0111] As an example, a Residual Network (ResNet34) is used as the backbone for the feature extraction module. ResNet34 has a specific structural design that effectively extracts valuable 2D features from both telephoto and short-focus input images. Specifically, the Residual Network structure includes a cross-layer connection to pass residual information. This identity mapping effectively alleviates other problems caused by excessive model depth and significantly improves the module's feature extraction capabilities.

[0112] In other embodiments, the feature extraction module includes a module for extracting features from telephoto / short-focus images, and also includes a module for extracting features from point cloud data.

[0113] Point cloud data is obtained by scanning the vehicle's direction of travel using radar (such as lidar). By inputting point cloud data and combining it with telephoto / short-range images, the final BEV features of the lane lines are obtained. This not only enhances the model's training effect but also improves the accuracy of lane line detection. This part will be discussed in detail in later chapters.

[0114] In other embodiments, the lane detection model includes two feature extraction modules: one feature extraction module extracts features from telephoto / short-focus images, and the other feature extraction module extracts features from point cloud data.

[0115] The third part is the spatial transformation module, which includes the image spatial transformation pyramid. It projects 2D features into the BEV space to obtain BEV features. Since we have two input images with different focal lengths, two feature transformation modules are also designed here: a telephoto feature transformation module and a short-focal-length feature transformation module.

[0116] Referring to Figure 6, the spatial transformation module transforms 2D features to the BEV view (i.e., transforming the 2D view from the top to the top-down view in Figure 6), which is particularly crucial for visual 3D tasks. However, depth-based and transformer-based methods are computationally expensive and not convenient to deploy on autonomous driving chips. To address this issue, a lightweight and easily deployable spatial transformation module is adopted here, namely the VRM (View-Transformation Module) based on MLP (MMulti-Layer Perceptron). The VRM uses several cascaded fully connected layers to learn the relationship between any two pixel positions in the flattened front-view features and the flattened BEV features.

[0117] Since VRM is a fixed mapping, it can only fit the change from image features to BEV features of a camera with one set of intrinsic and extrinsic parameters, and cannot fit the changes brought about by different camera parameters.

[0118] After unifying the virtual cameras mentioned above, there are only two different intrinsic / extrinsic parameters: telephoto virtual camera and short-focus virtual camera. Therefore, we will learn two sets of spatial transformation modules to handle the spatial transformation of these two intrinsic and extrinsic parameters respectively: the telephoto feature transformation module and the short-focus feature transformation module. The telephoto feature transformation module performs feature transformation on the telephoto image of the telephoto virtual camera; the short-focus feature transformation module performs feature transformation on the short-focus image of the short-focus virtual camera.

[0119] In some embodiments, the two spatial transformation modules are integrated into a single lane detection model. Here, we use a learnable valve branch in the model to switch between them. The valve branch is essentially a binary classification network that determines whether the input data is long-focus or short-focus data, and is used to switch between the features derived from the long-focus and short-focus feature transformation modules.

[0120] As an example, when long-focus data is input, the valve branch outputs 01; when short-focus data is input, the valve branch outputs 10. This allows switching between long / short-focus BEV characteristics.

[0121] It's important to note that during training, to enable the model to correctly learn and distinguish between telephoto and short-focus data, it needs to be provided with the correct labels or ground truth values. For this valve branch, the ground truth value is constructed based on whether the input data is telephoto or short-focus. As in the example above, if the input is telephoto data, the ground truth value can be set to 0 or 1; if the input is short-focus data, the ground truth value can be set to 10. These ground truth values ​​will be used to supervise the model's training process. During training, the model predicts the output encoding based on the input data, then compares the prediction result with the ground truth value, and measures the accuracy of the prediction by calculating a loss function. In this way, the model can learn how to correctly distinguish between telephoto and short-focus data and switch to the appropriate feature transformation branch based on different data types.

[0122] Part Four: BEV Lane Detection Module.

[0123] The BEV plane (i.e., the plane in road coordinates) is divided into N grids, each grid representing 0.5 meters.

[0124] During the model prediction phase, the BEV lane detection module predicts the segmented BEV plane and outputs four parts with the same resolution: confidence, embedding for clustering, offset from the grid center to the lane in the y direction, and average height of each grid.

[0125] The confidence level is used to determine the probability that each cell contains lane lines. For example, a cell with a high confidence level has a greater chance of containing lane lines; conversely, a cell with a low confidence level has a lower probability of containing lane lines.

[0126] Embedding is used to distinguish the ID of each lane line. In real-world scenarios, there may be multiple lane lines. Embedding helps the model differentiate between different lane lines and determine which specific lane line each cell belongs to.

[0127] The offset parameter represents the amount of deviation from the grid center to the lane in the y-direction (i.e., the direction of vehicle travel). This parameter can be used to accurately predict the position of lane lines. For example, by adjusting the offset, the specific position of the lane line within the grid can be determined more accurately, especially when the lane line is not perfectly aligned with the grid center.

[0128] The average height of each grid is used to predict the height of lane lines. In a 3D scene, lane line height information is crucial for accurately describing the position and shape of lane lines.

[0129] In the post-processing stage, a confidence threshold is set: cells with a confidence level below the threshold are excluded, and only cells with high confidence are retained for further processing. This reduces false positives and improves detection accuracy.

[0130] Embedding clustering: Clustering cells based on embedding values ​​to identify instances of each line. This determines which cells belong to the same lane line, further clarifying the specific shape of the lane line.

[0131] Offset adjustment: The offset value is used to adjust the precise coordinates of the lane lines. After the previous steps, the possible grid range where the lane lines may be located has been determined. The offset value can be used to more accurately determine the position of the lane lines within this range.

[0132] Height acquisition: The height of the lane lines is determined based on the average height of each grid. Combined with the previously determined planar position information, the 3D coordinates of the lane line point set are finally obtained.

[0133] In summary, during lane detection, the four parameters—confidence, embedding, offset, and height—play important roles at different stages, working together to achieve accurate 3D lane detection.

[0134] In some embodiments, the lane line detection model further includes a fifth part: a forward-looking detection module, used to assist the supervised feature extraction module.

[0135] The fifth part, the forward-looking detection module, receives 2D telephoto / short-focus images as input. These images can be real-time image data collected by sensors such as vehicle cameras. The forward-looking detection module processes the input images using specific algorithms and model structures to predict lane lines within the images. The predicted lane line results can be used as a supervisory signal to assist the feature extraction module.

[0136] Specifically, the predicted lane markings are compared with the actual lane markings, and a loss function is calculated. For example, mean squared error or cross-entropy loss functions can be used to measure the difference between the predicted results and the actual markings. This loss value reflects the performance of the feature extraction model when processing the current image. The calculated loss value is then passed back to the feature extraction module via backpropagation to adjust the model parameters. By continuously optimizing the model parameters, the model in the feature extraction module can better extract features from the image, thereby improving the accuracy of lane marking detection.

[0137] In this way, the forward detection module helps the feature extraction module converge to a better performance state more quickly by providing additional supervision information.

[0138] Understandably, for the lane detection model after training, the feature extraction module extracts features from the input telephoto / short-focus images used for training, obtaining 2D features of the telephoto / short-focus images. This process is simultaneously supervised by the forward-looking detection module to ensure that the extracted features are more conducive to lane detection. Subsequently, the spatial transformation module converts the 2D features of the telephoto / short-focus images from a 2D perspective into features from a BEV perspective, providing a more accurate input for the BEV lane detection module. Finally, the BEV lane detection module performs lane detection based on the BEV features, obtaining the lane detection results from the BEV perspective, i.e., the 3D coordinates of the lane point set.

[0139] In step 220, the telephoto image and the short-focus image are input into the lane detection model, and the lane detection model performs the following steps 230 to 250 on the input current image.

[0140] In step 220, the lane detection model integrates features from telephoto and short-focus images to detect lane lines. This includes processing the telephoto and short-focus images. To improve computational efficiency, the data is fed into the model together for processing.

[0141] As an example, step 220 may specifically include steps 221 to 222:

[0142] In step 221, batch data is constructed using the telephoto image and the short-focus image;

[0143] In step 222, the batch data is input into the lane detection model, which is used to process the input telephoto image and the short-focus image in parallel through the lane detection model to obtain the telephoto lane detection result corresponding to the telephoto image and the short-focus lane detection result corresponding to the short-focus image.

[0144] The term "batch" in this article refers to a batch of data samples. In image processing, a batch can be understood as a set of image data. For example, if the batch size is N, it means processing N images at a time. Therefore, by constructing batch data using telephoto and short-focus images, when the batch size is two images (telephoto and short-focus images), the hardware can fully utilize its parallel processing capabilities, performing the same operations on both images simultaneously, such as feature extraction, spatial transformation, and BEV lane detection.

[0145] In this embodiment, for both long-focus and short-focus images, a single batch input model allows for simultaneous processing of both images, yielding lane line coordinates for both frames. This means that processing two images in a single batch does not increase computation time; feeding the data together into the model effectively improves computational efficiency. Furthermore, by employing a reasonable batch processing strategy, simultaneously processing long-focus and short-focus images can achieve efficient processing of two different focal lengths without significantly increasing computation time.

[0146] In step 230, the feature extraction module extracts features from the current image to obtain the 2D features of the current image.

[0147] The lane detection model can process both real-time acquired telephoto and short-focus images simultaneously. Referring to Figure 5, the residual network in the feature extraction module extracts features from the input telephoto image to obtain the corresponding 2D features; similarly, the residual network in the feature extraction module extracts features from the input short-focus image to obtain the corresponding 2D features.

[0148] In step 240, the current image is converted from a 2D perspective to a BEV perspective by the spatial conversion module, and the 2D feature mapping of the current image to the BEV feature in the BEV space is obtained.

[0149] The spatial transformation module converts 2D features to the BEV perspective to more clearly present the shape and direction of lane lines, improving detection accuracy and robustness. It includes a telephoto feature transformation module and a short-focus feature transformation module, which perform spatial feature transformation on the 2D features of the extracted telephoto and short-focus images, respectively, to obtain BEV features mapped into the BEV space.

[0150] As an example, the telephoto feature conversion module converts the 2D features of the telephoto image to obtain BEV features mapped in the BEV space. Similarly, the short-focus feature conversion module converts the 2D features of the short-focus image to obtain BEV features mapped in the BEV space.

[0151] The two spatial transformation models need to be integrated into one model. Here, we use a learnable valve branch in the model to switch between them. Therefore, a valve module is set in the lane detection model to determine whether the input data is long-focus or short-focus data, and to switch the features after long-focus feature transformation and short-focus feature transformation.

[0152] As an example, step 240 may specifically include the following steps 241-242:

[0153] In step 241, the value output by the valve module is determined according to the data type of the 2D features of the image input to the spatial conversion module; the data type of the 2D features of the image includes telephoto and short-focus types.

[0154] The valve module is configured to switch to the corresponding telephoto / short-focus feature conversion module based on the input telephoto / short-focus image, so that the telephoto feature conversion module performs feature conversion processing on the 2D features of the telephoto image; and the short-focus feature conversion module performs feature conversion processing on the 2D features of the short-focus image, thereby realizing spatial conversion of images with different intrinsic / extrinsic parameters.

[0155] It should be noted that the numerical values ​​output by the valve module are used to indicate different data types, and their setting conditions can be based on predefined rules or conventions. For example, the first digit can represent the distinction between long-focus and short-focus, and the second digit can serve as a confirmation or verification bit. In this case, 0 can represent short-focus, and 1 can represent long-focus.

[0156] The settings can also be customized according to the specific model architecture and requirements. As long as all parts of the entire model processing flow can consistently understand and use this coding method, it will be fine.

[0157] The conditions can also be set based on the data source, collection method, or specific identification information.

[0158] All of the above settings are acceptable and are not specifically limited here; the actual requirements shall prevail. Factors to consider in actual requirements include, but are not limited to, the characteristics and advantages of long / short focal length data, processing efficiency, resolution, etc.

[0159] As an example, when the 2D features of a telephoto image are input, the valve module outputs a value of 01; when the 2D features of a short-focus image are input, the valve module outputs a value of 10.

[0160] In step 242, based on the value output by the valve module, the telephoto feature conversion module and the short-focus feature conversion module are switched so that the telephoto feature conversion module performs feature conversion on the 2D features of the telephoto image to obtain BEV features mapped in the BEV space; and the short-focus feature conversion module performs feature conversion on the 2D features of the short-focus image to obtain BEV features mapped in the BEV space.

[0161] The valve module, after training, can switch between the long-focus feature conversion module and the short-focus feature conversion module based on the output value. That is, when the valve module outputs a value of 01, it switches to the long-focus feature conversion module; when the valve module outputs a value of 10, it switches to the short-focus feature conversion module.

[0162] This embodiment enables the processing of both long-focus and short-focus data within the same lane detection model, improving the model's ability to adapt to different image characteristics, extracting features suitable for lane detection, further enhancing the accuracy of spatial transformation, and increasing the system's flexibility and scalability.

[0163] In some embodiments, the weights of telephoto and short-focus features are dynamically adjusted according to the type of input data, thereby better balancing the two features and making the resulting features more suitable for the current input data.

[0164] As an example, the BEV features of the long-focus image mapped in the BEV space are obtained by combining the numerical value output by the valve module and the BEV features after feature conversion of the 2D features of the long-focus image by the long-focus feature conversion module; or the BEV features of the short-focus image mapped in the BEV space are obtained by combining the numerical value output by the valve module and the BEV features after feature conversion of the 2D features of the short-focus image by the short-focus feature conversion module.

[0165] It can be understood that by using the valve output value as the weighted true value, and performing a weighted summation on the valve output value and the BEV feature after long / short focal length feature conversion, the switching of long / short focal length BEV features can be achieved.

[0166] When the 2D features of a telephoto image are input, the valve outputs a value of 0 or 1. The telephoto feature conversion module performs feature conversion on the 2D features of the telephoto image to obtain the converted telephoto BEV features. A weighted sum is then applied to the valve output value of 0 or 1 and the converted telephoto BEV features to obtain the telephoto BEV features output by the telephoto feature conversion module, which is the BEV feature of the telephoto image mapped in BEV space.

[0167] When the 2D features of a short-focus image are input, the valve outputs a value of 10. The short-focus feature conversion module performs feature conversion on the 2D features of the short-focus image to obtain the converted short-focus BEV features. A weighted sum is then applied to the valve output value of 10 and the converted short-focus BEV features to obtain the short-focus BEV features output by the short-focus feature conversion module, which is the BEV feature of the short-focus image mapped in the BEV space.

[0168] In this embodiment, the adaptability of the features and the robustness of the model can be improved by weighted summing of the numerical values ​​output by the valve module and the corresponding BEV features after feature conversion, thereby better meeting the needs of subsequent tasks such as lane line detection.

[0169] Based on the aforementioned introduction to the lane detection model, the lane detection model can also process the point cloud data of lane lines in the direction of vehicle travel to provide more comprehensive lane line information and improve the accuracy of lane line detection.

[0170] As an example, step 200 further includes steps 260 to 280:

[0171] In step 260, point cloud features are obtained after feature extraction of point cloud data by the feature extraction module. The point cloud data is obtained by scanning the vehicle's driving direction by radar. In step 270, the point cloud features are superimposed with the BEV features to obtain target BEV features. In step 280, lane line detection is performed by the BEV lane line detection module based on the target BEV features to obtain lane line detection results from the BEV perspective.

[0172] Point clouds are a form of 3D data representation that can provide information about the spatial location and shape of objects. Incorporating point clouds as input can provide models with more information dimensions, enhancing their understanding and perception of the environment. Furthermore, point clouds are advantageous for handling complex scenes and occlusion situations, compensating for the limitations of image information.

[0173] The lane detection model includes a feature extraction module that processes point cloud data. This module processes the point cloud data of lane lines collected by the vehicle's radar to obtain more lane-related features. The point cloud features are then overlaid with BEV features to obtain target BEV features. These target BEV features provide more accurate and comprehensive lane information for subsequent target lane detection. With the target BEV features incorporating the point cloud data, the subsequent BEV lane detection module performs lane detection based on these features, obtaining the lane detection results from the BEV's perspective. This process will be described in detail in later chapters.

[0174] In step 250, the BEV lane detection module performs lane detection based on the BEV features to obtain the lane detection result of the current image from the BEV perspective.

[0175] The BEV plane, i.e., the plane in road coordinates, is divided into N grids, each representing 0.5 meters. Four heads with the same resolution are predicted, including confidence, embedding for clustering, offset from the grid center to the lane in the vehicle's direction of travel, and the average height of each grid. After the model output, the confidence level is thresholded, the embedding is used for clustering to determine the instance of each line, the offset is adjusted to obtain the precise coordinates of the lane lines, and the height is obtained. This yields the lane line detection result from the BEV perspective, i.e., the 3D coordinates of the lane line point set corresponding to the telephoto / short-focus image.

[0176] As an example, step 250 may specifically include steps 251 to 252:

[0177] In step 251, the BEV plane based on the BEV features is divided into N grids, and multiple parameter values ​​of each grid are predicted at the same spatial resolution based on multiple preset lane line feature parameters; the multiple lane line feature parameters are used to reflect the position and features of the lane lines from the BEV perspective.

[0178] As an example, the plurality of lane line feature parameters include at least one of confidence level, embedding for clustering, offset of the grid center from the lane in the driving direction, and average height of each grid, which are used to reflect the position and state information of the lane lines in the BEV plane.

[0179] The BEV plane is a plane viewed from a bird's-eye view. Dividing it into multiple small grids allows for a more detailed description of the road environment. Each grid is 0.5 meters in size, thus establishing a discrete spatial representation. Based on multiple preset lane line feature parameters, the confidence score of each grid, the embedding used for clustering, the offset of the grid center from the lane in the driving direction, and the parameter values ​​corresponding to the average height of each grid are predicted at the same spatial resolution.

[0180] In step 252, the coordinate information of the lane line point set, which is the lane line detection result, is obtained by combining the multiple parameter values.

[0181] Thresholding Confidence Assessment: A confidence threshold is set, and only cells with confidence values ​​higher than this threshold are retained, as these cells are considered more likely to contain lane lines. Embedding Clustering: Clustering is performed on the embeddings to determine which cells belong to the same lane line, thus distinguishing different lane line instances. Offset Adjustment: The coordinates of the lane lines are adjusted using offsets to make them more accurate. Height Acquisition: The height of the lane line is determined based on the average height information of each cell. Through these processing steps, the three-dimensional coordinates of the lane line point set can be obtained, describing the position of the lane line in three-dimensional space.

[0182] The lane detection model described above processes both telephoto and short-focus images separately, meaning the model can perform specialized analysis and feature extraction based on the characteristics of images at different focal lengths. Telephoto images excel at providing a clear view of distant objects, allowing the model to focus on extracting detailed features from distant objects; short-focus images capture a wide view of nearby objects, enabling the model to refine its processing of rich information about nearby objects. This results in more refined features that are better adapted to different focal lengths. Furthermore, the fusion of telephoto and short-focus features in the model is solely for switching between the two types of features within a single model. This means that the model can flexibly choose to use telephoto or short-focus features in different scenarios or to achieve optimal perception results. This switching mechanism improves the model's adaptability and efficiency, avoiding unnecessary computation and resource waste.

[0183] In step 300, the long-focus lane line detection results and the short-focus lane line detection results are fused to obtain the target lane line.

[0184] As described above, the lane detection model yields both telephoto lane detection results (i.e., the 3D coordinates of lane point sets in the telephoto image) and short-focus lane detection results (i.e., the 3D coordinates of lane point sets in the short-focus image). The telephoto lane detection results primarily represent distant lanes, while the short-focus lane detection results cover a wider range. Therefore, the telephoto lane detection results are used for lanes within the distant telephoto range, while the short-focus lane detection results are used for lanes in the nearby range. At the intersection of the two focal lengths, the adjacent lanes from the telephoto and short-focus images are merged into a single lane line to obtain the final target lane line.

[0185] Referring to Figure 7, step 300 may specifically include steps 310 to 340:

[0186] In step 310, the distance where the focal lengths of the telephoto image and the short focal length image intersect is determined;

[0187] In step 320, the first point closest to the intersection of all long-focus lane lines in the long-focus lane line detection results is determined; and the second point farthest from the intersection of all short-focus lane lines in the short-focus lane line detection results is determined.

[0188] In step 330, the distance between the first point and the second point is determined;

[0189] In step 340, if the distance is less than a threshold, the long-focus lane line corresponding to the first point and the short-focus lane line corresponding to the second point are merged into a single target lane line.

[0190] The terminus of two focal lengths, as used in this article, refers to the transitional region in observation distance between telephoto and short focal length lenses. As the effective observation distance of a telephoto lens gradually decreases, and as the effective observation distance of a short focal length lens gradually increases, an intermediate region emerges. Within this region, the observation performance of neither the telephoto nor the short focal length lens is optimal, but their observation ranges overlap to some extent. This overlapping region is the junction of the two focal lengths.

[0191] Within this area, specific judgment methods are used, such as taking the nearest point of the telephoto lens and the farthest point of the short-focus lens and determining whether the distance is less than a threshold, to determine whether to merge the adjacent lane lines observed by the telephoto and short-focus lenses into a single lane line, in order to achieve more accurate lane line detection and fusion.

[0192] Specifically, referring to Figure 8, a first point A1 is determined among all long-focus lane lines in the long-focus lane line detection results, which is closest to the intersection point. A second point A2 is determined among all short-focus lane lines in the short-focus lane line detection results, which is farthest from the intersection point. The distance between the first point A1 and the second point A2 is calculated. If this distance is less than a set threshold, the long-focus lane line corresponding to the first point A1 and the short-focus lane line corresponding to the second point A2 are merged into one.

[0193] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0194] Firstly, images of the vehicle's driving direction are acquired for subsequent detection of target lane lines. These images include telephoto and short-focus images. Since short-focus images provide lane line information with a larger field of view in the vehicle's driving direction, while telephoto images provide lane line information with a longer viewing distance or field of view in the vehicle's driving direction, combining the advantages of both telephoto and short-focus images takes into account both the near-field and far-field range of the vehicle, improving the accuracy and comprehensiveness of lane line detection, especially enhancing the ability to detect lane lines at a distance.

[0195] Secondly, bird's-eye view BEV processing is performed on both telephoto and short-focus images to obtain lane line detection results for both the telephoto and short-focus images in BEV space. The target lane line is then detected by post-fusion of these two methods. This approach avoids diluting the features of the telephoto and short-focus images by combining them, instead complementing the advantages of the narrow, far-reaching telephoto and the wide, near-focus short-focus images, thus improving lane line detection accuracy.

[0196] Corresponding to the aforementioned application function implementation method embodiments, this application also provides embodiments of application function implementation apparatus.

[0197] Referring to FIG9, which is a block diagram of a lane line detection device according to an exemplary embodiment, the device includes:

[0198] The image acquisition module is used to acquire a telephoto image and a short-focus image of the vehicle's driving direction, wherein the field of view of the telephoto image is smaller than the field of view of the short-focus image, and the viewing distance of the telephoto image is greater than the viewing distance of the short-focus image.

[0199] The image processing module is used to perform bird's-eye view BEV processing on the telephoto image and the short-focus image respectively, to obtain the long-focus lane line detection result corresponding to the telephoto image and the short-focus lane line detection result corresponding to the short-focus image in the BEV space.

[0200] The lane line detection module is used to fuse the long-focus lane line detection results and the short-focus lane line detection results to obtain the target lane line.

[0201] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0202] Figure 10 illustrates a schematic diagram of the physical structure of a lane line detection device. As shown in Figure 10, the lane line detection device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840. The processor 810, communication interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a parking path planning method.

[0203] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the current technology, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0204] Accordingly, this application also provides a computer-readable storage medium storing a computer program for performing the lane detection method described in any of the first aspects above.

[0205] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0206] The methods and apparatus provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A lane line detection method characterized by, The method comprises: obtaining a long-focus image and a short-focus image of a driving direction of a vehicle, the long-focus image having a smaller field of view angle than the short-focus image, and the long-focus image having a larger view distance than the short-focus image; respectively performing bird's eye view (BEV) processing on the long-focus image and the short-focus image to obtain a long-focus lane line detection result corresponding to the long-focus image and a short-focus lane line detection result corresponding to the short-focus image in a BEV space; fusing the long-focus lane line detection result and the short-focus lane line detection result to obtain a target lane line.

2. The lane line detection method of claim 1, wherein the process of respectively performing bird's eye view (BEV) processing on the long-focus image and the short-focus image to obtain a long-focus lane line detection result corresponding to the long-focus image and a short-focus lane line detection result corresponding to the short-focus image in a BEV space is based on a BEV lane line detection method, and the BEV lane line detection method comprises: obtaining a lane line detection model, the lane line detection model comprising a feature extraction module, a space conversion module, and a BEV lane line detection module; inputting the long-focus image and the short-focus image into the lane line detection model, and performing the following steps on the input current image by the lane line detection model: extracting features of the current image by the feature extraction module to obtain 2D features of the current image; converting the current image from a 2D view to a BEV view by the space conversion module to obtain BEV features of the current image, in which the 2D features of the current image are mapped in the BEV space; performing lane line detection based on the BEV features by the BEV lane line detection module to obtain a lane line detection result of the current image in the BEV view.

3. The lane line detection method of claim 2, wherein the lane line detection model further comprises a front view detection module for assisting in supervising the feature extraction module.

4. The lane line detection method of claim 2, wherein the inputting the long-focus image and the short-focus image into the lane line detection model comprises: constructing batch data by the long-focus image and the short-focus image; inputting the batch data into the lane line detection model, so as to perform parallel processing on the input long-focus image and short-focus image by the lane line detection model to obtain a long-focus lane line detection result corresponding to the long-focus image and a short-focus line detection result corresponding to the short-focus image.

5. The lane line detection method of claim 2, wherein the space conversion module comprises a long-focus feature conversion module and a short-focus feature conversion module, the converting the current image from a 2D view to a BEV view by the space conversion module comprises: converting the 2D features of the long-focus image by the long-focus feature conversion module to obtain BEV features mapped in the BEV space; or The short-focus feature conversion module converts the 2D features of the short-focus image to obtain BEV features mapped in the BEV space.

6. The lane line detection method as described in claim 5, characterized in that, The space conversion module also includes a valve module; The method further includes: The value output by the valve module is determined based on the data type of the 2D features of the image input to the spatial conversion module; the data type of the 2D features of the image includes telephoto and short-focus types. Based on the value output by the valve module, the telephoto feature conversion module and the short-focus feature conversion module are switched so that the telephoto feature conversion module performs feature conversion on the 2D features of the telephoto image to obtain BEV features mapped in the BEV space; and the short-focus feature conversion module performs feature conversion on the 2D features of the short-focus image to obtain BEV features mapped in the BEV space.

7. The lane line detection method as described in claim 6, characterized in that, Obtaining the BEV features mapped in the BEV space includes: By combining the value output by the valve module and the BEV feature after the 2D feature of the telephoto image is transformed by the telephoto feature conversion module, the BEV feature of the telephoto image mapped in the BEV space is obtained. or By combining the numerical value output by the valve module and the BEV features obtained after feature conversion of the 2D features of the short-focus image by the short-focus feature conversion module, the BEV features of the short-focus image mapped in the BEV space are obtained.

8. The lane line detection method as described in claim 2, characterized in that, Before obtaining the lane line detection result from the BEV perspective by performing lane line detection based on the BEV features using the BEV lane line detection module, the method further includes: The point cloud features are obtained after feature extraction of the point cloud data by the feature extraction module. The point cloud data is obtained by scanning the vehicle's driving direction by radar. The point cloud features are superimposed with the BEV features to obtain the target BEV features; The BEV lane detection module performs lane detection based on the BEV features to obtain lane detection results from the BEV perspective, including: The BEV lane detection module performs lane detection based on the target BEV features to obtain lane detection results from the BEV perspective.

9. The lane line detection method as described in claim 2 or 8, characterized in that, The step of detecting lane lines based on the BEV features using the BEV lane line detection module to obtain the lane line detection result of the current image from the BEV perspective includes: The BEV plane, based on the BEV features, is divided into N grids. Multiple parameter values ​​for each grid are predicted at the same spatial resolution based on multiple preset lane line feature parameters. The multiple lane line feature parameters are used to reflect the position and features of the lane lines from the BEV perspective. By combining the values ​​of these multiple parameters, the coordinate information of the lane line point set, which serves as the lane line detection result, is obtained.

10. The lane line detection method as described in claim 9, characterized in that, The multiple lane line feature parameters include at least one of confidence level, embedding for clustering, offset of the grid center from the lane in the driving direction, and average height of each grid. These parameters are used to reflect the position and state information of the lane lines in the BEV plane.

11. The lane line detection method as described in claim 1, characterized in that, The acquisition of telephoto and short-focus images of the vehicle's driving direction includes: Obtain the original image of the vehicle's direction of travel; Based on the relationship between the field of view and the viewing distance, the original image is transformed to obtain a telephoto image and a short-focus image.

12. The lane line detection method as described in claim 2, characterized in that, The lane detection model is trained based on image data collected from multiple vehicles, and the multiple image data are uniformly processed based on the intrinsic and extrinsic parameters of the current vehicle's camera.

13. The lane line detection method as described in claim 1, characterized in that, The step of fusing the long-focus lane line detection results and the short-focus lane line detection results to obtain the target lane line includes: Determine the point where the distance between the focal lengths corresponding to the telephoto image and the short focal length image intersects. Determine the first point among all long-focus lane lines in the long-focus lane line detection results that is closest to the intersection with the distance; and determine the second point among all short-focus lane lines in the short-focus lane line detection results that is farthest from the intersection with the distance. Determine the distance between the first point and the second point; If the distance is less than a threshold, the long-focus lane line corresponding to the first point and the short-focus lane line corresponding to the second point will be merged into a single target lane line.

14. A lane line detection apparatus characterized by comprising: The method includes a memory, a processor, and a lane detection program stored in the memory and executable on the processor, wherein the processor, when executing the lane detection program, implements the steps of the lane detection method as described in any one of claims 1-13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a lane line detection program, which, when executed, implements the steps of the lane line detection method as described in any one of claims 1-13.

Images