Obstacle detection method and apparatus for motor vehicle, and computer-readable storage medium

Abstract

Embodiments of the present application provide an obstacle detection method and apparatus for a motor vehicle, and a computer-readable storage medium. The method comprises: extracting image frames frame by frame from a video of the surroundings of a motor vehicle collected and transmitted by a vehicle-mounted camera, and selecting a candidate region comprising an obstacle from each image frame; acquiring a ray projection image of the candidate region of each image frame on the basis of a ray projection mechanism of a camera imaging model; inputting all the ray projection images into a feature extraction network to extract ray projection image features, wherein, starting from a second image frame in time sequence, motion compensation alignment is performed on the ray projection image corresponding to the currently processed image frame and then the ray projection image is inputted into the feature extraction network; fusing the ray projection image features on the basis of the time sequence and motion compensation to obtain a fused feature; and inputting the fused feature into an obstacle boundary and perigee detection model to recognize the obstacle. The embodiments can effectively improve obstacle detection and recognition precision.

Description

Methods, devices and computer-readable storage media for detecting obstacles in motor vehicles Technical Field

[0001] This application relates to the field of motor vehicle obstacle detection technology, and in particular to a method, apparatus and computer-readable storage medium for motor vehicle obstacle detection. Background Technology

[0002] Motor vehicle driver assistance systems are typically equipped with image-based obstacle detection devices to monitor obstacles around the vehicle in real time while the vehicle is in motion, and to provide early warnings and / or take obstacle avoidance measures.

[0003] An existing method for detecting vehicle obstacles based on image vision mainly includes the following steps: first, a predetermined obstacle recognition algorithm is used to filter out candidate boxes containing obstacles around the vehicle from image frames captured by the vehicle-mounted camera; then, a predetermined feature extraction network model is used to extract image features from the candidate boxes; and finally, a deep learning-based network model is used to classify and identify the image features to determine the type of obstacle.

[0004] However, during implementation, the applicant found that the above method relies solely on image features in a single image frame for obstacle recognition, resulting in poor obstacle recognition accuracy and low spatial perception precision for obstacles. Technical issues

[0005] The technical problem to be solved by the embodiments of this application is to provide a method for detecting obstacles in motor vehicles, which can effectively improve the accuracy of obstacle detection and recognition.

[0006] The further technical problem to be solved by the embodiments of this application is to provide a motor vehicle obstacle detection device that can effectively improve the accuracy of obstacle detection and recognition.

[0007] A further technical problem to be solved by the embodiments of this application is to provide a computer-readable storage medium that can store a computer program that can effectively improve the accuracy of obstacle detection and recognition.

[0008] To address the aforementioned technical problems, this application first provides the following technical solution: a method for detecting obstacles in a motor vehicle, comprising the following steps:

[0009] Image frames are extracted frame by frame from video footage of the vehicle's surroundings captured and transmitted by the vehicle-mounted camera, and candidate regions containing obstacles are selected from each image frame.

[0010] The ray projection mechanism based on the camera imaging model is used to obtain the ray projection image of the candidate region for each image frame;

[0011] All ray projection images are input into the feature extraction network to extract ray projection image features. Starting from the second image frame in time sequence, the ray projection image corresponding to the currently processed image frame is first motion compensated and aligned before being input into the feature extraction network.

[0012] The fused features are obtained by fusing the projection features of each ray according to the time sequence and based on motion compensation; and

[0013] The fused features are input into the obstacle boundary and near-point detection model to identify the obstacle.

[0014] Furthermore, the ray projection mechanism based on the camera imaging model for obtaining the ray projection image of the candidate region for each image frame specifically includes:

[0015] The camera extrinsics based on the vehicle-mounted camera determine the pixel coordinates of the camera center point projected onto the road surface of the vehicle in the image frame.

[0016] The candidate region is divided into several sub-regions;

[0017] Construct target rays connecting the pixel coordinate points to the pixel center points of each of the sub-regions; and

[0018] A ray projection image is constructed based on each target ray in the image frame. The RGB values ​​of each row of pixels in the ray projection image correspond to the RGB values ​​of each pixel on a target ray according to the arrangement order of each target ray.

[0019] Furthermore, the step of performing motion compensation alignment on the ray projection image corresponding to the currently processed image frame before inputting it into the feature extraction network specifically includes:

[0020] Based on the inter-frame motion vectors of the vehicle-mounted camera, each target ray in the currently processed image frame is projected onto the historical image frame to generate a corresponding historical target ray in the historical image frame.

[0021] A historical projection image is constructed based on each historical target ray in the historical image frame; and

[0022] The historical projection image and the ray projection image corresponding to the currently processed image frame are input into the feature extraction network.

[0023] Furthermore, the step of obtaining fused features by fusing the features of each ray projection image in a time sequence and based on motion compensation specifically refers to fusing the historical image features extracted by the feature extraction network from the historical projection image and the current image feature corresponding to the ray projection image of the currently processed image frame with the current image features to obtain fused features.

[0024] Furthermore, the historical image features and the current image features are added together and fused to form the fused feature.

[0025] Furthermore, after constructing the ray projection image and the historical projection image, the RGB values ​​of missing pixels in each projection image are first filled in based on the principle of proximity of pixels in the same row, and then the projection image is input into the feature extraction network for feature extraction.

[0026] Furthermore, the obstacle boundary and near-point detection model is a three-layer MLP network structure model. The fused features are input into the three-layer MLP network structure model to identify the projection point of the obstacle on the driving road surface and the type of the obstacle.

[0027] Furthermore, a generalized obstacle detection algorithm model is used to filter out candidate regions containing obstacles around the vehicle from the image frame.

[0028] On the other hand, in order to solve the above-mentioned further technical problems, the present application provides the following technical solution: a motor vehicle obstacle detection device connected to an on-board camera, the device including a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the motor vehicle obstacle detection method as described in any of the above claims.

[0029] Furthermore, in order to solve the aforementioned technical problems, this application provides the following technical solution: a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the motor vehicle obstacle detection method as described in any of the above claims. Beneficial effects

[0030] After adopting the above technical solution, the embodiments of this application have at least the following beneficial effects: After screening out candidate regions containing obstacles in image frames, the embodiments of this application first obtain the ray projection image of the candidate region of each image frame based on the ray projection mechanism of the camera imaging model, and then input all ray projection images into the feature extraction network to extract ray projection image features. Starting from the second image frame in time sequence, for the currently processed image frame, the corresponding ray projection image is first aligned with motion compensation before being input into the feature extraction network, which can improve the accuracy of the image features of the currently processed image frame. Furthermore, after fusing the features of each ray projection image in time sequence and based on motion compensation to obtain fused features, the fused features are input into the preset corresponding obstacle boundary and near-point detection model to identify the obstacle, which can effectively improve the detection and recognition accuracy of obstacles and help improve the efficiency and success rate of obstacle detection. Attached Figure Description

[0031] Figure 1 is a flowchart of an optional embodiment of the vehicle obstacle detection method of this application.

[0032] Figure 2 is a schematic diagram of an image frame of an optional embodiment of the vehicle obstacle detection method of this application.

[0033] Figure 3 is a flowchart of step S2 of an optional embodiment of the vehicle obstacle detection method of this application.

[0034] Figure 4 is a schematic block diagram of an optional embodiment of the vehicle obstacle detection device of this application.

[0035] Figure 5 is a functional block diagram of an optional embodiment of the vehicle obstacle detection device of this application. Embodiments of the present invention

[0036] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments. It should be understood that the following illustrative embodiments and descriptions are only used to explain the present application and are not intended to limit the present application. Moreover, the embodiments and features in the embodiments of the present application can be combined with each other unless otherwise specified.

[0037] As shown in Figure 1, one optional embodiment of this application provides a method for detecting obstacles in a motor vehicle, including the following steps:

[0038] S1: Extract image frames frame by frame from the video images of the vehicle's surrounding environment captured and transmitted by the vehicle-mounted camera 1, and filter out candidate areas containing obstacles in each image frame;

[0039] S2: Obtain the ray projection image of the candidate region for each image frame based on the ray projection mechanism of the camera imaging model;

[0040] S3: Input all ray projection images into the feature extraction network to extract ray projection image features. Starting from the second image frame in time sequence, the ray projection image corresponding to the currently processed image frame is first motion compensated and aligned before being input into the feature extraction network.

[0041] S4: Obtain fused features by fusing the features of each ray projection image sequentially and based on motion compensation; and

[0042] S5: Input the fused features into the obstacle boundary and the near-point detection model to identify the obstacle.

[0043] This embodiment of the application, after filtering out candidate regions containing obstacles in image frames, first obtains the ray projection image of the candidate region for each image frame based on the ray projection mechanism of the camera imaging model. Then, all ray projection images are input into a feature extraction network to extract ray projection image features. Starting from the second image frame in time sequence, for the currently processed image frame, the corresponding ray projection image is first motion-compensated and aligned before being input into the feature extraction network, which can improve the accuracy of image features of the currently processed image frame. Furthermore, after fusing the features of each ray projection image in time sequence and based on motion compensation to obtain fused features, the fused features are input into a preset corresponding obstacle boundary and near-point detection model to identify the obstacle, which can effectively improve the detection and recognition accuracy of obstacles and help improve the efficiency and success rate of obstacle detection.

[0044] In specific implementation, in step S4, the backbone network of the feature extraction network adopts the ResNet-34 network model. Of course, in order to facilitate the addition and fusion of the extracted features, a 1×1 convolutional layer can be connected after the ResNet-34 network model to convert the feature size output by the ResNet-34 network model to a specific size (e.g., 3×254×254).

[0045] In an optional embodiment of this application, as shown in Figures 2 and 3, step S2 specifically includes:

[0046] S21: Based on the camera extrinsic parameters of the vehicle-mounted camera 1, determine the pixel coordinates of the camera center point of the vehicle-mounted camera 1 projected onto the road surface of the motor vehicle in the image frame.

[0047] S22: Divide the candidate region into several sub-regions;

[0048] S23: Construct target rays connecting the pixel coordinate points to the pixel center points of each sub-region; and

[0049] S24: Construct a ray projection image based on each target ray in the image frame, wherein the RGB values ​​of each row of pixels in the ray projection image correspond to the RGB values ​​of each pixel on a target ray according to the arrangement order of each target ray.

[0050] In this embodiment, the pixel coordinates of the center point of the vehicle-mounted camera 1 projected onto the road surface of the vehicle in the image frame are first determined. Then, by dividing the candidate region into several sub-regions, multiple target rays are constructed by sequentially connecting the pixel coordinates with the center point of each sub-region. By sequentially mapping each target ray in the image frame to the pixels of a row of ray projection images, a ray projection image is finally formed. In specific implementation, the RGB values ​​of each row of pixels in the ray projection image correspond one-to-one with each target ray according to the clockwise arrangement order.

[0051] Referring to a specific application example shown in Figure 2, the candidate region is R_seed, which is the pedestrian bounding box in the figure. The candidate region is divided into multiple 5×5 resolution sub-regions, and the center point of each sub-region is represented as (x_c, y_c). The pixel coordinates of the camera center point of the vehicle-mounted camera projected onto the road surface of the vehicle in the current image frame are represented as (X0_bev, Y0_bev), and the target ray is represented as Line_raw. According to specific experiments, the spatial perception accuracy of the vehicle obstacle detection method provided in this embodiment is less than 20cm.

[0052] In one optional embodiment of this application, the process of performing motion compensation alignment on the ray projection image corresponding to the currently processed image frame before inputting it into the feature extraction network specifically includes:

[0053] Based on the inter-frame motion vector of the vehicle-mounted camera 1, each target ray in the currently processed image frame is projected onto the historical image frame to generate a corresponding historical target ray in the historical image frame.

[0054] A historical projection image is constructed based on each historical target ray in the historical image frame; and

[0055] The historical projection image and the ray projection image corresponding to the currently processed image frame are input into the feature extraction network.

[0056] In this embodiment, after projecting each target ray in the currently processed image frame onto the historical image frame using the inter-frame motion vectors of the vehicle-mounted camera 1, a historical projection image is constructed based on each historical target ray in the historical image frame using the same method. Finally, the historical projection image and the ray projection image corresponding to the currently processed image frame are respectively input into the feature extraction network, thereby achieving motion compensation alignment of the ray projection image corresponding to the currently processed image frame. In specific implementation, the historical image frame can be the previous frame or several previous frames, depending on the computational cost and detection efficiency.

[0057] In an optional embodiment of this application, step S4 specifically refers to: fusing the historical image features and current image features extracted by the feature extraction network from the historical projection image and the ray projection image corresponding to the currently processed image frame to obtain fused features. In this embodiment, the historical image features and current image features extracted by the feature extraction network from the historical projection image and the ray projection image corresponding to the currently processed image frame are fused to form fused features, thereby improving the robustness of image features and achieving motion compensation for the current image features.

[0058] In one optional embodiment of this application, the historical image features and the current image features are added and fused to form the fused feature. In this embodiment, image feature fusion is achieved by feature addition, which is a simple method with high data processing efficiency.

[0059] In an optional embodiment of this application, after constructing the ray projection image and the historical projection image, the RGB values ​​of missing pixels in each projection image are first filled in based on the proximity principle of pixels in the same row, and then the projection image is input into the feature extraction network for feature extraction. In this embodiment, since the number of pixels on different target rays in the same image frame is different, after constructing projection images based on each target ray in the same image frame, the number of effective pixels in each row of the projection image is different, which will lead to missing pixels in some rows of the projection image. Therefore, in order to improve the comprehensiveness of subsequent image feature extraction and avoid feature extraction errors, the projection image is processed by filling in the missing pixels in the same row based on the proximity principle of pixels in the same row, that is: the RGB values ​​of effective pixels of adjacent missing pixels in the same row are used to fill in the RGB values ​​of the missing pixels.

[0060] In one optional embodiment of this application, the obstacle boundary and near-point detection model is a three-layer MLP (Multi-Layer Perceptron) network structure model. The fused features are input into the three-layer MLP network structure model to identify and obtain the projection point of the obstacle on the driving road surface and the type of the obstacle. In this embodiment, the three-layer MLP network structure model is used as the detection head to detect, identify, and predict the fused features, resulting in high recognition accuracy. It can be understood that for obstacles in contact with the driving road surface, the projection point of the obstacle on the driving road surface is the contact point; for suspended obstacles, the projection point of the obstacle on the driving road surface is the intersection of the suspended obstacle's projection onto the driving road surface. In addition, the type of obstacle is the classification of obstacles, such as pedestrians, trees, vehicles, etc.

[0061] In one optional embodiment of this application, a generalized obstacle detection algorithm model is used to filter candidate regions containing obstacles around the vehicle from the image frame. In this embodiment, the generalized obstacle detection algorithm model is a common algorithm that can achieve obstacle detection without object detection or semantic segmentation. For example, patent application CN 116092050 A, entitled "Motor Vehicle Obstacle Detection Method," will not be described in detail here.

[0062] In this application, the vehicle-mounted camera 1 is a fisheye camera. Due to image distortion, the target rays in the distorted fisheye image should be arranged in a fan shape with the camera center as the origin. The image frame is an image after distortion correction. Based on this, the target rays within the -60° to +60° field of view can be sequentially constructed into a projection image in a clockwise order. Correspondingly, the three-layer MLP network structure model can also set the output to 120 sensing interfaces, with the 120 sensing interfaces corresponding to a horizontal viewing angle range of 60° to the left and right of the camera centerline as 0°.

[0063] On the other hand, as shown in FIG4, an optional embodiment of this application provides a motor vehicle obstacle detection device 3 connected to a vehicle-mounted camera 1. The device 3 includes a processor 30, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor 30. When the processor 30 executes the computer program, it implements the motor vehicle obstacle detection method as described in any of the above claims.

[0064] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the motor vehicle obstacle detection device 3. For example, the computer program can be divided into functional modules in the motor vehicle obstacle detection device 3 shown in FIG. 5, wherein the image extraction and region filtering module 41, the ray projection image construction module 42, the image compensation module 43, the feature fusion module 44, and the obstacle recognition module 45 respectively perform steps S1-S5 above.

[0065] The vehicle obstacle detection device 3 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The vehicle obstacle detection device 3 may include, but is not limited to, a processor 30 and a memory 32. Those skilled in the art will understand that the schematic diagram is merely an example of the vehicle obstacle detection device 3 and does not constitute a limitation on the vehicle obstacle detection device 3. It may include more or fewer components than shown, or combine certain components, or use different components. For example, the vehicle obstacle detection device 3 may also include input / output devices, network access devices, buses, etc.

[0066] The processor 30 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 30 is the control center of the vehicle obstacle detection device 3, connecting all parts of the device via various interfaces and lines.

[0067] The memory 32 can be used to store the computer program and / or modules. The processor 30 implements various functions of the motor vehicle obstacle detection device 3 by running or executing the computer program and / or modules stored in the memory 32 and calling the data stored in the memory 32. The memory 32 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as image recognition function, image overlay function, etc.), etc.; the data storage area may store data created according to the use of the control device (such as image data, etc.). In addition, the memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0068] If the functions described in the embodiments of this application are implemented in the form of software functional modules or units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the embodiments of this application can implement all or part of the processes in the methods of the above embodiments, or they can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor 30, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted from the content as required by the legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0069] In another aspect, an optional embodiment of this application provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the motor vehicle obstacle detection method as described in any of the preceding claims.

[0070] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0071] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the scope of protection of this application.

Claims

1. A method for detecting obstacles in a motor vehicle, characterized in that, The method includes the following steps: Image frames are extracted frame by frame from video footage of the vehicle's surroundings captured and transmitted by the vehicle-mounted camera, and candidate regions containing obstacles are selected from each image frame. The ray projection mechanism based on the camera imaging model is used to obtain the ray projection image of the candidate region for each image frame; All ray projection images are input into the feature extraction network to extract ray projection image features. Starting from the second image frame in time sequence, the ray projection image corresponding to the currently processed image frame is first motion compensated and aligned before being input into the feature extraction network. The fused features are obtained by fusing the projection features of each ray according to the time sequence and based on motion compensation; and The fused features are input into the obstacle boundary and near-point detection model to identify the obstacle.

2. The method for detecting obstacles to motor vehicles as described in claim 1, characterized in that, The ray projection mechanism based on the camera imaging model for obtaining the ray projection image of the candidate region in each image frame specifically includes: The camera extrinsics based on the vehicle-mounted camera determine the pixel coordinates of the camera center point projected onto the road surface of the vehicle in the image frame. The candidate region is divided into several sub-regions; Construct target rays connecting the pixel coordinate points to the pixel center points of each of the sub-regions; and A ray projection image is constructed based on each target ray in the image frame. The RGB values ​​of each row of pixels in the ray projection image correspond to the RGB values ​​of each pixel on a target ray according to the arrangement order of each target ray.

3. The method for detecting obstacles to motor vehicles as described in claim 1 or 2, characterized in that, The step of performing motion compensation and alignment on the ray projection image corresponding to the currently processed image frame before inputting it into the feature extraction network specifically includes: Based on the inter-frame motion vectors of the vehicle-mounted camera, each target ray in the currently processed image frame is projected onto the historical image frame to generate a corresponding historical target ray in the historical image frame. A historical projection image is constructed based on each historical target ray in the historical image frame; and The historical projection image and the ray projection image corresponding to the currently processed image frame are input into the feature extraction network.

4. The method for detecting obstacles to motor vehicles as described in claim 3, characterized in that, The phrase "fusing features of each ray projection image in a time sequence and based on motion compensation to obtain fused features" specifically refers to fusing the historical image features extracted by the feature extraction network from the ray projection image corresponding to the historical projection image and the currently processed image frame with the current image features to obtain fused features.

5. The method for detecting obstacles to motor vehicles as described in claim 4, characterized in that, The historical image features and the current image features are added together and fused to form the fused feature.

6. The method for detecting obstacles to motor vehicles as described in claim 3, characterized in that, After constructing the ray projection image and the historical projection image, the RGB values ​​of missing pixels in each projection image are first filled in based on the principle of proximity of pixels in the same row, and then the projection image is input into the feature extraction network for feature extraction.

7. The method for detecting obstacles to motor vehicles as described in claim 1, characterized in that, The obstacle boundary and near-point detection model is a three-layer MLP network structure model. The fused features are input into the three-layer MLP network structure model to identify the projection point of the obstacle on the driving road surface and the type of the obstacle.

8. The method for detecting obstacles to motor vehicles as described in claim 1, characterized in that, A generalized obstacle detection algorithm model is used to filter out candidate regions containing obstacles in the image frame.

9. A motor vehicle obstacle detection device, connected to an onboard camera, characterized in that, The apparatus includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the motor vehicle obstacle detection method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the motor vehicle obstacle detection method as described in any one of claims 1 to 8.

Images