Method and apparatus for sensor anomaly detection, and device and storage medium

Abstract

According to embodiments of the present disclosure, provided are a method and apparatus for sensor anomaly detection, and a device and a storage medium. The method comprises: acquiring a plurality of images collected by a plurality of image sensors of a vehicle, wherein the plurality of image sensors comprise a first image sensor and a second image sensor, the first image sensor corresponds to a first image among the plurality of images, and the second image sensor corresponds to a second image among the plurality of images; in response to determining that there is an overlap between a first perception range of the first image sensor and a second perception range of the second image sensor, determining a first sub-image corresponding to the overlap in the second image; and determining, at least on the basis of the first image and the first sub-image, a first soiling level corresponding to the first image sensor. Therefore, the embodiments of the present disclosure can improve the comprehensiveness and accuracy of environmental perception, thereby effectively improving the accuracy and reliability of sensor soiling detection.

Description

Methods, apparatus, devices, and storage media for detecting sensor anomalies.

[0001] This application claims priority to Chinese Patent Application No. 202411747432.9, filed on November 29, 2024, entitled "Method, Apparatus, Device and Storage Medium for Detecting Sensor Anomalies", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for detecting sensor anomalies. Background Technology

[0003] In the field of autonomous driving, accurate environmental perception by sensors is crucial for safe vehicle operation. However, in practical use, sensors are susceptible to external environmental influences, which can lead to anomalies and potentially affect their performance. Therefore, improving the accuracy of anomaly detection by sensors is a key concern. Summary of the Invention

[0004] In a first aspect of this disclosure, a method for detecting sensor anomalies is provided. The method includes: acquiring multiple images captured by multiple image sensors of a vehicle, the multiple image sensors including a first image sensor and a second image sensor, the first image sensor corresponding to a first image in the multiple images, and the second image sensor corresponding to a second image in the multiple images; in response to determining that a first sensing range of the first image sensor and a second sensing range of the second image sensor overlap, determining a first sub-image in the second image corresponding to the overlapping region; and determining a first level of dirtiness corresponding to the first image sensor, at least based on the first image and the first sub-image.

[0005] In a second aspect of this disclosure, an apparatus for detecting sensor anomalies is provided. The apparatus includes: a first acquisition module configured to acquire multiple images collected by multiple image sensors of a vehicle, the multiple image sensors including a first image sensor and a second image sensor, the first image sensor corresponding to a first image among the multiple images, and the second image sensor corresponding to a second image among the multiple images; a first determination module configured to determine a first sub-image in the second image corresponding to the overlapping region in response to determining that a first sensing range of the first image sensor and a second sensing range of the second image sensor overlap; and a second determination module configured to determine a first level of dirt corresponding to the first image sensor, based at least on the first image and the first sub-image.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0008] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0009] It should be understood that the content described in this summary section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] Figure 2 illustrates a schematic diagram of a process for detecting sensor anomalies according to some embodiments of the present disclosure;

[0013] Figure 3 illustrates a schematic diagram of detecting image sensor anomalies according to some embodiments of the present disclosure;

[0014] Figure 4 illustrates a schematic diagram of a process for detecting sensor anomalies according to some other embodiments of the present disclosure;

[0015] Figure 5 is a schematic diagram illustrating the detection of lidar anomalies according to some embodiments of the present disclosure;

[0016] Figure 6 shows a schematic structural block diagram of an apparatus for detecting sensor anomalies according to certain embodiments of the present disclosure;

[0017] Figure 7 shows a block diagram of an electronic device capable of implementing several embodiments of the present disclosure. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0020] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0021] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0022] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0023] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," a "machine learning network," or simply a "network," and these terms are used interchangeably.

[0024] Embodiments of this disclosure provide a scheme for detecting sensor anomalies. According to various embodiments of this disclosure, multiple images are acquired from multiple image sensors of a vehicle, including a first image sensor and a second image sensor, the first image sensor corresponding to a first image among the multiple images, and the second image sensor corresponding to a second image among the multiple images; in response to determining that a first sensing range of the first image sensor and a second sensing range of the second image sensor overlap, a first sub-image in the second image corresponding to the overlapping region is determined; and a first level of dirtiness corresponding to the first image sensor is determined, at least based on the first image and the first sub-image.

[0025] The embodiments of this disclosure can detect whether there are any abnormalities in multiple image sensors based on multiple images collected by multiple image sensors on a vehicle, which helps to improve the comprehensiveness and accuracy of environmental perception, and thus can effectively improve the accuracy and reliability of dirt detection of sensors.

[0026] Example Environment

[0027] Figure 1 shows a schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented.

[0028] In this example environment 100, some typical objects are schematically shown, including vehicle 110. In the example of Figure 1, vehicle 110 can be any type of vehicle capable of carrying people and / or goods and moving via a power system such as an engine. Examples of vehicle 110 include, but are not limited to, cars, trucks, buses, electric vehicles, motorcycles, RVs, trains, etc. Vehicle 110 in environment 100 is a vehicle with some level of assisted driving capability or autonomous driving capability; such vehicles are also referred to as intelligent driving vehicles.

[0029] In this example environment 100, multiple image sensors can be installed at predetermined locations on vehicle 110, each capable of acquiring images within a predetermined range. The number of these multiple image sensors can be configured as needed, for example, two, four, etc.

[0030] Vehicle 110 can be communicatively coupled to electronic device 120. Although shown as a separate entity, electronic device 120 may also be embedded within vehicle 110. Alternatively, electronic device 120 may be an entity external to vehicle 110 and may communicate with vehicle 110 via a wireless network. For example, electronic device 120 may be deployed on a roadside or as a remote server. Electronic device 120 may be implemented as one or more computing devices, including at least a processor, memory, and other components typically found in general-purpose computers, to perform functions such as computing, storage, communication, and control.

[0031] Electronic device 120 can detect whether there are any abnormalities in the multiple image sensors based on the multiple images acquired by these multiple image sensors.

[0032] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0033] Example process

[0034] Figure 2 shows a flowchart of a process 200 for detecting sensor anomalies according to some embodiments of the present disclosure. Process 200 can be implemented at electronic device 120. Process 200 is described below with reference to Figure 1.

[0035] In box 210, electronic device 120 acquires multiple images collected by multiple image sensors of vehicle 110, including a first image sensor and a second image sensor, the first image sensor corresponding to a first image among the multiple images, and the second image sensor corresponding to a second image among the multiple images.

[0036] In some embodiments, the number of these multiple image sensors (such as cameras, webcams, etc.) can be set as needed, and these multiple image sensors can be installed at any appropriate location on the vehicle 110. As an example, in order to obtain global information about the surroundings of the vehicle 110, the number of these multiple image sensors can be four, and these four image sensors can be installed at the front left, front rear, rear left, and rear right of the vehicle 110, respectively.

[0037] It should be noted that the terms "first image sensor" and "second image sensor" merely indicate that the plurality of image sensors include different image sensors, and this disclosure does not limit the number of such multiple image sensors. Similarly, "first image" and "second image" merely indicate that the plurality of images include different images, and do not limit the number of such multiple images.

[0038] In block 220, electronic device 120 determines a first sub-image in the second image corresponding to the overlapping region in response to determining that there is an overlapping region between the first sensing range of the first image sensor and the second sensing range of the second image sensor.

[0039] In some embodiments, the first sensing range is the environmental area that can be covered and sensed by the first image sensor. The second sensing range is the environmental area that can be covered and sensed by the second image sensor. The first sensing range and the second sensing range may or may not overlap, and whether or not they overlap may be related to factors such as the positions where the first and second image sensors are positioned.

[0040] In some embodiments, if the first sensing range and the second sensing range overlap, then the first image sensor and the second image sensor have overlapping fields of view. That is, a certain scene appears in both the first and second images. The first sub-image is a portion of the second image, and this first sub-image corresponds to the scene that appears in both the first and second images.

[0041] In frame 230, electronic device 120 determines a first level of dirtiness corresponding to the first image sensor, based at least on the first image and the first sub-image.

[0042] In some embodiments, the soiling level is a quantification of the degree of soiling of the image sensor. The soiling level may include at least two levels to distinguish different degrees of soiling. As an example, the soiling level may include, but is not limited to: light, moderate, heavy, and extreme.

[0043] As an example, the electronic device 120 can determine a first type of dirt corresponding to the first image sensor based on a first image and a first sub-image. The dirt type can be any suitable type, such as, but not limited to, snow cover, raindrop cover, mud cover, water droplet cover, air cover, etc. Furthermore, the electronic device 120 can determine the dirt level corresponding to the first image sensor based on the first type of dirt corresponding to the first image sensor and the correspondence between dirt types and dirt levels. In some embodiments, the correspondence between dirt types and dirt levels can also be set as needed, for example, setting the dirt level corresponding to snow cover to a light level, setting the dirt level corresponding to mud cover to a heavy level, etc.

[0044] In some embodiments, the electronic device 120 may use a target model to determine a first type of dirt corresponding to the first image sensor based on a first image and a second sub-image. The target model can be any suitable machine learning model, such as a classification model, etc., which will not be elaborated here.

[0045] As an example, the electronic device 120 can stitch together a first image and a second sub-image to obtain a stitched image. Furthermore, the electronic device 120 can utilize a target model to determine a first type of contamination corresponding to the first image sensor based on the stitched image.

[0046] Taking Figure 3 as an example, assuming that vehicle 110 is equipped with four image sensors (camera 1, camera 2, camera 3, and camera 4), and that camera 1 captures image 301, camera 2 captures image 302, camera 3 captures image 303, and camera 4 captures image 304, electronic device 120 can use a target model to determine the type of dirt 321 corresponding to camera 1, the type of dirt 322 corresponding to camera 2, the type of dirt 323 corresponding to camera 3, and the type of dirt 323 corresponding to camera 4 based on images 301, 302, 303, and 304. Specifically, assuming that the sensing ranges of cameras 1 and 2 overlap, and the sensing ranges of cameras 1 and 3 overlap, when electronic device 120 uses target model 310 to determine the type of dirt corresponding to camera 1, it can determine the type of dirt corresponding to camera 1 based on the sub-images corresponding to the overlapping areas of cameras 1 and 2 in images 301 and 302, and the sub-images corresponding to the overlapping areas of cameras 1 and 3 in image 303.

[0047] To improve the accuracy of dirt detection by the sensor, the electronic device 120 can determine the final first dirt level corresponding to the first image sensor based on the dirt level detected at multiple times.

[0048] As an example, the electronic device 120 can determine a candidate dirt level corresponding to a target time based at least on a first image and a first sub-image. The target time can be the current time. Further, the electronic device 120 can acquire multiple historical dirt levels of the first image sensor at multiple historical times. These multiple historical dirt levels can be determined from historical images captured by the first image sensor at its corresponding historical time and images captured by other image sensors (specifically, sub-images corresponding to the overlapping areas) that have an overlapping area with the first image sensor's sensing range. The determination process is similar to the determination process for the dirt level at the target time and will not be elaborated here. Further, the electronic device 120 can determine a first dirt level based on the candidate dirt level and multiple historical dirt levels. As an example, the electronic device 120 can determine the dirt level that appears most frequently among the candidate dirt levels and multiple historical dirt levels as the first dirt level. For example, if electronic device 120 determines that the dirt level corresponding to the first image sensor at the current moment is heavy, and seven of the nine historical dirt levels corresponding to the nine historical moments are moderate and two are light, then electronic device 120 can determine that the first dirt level is moderate.

[0049] In some embodiments, the electronic device 120 may further determine candidate contamination types corresponding to a target time moment based at least on the first image and the first sub-image. Further, the electronic device 120 may acquire multiple historical contamination types of the first image sensor at multiple historical times. Further, the electronic device 120 may determine a first contamination type based on the candidate contamination type and the multiple historical contamination types. Further, the electronic device 120 may determine a first contamination level corresponding to the first image sensor based on this first contamination type.

[0050] Taking Figure 3 as an example, and assuming that electronic device 120 obtains dirt type 321 corresponding to camera 1, dirt type 322 corresponding to camera 2, dirt type 323 corresponding to camera 3, and dirt type 324 corresponding to camera 4 at a target time, for each camera, electronic device 120 can use a voting system to count each dirt type based on the dirt type of that camera at the target time and the historical dirt types corresponding to multiple historical times. Furthermore, based on the counting results, electronic device 120 can determine the dirt type that appears most frequently as the voted dirt type corresponding to that camera.

[0051] In some embodiments, similar to determining the first type of dirt on the first image sensor, the electronic device 120 may also consider the overlapping area between the first sensing range of the first image sensor and the second sensing range of the second image sensor when determining the second type of dirt on the second image sensor. Specifically, the electronic device 120 may also determine a second sub-image in the first image corresponding to the overlapping area. Further, the electronic device 120 determines a second level of dirt corresponding to the second image sensor, based at least on the second image and the second sub-image.

[0052] It should be noted that the process for determining the second level of dirtiness is the same as that for determining the first level of dirtiness, and will not be repeated here.

[0053] Using Figure 3 as an example, the electronic device 120 can send the determined dirt level information of these multiple image sensors to a remote device. The remote device can display the interface 330 shown in Figure 3. The remote device can display the dirt level information corresponding to these multiple image sensors on the remote interface 330. The remote device can also provide operation controls for the image sensors on the remote interface 330 to allow operators to control these multiple image sensors based on this remote interface 330. For example, controlling any one of the multiple image sensors to spray air or water to clean the image sensor, etc.

[0054] The following description will continue with reference to the accompanying drawings, which will illustrate some exemplary embodiments of the present disclosure. Figure 4 illustrates a schematic diagram 400 of a process for detecting sensor anomalies according to other embodiments of the present disclosure.

[0055] In box 410, electronic device 120 acquires point cloud information collected by lidar.

[0056] In some embodiments, the electronic device 120 may use a lidar to emit laser pulses and measure the time it takes for them to reflect back in order to acquire point cloud information of the surrounding environment of the vehicle 110. The point cloud information can be information representing the geometry of the surrounding environment corresponding to the vehicle 110 in three-dimensional space. For each point in the surrounding environment, the corresponding point cloud information may include position information (typically X, Y, Z coordinates) and other attribute information, such as color, intensity, or temperature.

[0057] In box 420, electronic device 120 determines the target region from multiple candidate regions based on the number of point clouds in multiple candidate regions.

[0058] In some embodiments, the candidate region can also be called an anchor box. These candidate regions can be areas included in the range that the lidar can collect, and can be any appropriately sized area. Since the shortest ground line loss problem occurs within 10m in front of the vehicle, the candidate region can be set as a rectangular area of ​​1m × 1m × am, where a∈[5,50] and a is the longitudinal distance, so as to calculate the ground line length more precisely and avoid statistically exceeding the point cloud.

[0059] As an example, the electronic device 120 can determine a first region whose point cloud count is less than a first threshold from multiple candidate regions. The first threshold can be any suitable threshold, such as 0, etc. As an example, the electronic device 120 can traverse all candidate regions from farthest to nearest based on the longitudinal distance from the vehicle 110, and determine whether the point cloud count corresponding to each of these candidate regions is less than the first threshold. Further, the electronic device 120 can determine the point cloud count of a set of neighboring regions of the first region. The number of this set of neighboring regions can be set according to needs, such as 8, etc. This set of neighboring regions can be regions within a predetermined range where the first region is located, such as regions within 1m of the first region. Further, the electronic device 120 can determine the first region as the target region in response to the number of regions in a set of neighboring regions whose point cloud count is less than a second threshold being greater than a preset number. The second threshold can be any suitable threshold, such as 4, etc.

[0060] In box 430, electronic device 120 determines the dirt level of the lidar of vehicle 110 based on the target distance from the target area to vehicle 110.

[0061] In some embodiments, the electronic device 120 can determine the target ground line length based on the target distance from the target area to the vehicle 110. The target distance can be any suitable distance, such as a longitudinal distance. Further, the electronic device 120 can determine the dirt level of the LiDAR on the vehicle 110 based on the target ground line length and the correspondence between the ground line length and the dirt level. The dirt level of the LiDAR is a quantification of the degree of dirtiness of the LiDAR. The dirt level can include at least two levels to distinguish different degrees of dirtiness. As an example, the dirt level can include, but is not limited to: light level, moderate level, heavy level, and extreme level.

[0062] In some embodiments, Table 1 is an example table showing the correspondence between ground line length and soiling level:

[0063] Table 1

[0064] To improve the accuracy of dirt detection by the lidar, the electronic device 120 can determine the final dirt level determined by the lidar based on the target distance and multiple historical distances.

[0065] As an example, electronic device 120 can determine the degree of distance deviation based on the target distance and multiple historical distances. These multiple historical distances are determined based on historical point cloud information, and these multiple historical distances are the distances from the target area to vehicle 110 determined at historical moments. The degree of distance deviation can include, but is not limited to, difference, standard deviation, etc. Further, in response to the degree of distance deviation being less than a threshold, electronic device 120 can determine the average of the target distance and multiple historical distances. At this point, the distance from the target area to vehicle 110 is a relatively stable value over a certain period of time (e.g., slowly increasing, remaining unchanged, slowly decreasing, etc.), indicating that a stable point loss problem has occurred on the ground line. Further, electronic device 120 can determine the dirt level of the LiDAR of vehicle 110 based on the average. As an example, electronic device 120 can update the target distance based on this average. Further, electronic device 120 can determine the dirt level of the LiDAR of vehicle 110 based on the updated target distance and the correspondence between ground line length and dirt level.

[0066] In other embodiments, the electronic device 120 may determine that there is no missing point problem on the ground line at this time in response to a distance deviation determined based on the target distance and multiple historical distances being greater than or equal to a threshold.

[0067] Figure 5 illustrates a schematic diagram of the process for detecting lidar anomalies according to some embodiments of the present disclosure.

[0068] In box 501, electronic device 120 can initialize multiple candidate regions (anchor boxes).

[0069] Specifically, the electronic device 120 can initialize information such as the size, position, and number of multiple anchor boxes.

[0070] In box 502, electronic device 120 can update the state of the anchor box.

[0071] In box 503, electronic device 120 can set the initial values ​​of row and column to 0 and start traversing each anchor box. That is, electronic device 120 can traverse each anchor box in order of distance from vehicle 110 from closest to farthest. When the initial value of row and column is 0, it corresponds to the anchor box closest to vehicle 110.

[0072] In box 504, electronic device 120 can detect whether the number of point clouds inside the current anchor box (anchor[row][col]) is less than a first threshold.

[0073] Electronic device 120 may execute operation 505 in response to determining that the number of point clouds inside the current anchor box is less than a first threshold. Electronic device 120 may execute operation 506 in response to determining that the number of point clouds inside the current anchor box is greater than or equal to the first threshold.

[0074] In box 505, electronic device 120 can detect whether the number of point clouds inside the neighboring anchor boxes corresponding to the current anchor box exceeds a preset number.

[0075] Specifically, electronic device 120 can execute operation 506 in response to the fact that the number of missing point clouds inside the neighboring anchor boxes corresponding to the current anchor box does not exceed a preset number. Electronic device 120 can execute operation 507 in response to the fact that the number of missing point clouds inside the neighboring anchor boxes corresponding to the current anchor box exceeds a preset number.

[0076] In box 506, electronic device 120 can traverse to the next anchor box.

[0077] In box 507, electronic device 120 can calculate the ground line length of the current anchor box from vehicle 110.

[0078] In box 508, electronic device 120 can calculate the average of the ground line length based on the ground line length and multiple historical ground line lengths.

[0079] In box 509, electronic device 120 can determine the segmentation result corresponding to the mean, that is, determine which segment corresponds to the dirt level of the mean, where the dirt level is different when the mean is located in different segments.

[0080] In box 510, electronic device 120 can determine the dirt level of the lidar based on the segmentation results.

[0081] In box 511, electronic device 120 can publish a result message indicating the level of contamination corresponding to the lidar.

[0082] In frame 512, electronic device 120 can detect whether the historical anchor boxes corresponding to the current anchor box in 10 frames all include point clouds.

[0083] Specifically, electronic device 120 can execute operation 513 in response to detecting that none of the 10 historical anchor boxes corresponding to the current anchor box include point clouds. Electronic device 120 can also execute operation 514 in response to detecting that not all of the 10 historical anchor boxes corresponding to the current anchor box include point clouds.

[0084] In frame 513, electronic device 120 can detect whether the number of neighboring anchor boxes corresponding to the current anchor box that have no point cloud in the past 10 frames exceeds a threshold.

[0085] Electronic device 120 can execute the operation of box 515 in response to detecting that the number of neighboring anchor boxes corresponding to the current anchor box has not exceeded the threshold in the past 10 frames.

[0086] In box 514, electronic device 120 can traverse to the next anchor box.

[0087] In frame 515, electronic device 120 can determine that there are anomalies in multiple frames.

[0088] In box 516, electronic device 120 can publish events indicating lidar anomalies.

[0089] The following describes the processing procedure after determining the level of dirtiness of the target sensor (image sensor or LiDAR):

[0090] In some embodiments, the electronic device 120 may execute a set of response actions corresponding to a target dirt level, wherein the target sensor includes an image sensor or a LiDAR, in response to a target dirt level of a target sensor among a plurality of sensors exceeding a threshold. The threshold can be any suitable location, such as a light dirt level.

[0091] As an example, electronic device 120 can determine a set of response actions based on the operating status of vehicle 110 and the target level of dirt. The operating status of vehicle 110 can indicate at least one of the following: the test status of vehicle 110, the travel status of vehicle 110, and the driver's seat status of vehicle 110. The test status can include, but is not limited to, before, during, and after the test. The travel status of vehicle 110 includes, but is not limited to, a driving state and a non-driving state. The driver's seat status of vehicle 110 can include, but is not limited to, a driver-occupied state and a driver-unoccupied state.

[0092] Furthermore, the electronic device 120 can execute a defined set of response actions.

[0093] As an example, electronic device 120 can send an assistance request to a remote device in response to a target contamination level exceeding a first level. The first level can be any appropriate level, and its corresponding first level can vary depending on the driver's status in vehicle 110. For instance, if the driver's status in vehicle 110 is "driver in," the first level could be "severe"; if the driver's status in vehicle 110 is "driver unoccupied," the first level could be "moderate." The assistance request can be a confirmation request regarding whether the target contamination level exceeds the first level, allowing personnel to further input confirmation information about the target contamination level via a remote device.

[0094] Furthermore, electronic device 120 can control vehicle 110 to pull over to the side of the road in response to receiving a confirmation message from a remote device regarding a target contamination level higher than a first level. As an example, if the driver's seat of vehicle 110 is in an unmanned state, electronic device 120 can, in response to receiving the confirmation message, determine whether to control vehicle 110 to pull over to the side of the road by the minimum risk condition MRC system or by a redundant system, based on a comparison of the target contamination level and a second level. The second level is greater than the first level. As an example, the second level can be an extreme level, and the first level can be a severe level. In some embodiments, electronic device 120 can determine whether to control vehicle 110 to pull over to the side of the road by the MRC system in response to a target contamination level less than the second level. In other embodiments, electronic device 120 can determine whether to control vehicle 110 to pull over to the side of the road by the redundant system in response to a target contamination level greater than or equal to the second level.

[0095] As another example, if the driver's seat of vehicle 110 is in a driver-occupied state, electronic device 120 can respond to receiving a confirmation message and determine that the redundant system controls vehicle 110 to park on the side of the road.

[0096] In other embodiments, electronic device 120 may control vehicle 110 to slow down in response to a target dirt level being lower than or equal to level one but higher than level three. Level three may be a light level.

[0097] In other embodiments, the electronic device 120 may, in response to the absence of any sensor among the plurality of sensors having a dirt level higher than a threshold, not specify any response action and thus not process the sensor. As an example, the electronic device 120 may, in response to the dirt level being equal to the first level, not process the sensor.

[0098] In some embodiments, the electronic device 120 may also activate the cleaning device of the target sensor, such as activating the water spray device of the target sensor to spray water to achieve self-cleaning, or activating the air jet device of the target sensor to spray air to achieve self-cleaning.

[0099] In some embodiments, the electronic device 120 may also report a work order to a remote device after executing a set of response actions corresponding to the target level of dirt. Based on the work order reported by the remote device, personnel can be dispatched to perform offline cleaning of the target sensor. After completing the cleaning, personnel can take and upload photos including the cleaned target sensor. Furthermore, the vehicle 110 can be stopped from executing this set of response actions, for example, by the MRC system or by a redundant system stopping the vehicle 110 from parking at the roadside.

[0100] In some embodiments, the electronic device 120 can also, in response to the vehicle 110 being in a pre-test state, have its sensors cleaned by personnel to ensure a high level of cleanliness, thereby reducing the rescue costs caused by sensor contamination during vehicle 110 testing. After cleaning, personnel can also take photos including the cleaned sensors and upload them to the inspection system for subsequent verification.

[0101] The embodiments of this disclosure can detect whether there are any abnormalities in multiple image sensors based on multiple images collected by multiple image sensors on the vehicle 110, which helps to improve the comprehensiveness and accuracy of environmental perception, and thus can effectively improve the accuracy and reliability of dirt detection of sensors.

[0102] Example devices and equipment

[0103] Figure 6 shows a schematic structural block diagram of an apparatus 600 for detecting sensor anomalies according to certain embodiments of the present disclosure. The apparatus 600 may be implemented as or included in electronic device 120. The various modules / components in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0104] As shown in the figure, the device 600 includes a first acquisition module 610, configured to acquire multiple images collected by multiple image sensors of the vehicle, the multiple image sensors including a first image sensor and a second image sensor, the first image sensor corresponding to a first image in the multiple images, and the second image sensor corresponding to a second image in the multiple images; a first determination module 620, configured to determine a first sub-image in the second image corresponding to the overlapping area in response to determining that there is an overlapping area between the first sensing range of the first image sensor and the second sensing range of the second image sensor; and a second determination module 630, configured to determine a first level of dirt corresponding to the first image sensor based at least on the first image and the first sub-image.

[0105] In some embodiments, the apparatus 600 further includes a third determining module configured to: determine a second sub-image in the first image corresponding to the overlapping region; and a fourth determining module configured to: determine a second level of dirt corresponding to the second image sensor based at least on the second image and the second sub-image.

[0106] In some embodiments, the second determining module 630 is further configured to: determine a candidate dirt level corresponding to a target time based at least on the first image and the first sub-image; acquire multiple historical dirt levels of the first image sensor at multiple historical times; and determine a first dirt level based on the candidate dirt level and the multiple historical dirt levels.

[0107] In some embodiments, the device 600 further includes a second acquisition module configured to acquire point cloud information collected by the lidar; a fifth determination module configured to determine a target region from multiple candidate regions based on the number of point clouds in multiple candidate regions; and a sixth determination module configured to determine the dirt level of the lidar of the vehicle based on the target distance from the target region to the vehicle.

[0108] In some embodiments, the fifth determining module is further configured to: determine a first region from a plurality of candidate regions whose point cloud count is less than a first threshold; determine the point cloud count of a set of neighboring regions of the first region; and determine the first region as a target region in response to the number of regions in a set of neighboring regions whose point cloud count is less than a second threshold being greater than a preset number.

[0109] In some embodiments, the sixth determining module is further configured to: determine the degree of distance deviation based on the target distance and multiple historical distances, wherein the multiple historical distances are determined based on historical point cloud information; determine the average of the target distance and the multiple historical distances in response to the degree of distance deviation being less than a threshold; and determine the dirt level of the vehicle's lidar based on the average.

[0110] In some embodiments, the apparatus 600 further includes an execution module configured to: in response to a target dirt level of a target sensor among a plurality of sensors being higher than a threshold, execute a set of response actions corresponding to the target dirt level, wherein the target sensor includes an image sensor or a lidar.

[0111] In some embodiments, the execution module is further configured to: determine a set of response actions based on the vehicle's operating state and the target level of dirt; and execute the determined set of response actions.

[0112] In some embodiments, the vehicle's operating status indicates at least one of the following: the vehicle's test status; the vehicle's travel status; and the vehicle's driver status.

[0113] In some embodiments, the execution module is further configured to: send an assistance request to a remote device in response to a target dirt level being higher than a first level; and control the vehicle to stop at the roadside in response to receiving a confirmation message from the remote device that the target dirt level is higher than the first level.

[0114] In some embodiments, the execution module is further configured to: in response to receiving a confirmation message, determine whether to control the vehicle to stop at the roadside by the minimum risk condition MRC system or by the redundant system based on a comparison of the target dirt level with the second level.

[0115] In some embodiments, the execution module is further configured to control the vehicle to slow down in response to a target dirt level that is lower than or equal to the first level and higher than the third level.

[0116] In some embodiments, a set of response actions further includes activating a cleaning device for the target sensor.

[0117] Figure 7 shows a block diagram illustrating a computing device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the computing device 700 shown in Figure 7 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The computing device 700 shown in Figure 7 can be used to implement the electronic device 120 of Figure 1.

[0118] As shown in Figure 7, the computing device 700 is in the form of a general-purpose computing device. Components of the computing device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage devices 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in the memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the computing device 700.

[0119] Computing device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to computing device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within computing device 700.

[0120] The computing device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 7, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0121] The communication unit 740 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 700 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 700 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or another network node.

[0122] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 800 can also communicate as needed with one or more external devices (not shown) via communication unit 740. These external devices, such as storage devices, display devices, etc., can communicate with one or more devices that enable user interaction with computing device 700, or with any device (e.g., network card, modem, etc.) that enables computing device 700 to communicate with one or more other computing devices. Such communication can be performed via input / output (I / O) interfaces (not shown).

[0123] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0124] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0125] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0126] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0128] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for detecting sensor anomalies, comprising: The vehicle acquires multiple images captured by multiple image sensors, including a first image sensor and a second image sensor, wherein the first image sensor corresponds to a first image among the multiple images, and the second image sensor corresponds to a second image among the multiple images; In response to determining that there is an overlapping region between the first sensing range of the first image sensor and the second sensing range of the second image sensor, a first sub-image in the second image corresponding to the overlapping region is determined; as well as Based at least on the first image and the first sub-image, a first level of dirtiness corresponding to the first image sensor is determined.

2. The method according to claim 1, further comprising: Determine the second sub-image in the first image that corresponds to the overlapping region; Based at least on the second image and the second sub-image, a second level of dirtiness corresponding to the second image sensor is determined.

3. The method of claim 1, wherein determining the first level of dirt corresponding to the first image sensor, based at least on the first image and the first sub-image, comprises: Based at least on the first image and the first sub-image, determine the candidate dirt level corresponding to the target time. Obtain multiple historical dirt levels of the first image sensor at multiple historical moments; as well as The first dirt level is determined based on the candidate dirt levels and the plurality of historical dirt levels.

4. The method according to claim 1, further comprising: Acquire point cloud information collected by lidar; Based on the number of point clouds in multiple candidate regions, a target region is determined from the multiple candidate regions; as well as The dirt level of the vehicle's lidar is determined based on the target distance from the target area to the vehicle.

5. The method of claim 4, wherein determining the target region from the plurality of candidate regions based on the number of point clouds in the plurality of candidate regions comprises: From the plurality of candidate regions, a first region is determined whose point cloud number is less than a first threshold; Determine the number of point clouds in a group of neighboring regions of the first region; as well as In response to the fact that the number of regions in the set of neighboring regions with a point cloud count less than a second threshold is greater than a preset number, the first region is determined as the target region.

6. The method according to claim 4, wherein determining the dirt level of the vehicle's lidar based on the target distance from the target area to the vehicle comprises: Based on the target distance and multiple historical distances, the degree of distance deviation is determined, wherein the multiple historical distances are determined based on historical point cloud information; In response to the distance deviation being less than a threshold, the average of the target distance and the plurality of historical distances is determined; as well as Based on the mean value, the dirt level of the vehicle's lidar is determined.

7. The method according to claim 1, further comprising: In response to a target contamination level of a target sensor among a plurality of sensors exceeding a threshold, a set of response actions corresponding to the target contamination level are executed, wherein the target sensor includes an image sensor or a LiDAR.

8. The method of claim 7, wherein performing a set of response actions corresponding to the target level of dirtiness includes: Based on the vehicle's operating status and the target level of dirt, determine the set of response actions; as well as Execute the determined set of response actions.

9. The method of claim 8, wherein the operating state of the vehicle indicates at least one of the following: The test status of the vehicle; The travel status of the vehicle; The vehicle is in the driver's seat position.

10. The method of claim 7, wherein Performing the determined set of response actions includes: In response to the target contamination level being higher than the first level, an assistance request is sent to a remote device; And in response to receiving a confirmation message from the remote device that the target dirt level is higher than the first level, control the vehicle to stop at the roadside; or Executing the determined set of response actions includes: in response to the target contamination level being higher than the first level, sending an assistance request to a remote device; In response to receiving a confirmation message from the remote device that the target dirt level is higher than the first level, and in response to receiving the confirmation message, determining, based on a comparison of the target dirt level with the second level, to control the vehicle to park at the roadside by the minimum risk condition MRC system or by the redundant system.

11. The method according to claim 10, Executing the determined set of response actions further includes: In response to the target dirt level being lower than or equal to the first level but higher than the third level, the vehicle is controlled to reduce its speed. and / or The set of response actions also includes activating a cleaning device for the target sensor.

12. An apparatus for detecting sensor malfunctions, comprising: The first acquisition module is configured to acquire multiple images collected by multiple image sensors of the vehicle, the multiple image sensors including a first image sensor and a second image sensor, the first image sensor corresponding to a first image among the multiple images, and the second image sensor corresponding to a second image among the multiple images; The first determining module is configured to determine a first sub-image in the second image corresponding to the overlapping region in response to determining that there is an overlapping region between the first sensing range of the first image sensor and the second sensing range of the second image sensor. as well as The second determining module is configured to determine a first level of dirtiness corresponding to the first image sensor, based at least on the first image and the first sub-image.

13. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.

14. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 11.

15. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 11.

Images