Control method and apparatus, and system

Abstract

A control method and apparatus, and a system. In the method, a first control apparatus may acquire an operating parameter of a second sensing module; and on the basis of the operating parameter of the second sensing module, a first sensing module is controlled to achieve scanning timing registration between the first sensing module and the second sensing module, wherein the second sensing module and the first sensing module share the same receiving optical assembly. The method can achieve coordinated spatio-temporal detection for different sensing modules, thereby improving the accuracy of corresponding fusion sensing algorithms.

Description

A control method, device and system Technical Field

[0001] This application relates to the field of sensing technology, and in particular to a control method, device and system. Background Technology

[0002] Currently, autonomous vehicles use multiple sensors to detect the external environment and fuse the perception data from these sensors through algorithms to achieve a more comprehensive perception and recognition of the environment or targets, thereby improving the external perception capabilities of the autonomous driving system. Typically, it is physically difficult to achieve good temporal and spatial coordination among multiple sensors; therefore, spatiotemporal fusion is mainly achieved through algorithms. Insufficient optimization of the perception fusion algorithm can easily lead to risks, thereby affecting the safety of autonomous driving.

[0003] Some solutions use a single sensor as a benchmark for multi-sensor collaboration. For example, when the turntable lidar turns to an angle, other sensors at the corresponding angle are activated for detection, thereby minimizing the time lag in obtaining sensing data from different sensors.

[0004] However, due to the different working principles of different sensors, it is impossible to achieve high-precision synchronization between different sensors, so the synchronous detection capability of different sensors is still limited. Summary of the Invention

[0005] This application provides a control method, apparatus, and system for enabling coordinated temporal and spatial detection of different sensing modules, thereby improving the accuracy of the corresponding fusion sensing algorithm.

[0006] In a first aspect, this application provides a control method, which can be implemented by a first control device. The first control device can be a device independent of the first sensing module and the second sensing module, or it can be a control device deployed in the same sensing device as the first sensing device, or it can be a control device integrated into the first sensing module. The embodiments of this application do not specifically limit the implementation of the first control device.

[0007] In specific implementation, the method may include: a first control device acquiring the operating parameters of the second sensing module; and controlling the first sensing module according to the operating parameters of the second sensing module to achieve scanning timing registration between the first sensing module and the second sensing module, wherein the second sensing module and the first sensing module share the same receiving optical component.

[0008] Using the above method, based on the spatial registration of the first and second sensing modules using the same receiving optical component, the first control device can adjust the working state of the first sensing module in a timely manner according to the dynamic changes in the working state of the second sensing module. In the event of a change in the working timing of the two sensing modules that makes them unable to be registered, timely adjustment and control can be used to match the rolling scans of the two sensing modules in time, thereby maintaining the spatiotemporal registration of different sensing data in a better possible way, thus improving the accuracy of the fusion sensing algorithm.

[0009] In conjunction with the first aspect, in one possible implementation, the method may further include: acquiring first indication information, the first indication information indicating a second scanning timing of the second sensing module; the step of controlling the first sensing module according to the operating parameters of the second sensing module to achieve scanning timing registration between the first sensing module and the second sensing module includes: controlling the first sensing module according to the operating parameters of the second sensing module and / or the first indication information to achieve first scanning timing registration between the first sensing module and the second scanning timing.

[0010] Using the above method, the first control device can also obtain the first instruction information to know the second scanning timing of the second sensing module, and then control the first sensing module in combination with the second scanning timing to achieve scanning timing registration between the first sensing module and the second sensing module.

[0011] In one example, acquiring the first indication information may include determining the first indication information based on the operating parameters of the second sensing module. Therefore, the first control device can determine (or derive) the second scan timing of the second sensing module by analyzing its operating parameters. In this example, even if the first and second sensing modules are from different manufacturers, the control device can still determine (or derive) the second scan timing of the second sensing module by analyzing its operating parameters; that is, it does not restrict whether the first and second sensing modules are from the same manufacturer.

[0012] In another example, acquiring the first indication information may include receiving the first indication information from a second control device of the second sensing module. Thus, the second control device can indicate the second scan timing of the second sensing module to the first control device, allowing the first control device to directly control the first sensing module based on the second scan timing, i.e., without the first control device needing to derive the second scan timing of the second sensing module. In an optional implementation of this example, the first and second sensing modules may be provided by the same manufacturer.

[0013] In conjunction with the first aspect, in one possible implementation, the second sensing module may include a camera module, and the operating parameters of the second sensing module include at least one of the following parameters of the camera module: exposure time, sensitivity, frame rate, and line scan start interval; or, the second sensing module may include a lidar module, and the operating parameters of the second sensing module include at least one of the following parameters of the lidar module: frame rate, number of shots, transmission power, interval per shot, line scan start interval, scan angle interval, field of view (FOV) area size or region of interest (ROI) location, and range.

[0014] Wherein, if the second sensing module is the camera module, in the same target scene, the exposure duration and line scan start interval of the camera module are positively correlated with the frame interval of the first sensing module; the photosensitivity and frame rate of the camera module are negatively correlated with the frame interval of the first sensing module. If the second sensing module is the lidar module, in the same target scene, one or more of the following parameters of the lidar module are positively correlated with the frame interval of the first sensing module: number of shots, interval per shot, line scan start interval, FOV area size, and range; one or more of the following parameters of the lidar module are negatively correlated with the frame interval of the first sensing module: frame rate and scan angle interval.

[0015] For example, the second sensing module is the camera module. Controlling the first sensing module according to the operating parameters of the second sensing module to achieve scanning timing registration between the first and second sensing modules includes at least one of the following: if the exposure time of the camera module increases and / or the photosensitivity decreases, controlling the frame interval of the first sensing module to increase to achieve scanning timing registration between the first and second sensing modules; if the line scan start interval of the camera module decreases, controlling the frame interval of the first sensing module to decrease to achieve scanning timing registration between the first and second sensing modules; if the frame rate of the camera module increases, controlling the frame interval of the first sensing module to decrease to achieve scanning timing registration between the first and second sensing modules.

[0016] The first sensing module is a lidar module. The frame interval of the first sensing module is positively correlated with one or more of the following parameters of the lidar module: number of shots, interval per shot, line scan start interval, FOV area size, and measurement range. The frame interval of the first sensing module is negatively correlated with one or more of the following parameters of the lidar module: frame rate and scan angle interval. In specific implementations, controlling the frame interval of the first sensing module to increase may include, for example, controlling one or more of the following parameters of the lidar module to increase: number of shots, interval per shot, line scan interval, FOV area size, and measurement range; or, controlling the frame interval of the first sensing module to decrease may include, for example, controlling one or more of the following parameters of the lidar module to increase: frame rate and scan angle interval.

[0017] For example, the second sensing module is the lidar module. Controlling the first sensing module according to the operating parameters of the second sensing module to achieve scanning timing registration between the first and second sensing modules includes at least one of the following: if one or more of the following parameters of the lidar module decrease, controlling the first sensing module to accelerate the single-frame scanning speed to achieve scanning timing registration between the first sensing module and the lidar module: number of shots, line scan start interval, FOV area size, and range; if the ROI position of the lidar module moves, changing the single-frame scan start time of the first sensing module to achieve scanning timing registration between the first sensing module and the lidar module; if the scanning angle interval of the lidar module increases, controlling the first sensing module to accelerate the single-frame scanning speed to achieve scanning timing registration between the first sensing module and the lidar module.

[0018] Wherein, the first sensing module is a camera module, and controlling the first sensing module to speed up the single-frame scanning speed may include: reducing the line scanning start interval or exposure time of the camera module; or increasing the frame rate or sensitivity of the camera module.

[0019] For example, the second sensing module is a LiDAR module and the first sensing module is a camera module. The single-frame point cloud detected by the LiDAR module includes at least two Regions of Interest (ROIs), and the single-frame image detected by the camera module includes at least two subframes corresponding to the at least two ROIs. If one or more of the following parameters associated with the first ROI among the at least two ROIs change accordingly: the scanning angle interval increases, the number of shots decreases, the line scan start interval decreases, and the measurement range decreases, the step of controlling the first sensing module according to the operating parameters of the second sensing module to achieve scanning timing registration between the first sensing module and the second sensing module includes: if the line scan start interval of the camera module is adjustable, controlling the camera module to accelerate the scanning speed of the subframes corresponding to the first ROI to achieve scanning timing registration between the first sensing module and the LiDAR module; or, if the line scan start interval of the camera module is not adjustable, controlling the camera module to pause scanning other subframes after scanning the subframes corresponding to the first ROI to achieve scanning timing registration between the camera module and the LiDAR module.

[0020] In conjunction with the first aspect, in one possible implementation, the camera module can be any of the following types: a visible light camera module, an infrared camera module, or a thermal imaging camera module.

[0021] Secondly, this application provides a control device, comprising: an acquisition unit for acquiring operating parameters of a second sensing module; and a control unit for controlling a first sensing module according to the operating parameters of the second sensing module, so as to achieve scanning timing registration between the first sensing module and the second sensing module, wherein the second sensing module and the first sensing module share the same receiving optical component.

[0022] In conjunction with the second aspect, in one possible implementation, the acquisition unit may further be used to: acquire first indication information, the first indication information indicating the second scanning timing of the second sensing module; the control unit is used to: control the first sensing module according to the operating parameters of the second sensing module and / or the first indication information, so as to achieve registration of the first scanning timing of the first sensing module with the second scanning timing.

[0023] In conjunction with the second aspect, in one possible implementation, the acquisition unit may be specifically used to: determine the first indication information based on the operating parameters of the second sensing module; or, receive the first indication information from the second control device of the second sensing module.

[0024] In conjunction with the second aspect, in one possible implementation, the second sensing module may include a camera module, and the operating parameters of the second sensing module include at least one of the following parameters of the camera module: exposure time, sensitivity, frame rate, and line scan start interval; or, the second sensing module may include a LiDAR module, and the operating parameters of the second sensing module include at least one of the following parameters of the LiDAR module: frame rate, number of shots, transmission power, interval per shot, line scan start interval, scan angle interval, field of view (FOV) area size or region of interest (ROI) position, and range.

[0025] Wherein, the second sensing module is the camera module, and in the same target scene, the exposure time and line scan start interval of the camera module are positively correlated with the frame interval of the first sensing module; the sensitivity and frame rate of the camera module are negatively correlated with the frame interval of the first sensing module. If the second sensing module is the lidar module, in the same target scene, one or more of the following parameters of the lidar module are positively correlated with the frame interval of the first sensing module: number of shots, interval per shot, line scan start interval, FOV area size, and range; one or more of the following parameters of the lidar module are negatively correlated with the frame interval of the first sensing module: frame rate and scan angle interval.

[0026] For example, the second sensing module is the camera module, and the control unit can specifically be used to perform at least one of the following: if the exposure time of the camera module increases and / or the photosensitivity decreases, control the frame interval of the first sensing module to increase, so as to achieve scanning timing registration between the first sensing module and the camera sensing module; if the line scan start interval of the camera module decreases, control the frame interval of the first sensing module to decrease, so as to achieve scanning timing registration between the first sensing module and the camera sensing module; if the frame rate of the camera module increases, control the frame interval of the first sensing module to decrease, so as to achieve scanning timing registration between the first sensing module and the camera sensing module.

[0027] The first sensing module is a lidar module. The frame interval of the first sensing module is positively correlated with one or more of the following parameters of the lidar module: number of shots, interval between shots, line scan start interval, FOV area size, and measurement range. The frame interval of the first sensing module is negatively correlated with one or more of the following parameters of the lidar module: frame rate and scan angle interval. In specific implementations, the control unit can control the frame interval of the first sensing module to increase, for example, by controlling the increase of one or more of the following parameters of the lidar module: number of shots, interval between shots, line scan interval, FOV area size, and measurement range; or, the control unit can control the frame interval of the first sensing module to decrease, for example, by controlling the increase of one or more of the following parameters of the lidar module: frame rate and scan angle interval.

[0028] For example, the second sensing module is the lidar module, and the control unit can specifically be used to perform at least one of the following: if one or more of the following parameters of the lidar module decrease, control the first sensing module to accelerate the single-frame scanning speed to achieve scanning timing registration between the first sensing module and the lidar module: number of shots, line scanning start interval, FOV area size, and range; if the ROI position of the lidar module moves, change the single-frame scanning start time of the first sensing module to achieve scanning timing registration between the first sensing module and the lidar module; if the scanning angle interval of the lidar module increases, control the first sensing module to accelerate the single-frame scanning speed to achieve scanning timing registration between the first sensing module and the lidar module.

[0029] The first sensing module is a camera module. The control unit controls the first sensing module to speed up the single-frame scanning speed, which may include: reducing the line scanning start interval or exposure time of the camera module; or increasing the frame rate or sensitivity of the camera module.

[0030] For example, the second sensing module is a lidar module and the first sensing module is a camera module. The single-frame point cloud detected by the lidar module includes at least two regions of interest (ROIs), and the single-frame image detected by the camera module includes at least two subframes corresponding to the at least two ROIs. If one or more of the following parameters associated with the first ROI among the at least two ROIs change accordingly: the scanning angle interval increases, the number of shots decreases, the line scan start interval decreases, and the range decreases, the control unit can specifically be used to: if the line scan start interval of the camera module is adjustable, control the camera module to accelerate the scanning speed of the subframe corresponding to the first ROI, so as to achieve scanning timing registration between the first sensing module and the lidar module; or, if the line scan start interval of the camera module is not adjustable, control the camera module to pause scanning other subframes after scanning the subframe corresponding to the first ROI, so as to achieve scanning timing registration between the camera module and the lidar module.

[0031] In conjunction with the second aspect, in one possible implementation, the first control device is integrated into the first sensing module; or, the first control device and the first sensing module belong to the same sensing device, and the control device is independent of the first sensing module; or, the first control device is independent of the sensing device to which the first sensing module belongs, and independent of the sensing device to which the second sensing module belongs.

[0032] In conjunction with the second aspect, in one possible implementation, the camera module is any of the following types: a visible light camera module, an infrared camera module, or a thermal imaging camera module.

[0033] Thirdly, embodiments of this application provide a control device, including at least one processor and an interface circuit, wherein the interface circuit is used to provide data or code instructions to the at least one processor, and the at least one processor is used to implement the method as described in the first aspect above and any possible design of the first aspect through logic circuits or executing code instructions.

[0034] Fourthly, embodiments of this application provide a detection system, including a control device, a first sensing module, and a second sensing module. The first sensing module and the second sensing module share the same receiving optical component. The control device is used to implement the method described in the first aspect and any possible design of the first aspect.

[0035] Fifthly, embodiments of this application provide a terminal device including a control device for implementing the second aspect and any possible design of the second aspect described above.

[0036] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing program code that, when executed on a computer, causes the computer to perform the method described in the first aspect and any possible implementation thereof.

[0037] In a seventh aspect, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the method described in the first aspect and any possible implementation thereof.

[0038] Eighthly, embodiments of this application provide a terminal device including units for implementing the method described in the first aspect and any possible design of the first aspect. For example, the terminal device includes, but is not limited to: intelligent transportation equipment (such as automobiles, ships, drones, trains, freight trucks, etc.), intelligent manufacturing equipment (such as robots, industrial equipment, intelligent logistics, intelligent factories, etc.), and intelligent terminals (mobile phones, computers, tablets, PDAs, desktop computers, headphones, speakers, wearable devices, in-vehicle equipment, etc.).

[0039] Based on the implementations provided in the above aspects, the embodiments of this application can be further combined to provide more implementations.

[0040] The technical effects that can be achieved by any possible implementation of any of the second to eighth aspects mentioned above can be described with reference to the technical effects that can be achieved by any possible implementation of the first aspect mentioned above, and the repetitions will not be discussed. Attached Figure Description

[0041] Figure 1 shows a schematic diagram of the system architecture of the detection system applicable to the embodiments of this application;

[0042] Figure 2A shows a schematic diagram of the structure of an example sensing module assembly according to an embodiment of this application;

[0043] Figure 2B shows an example diagram of the scanning areas of different sensing modules in an embodiment of this application;

[0044] Figures 3A-3E illustrate schematic diagrams of different system architecture examples of the detection system according to embodiments of this application;

[0045] Figures 3F-3G illustrate the principle of the control method according to an embodiment of this application;

[0046] Figure 4 shows a flowchart illustrating an example of a control method according to an embodiment of this application;

[0047] Figure 5 shows a flowchart illustrating another example of a control method according to an embodiment of this application;

[0048] Figure 6 shows a schematic diagram of an example control method according to an embodiment of this application;

[0049] Figure 7 shows a schematic diagram of another example control method according to an embodiment of this application;

[0050] Figure 8 shows a schematic diagram of the detection principle of different sensing modules in an embodiment of this application;

[0051] Figure 9 shows a schematic diagram of the scanning timing registration of different sensing modules in an embodiment of this application;

[0052] Figures 10-13 illustrate different examples of control methods in embodiments of this application. Detailed Implementation

[0053] To facilitate understanding, some terms used in the embodiments of this application will be explained below:

[0054] 1. LiDAR: A remote sensing technology that uses laser pulses to measure distance.

[0055] Typically, a lidar system includes a transmitter, a receiver, and an information processing unit. The transmitter emits laser pulses, which are reflected when they strike an object's surface. The receiver detects the reflected pulse signal and measures the time difference between the emission and reception of the pulse. The information processing unit calculates the distance between the lidar and the object based on the speed of light and the time difference.

[0056] In practical applications, lidar can not only measure distances but also construct three-dimensional structural information of target objects by analyzing information such as the energy, spectral amplitude, frequency, and phase of reflected light. By emitting multiple laser pulses and combining them with time range, scanning angle, and GPS location information, lidar can create environmental maps, providing crucial perception capabilities for autonomous vehicles and other applications.

[0057] The lidar in this embodiment can be a solid-state lidar, using electronic scanning to control the direction of the laser beam through technologies such as optical phased arrays or flash lidar, and employing line scanning or global scanning to obtain the three-dimensional point cloud data of the target. Line scanning can include row scanning or column scanning. Global scanning can be referred to as full-domain scanning, full-flash scanning, full-domain shutter scanning, or global shutter scanning, etc.

[0058] 2. Camera scanning principle: It is the process of capturing objects and converting them into electronic signals using the imaging function of a camera. When a target object is placed in front of the camera, the camera converts light into electronic signals through its photosensitive device. These signals are then converted into digital or analog signals depending on the camera's operating mode.

[0059] During the scanning process, the camera divides the image captured by the photosensitive device into multiple pixels. Each pixel represents the smallest unit in the image, possessing a specific color and brightness. In one implementation, the camera can be a rolling shutter camera, employing a line scan method during exposure, scanning each pixel in the image line by line from top to bottom and converting it into an electronic signal. Through continuous line scanning, the camera can acquire information about the entire image. In another implementation, the camera can be a global shutter camera, capable of capturing all pixels simultaneously during exposure, thereby simultaneously reading the image across the entire photosensitive element without requiring line-by-line scanning.

[0060] The LiDAR or cameras described above can be deployed independently on a vehicle to provide rich perception data for the vehicle's autonomous driving system or intelligent driver assistance system. The vehicle's autonomous driving system or intelligent driver assistance system can use algorithms to fuse perception data from different sensors (this algorithm can also be called a perception fusion algorithm) to achieve a more comprehensive perception and recognition of the environment or targets, improve the external perception capability of the autonomous driving system or intelligent driver assistance system, and thus achieve more precise autonomous driving control or intelligent driver assistance function control.

[0061] Typically, it is physically difficult for multiple sensors to coordinate well in time and space; spatiotemporal fusion is mainly achieved through algorithms. If the perception fusion algorithm is not optimized sufficiently, it can easily lead to risks, thereby affecting the safety of autonomous driving or intelligent assisted driving.

[0062] Some solutions use a single sensor as a baseline for multi-sensor collaboration. For example, when the turntable lidar rotates to a certain angle, other sensors at the corresponding angle are activated to detect, thereby minimizing the time lag in data acquisition from different sensors. However, because different sensors operate on different principles, and sensor control cannot penetrate to the internal timing control of the sensors, high-precision synchronization of their operation cannot be achieved. Therefore, the synchronous detection capability of different sensors remains limited.

[0063] To address the aforementioned problems, this application provides a control method, apparatus, and system for coordinating the temporal and spatial aspects of different sensing modules, thereby improving the accuracy of the corresponding fusion sensing algorithm. The method and apparatus are based on the same technical concept. Since the principles underlying the problems solved by the method and apparatus are similar, their implementations can be mutually referenced, and repeated details will not be elaborated upon. Furthermore, in the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between the embodiments are consistent and can be mutually referenced. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0064] The control scheme in this application embodiment can be applied to vehicle-to-everything (V2X), long-term evolution-vehicle (LTE-V), and vehicle-to-vehicle (V2V) communication. For example, it can be applied to vehicles with driving mobility functions, or other devices within vehicles with driving mobility functions. These other devices include, but are not limited to, vehicle-mounted terminals, vehicle-mounted controllers, vehicle-mounted modules, vehicle-mounted components, vehicle-mounted chips, vehicle-mounted units, vehicle-mounted radar, or vehicle-mounted cameras, and other sensors. Vehicles can implement the vehicle control method provided in this application through these vehicle-mounted terminals, controllers, modules, components, chips, units, radar, or cameras. Of course, the control scheme in this application embodiment can also be used in other intelligent terminals with mobility control functions besides vehicles, or installed in other intelligent terminals with mobility control functions besides vehicles, or installed in components of such intelligent terminals. These intelligent terminals can be intelligent transportation equipment, smart home devices, robots, etc. Examples include, but are not limited to, smart terminals or controllers, chips, radar or cameras, and other sensors and components within smart terminals.

[0065] It should be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0066] Furthermore, unless otherwise specified, the ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used to distinguish multiple objects and are not used to limit the priority or importance of multiple objects. For example, "first sensing module" and "second sensing module" are only used to distinguish different sensing modules, and do not indicate that the priority or importance of these two sensing modules is different.

[0067] To facilitate understanding, the following description is provided in conjunction with the accompanying drawings and embodiments.

[0068] Figure 1 shows a schematic diagram of the system architecture of the detection system applicable to the embodiments of this application. As shown in Figure 1, the system architecture may include a control device 11, a first sensing module 12, and a second sensing module 13.

[0069] The first sensing module 12 and the second sensing module 13 can be used to detect the external environment and obtain sensing data. In specific implementation, the first sensing module and the second sensing module can adopt the same detection principle, and can be sensing modules of the same type or specification, or they can be sensing modules of different types or specifications.

[0070] For example, both the first and second sensing modules can use the line scanning method described above to detect the external environment and obtain corresponding sensing data. Alternatively, both the first and second sensing modules can use the full-area scanning method described above to detect the external environment and obtain corresponding sensing data.

[0071] In practical implementation, the first sensing module and the second sensing module can be the same camera module. Alternatively, for example, the first sensing module and the second sensing module can be different types or specifications of camera modules. Alternatively, for example, the second sensing module and the second sensing module can be the same LiDAR module. Alternatively, for example, the first sensing module and the second sensing module can be different types or specifications of LiDAR modules. Alternatively, for example, the first sensing module can be a camera module and the second sensing module can be a LiDAR module. Alternatively, for example, the first sensing module can be a LiDAR module and the second sensing module can be a camera module.

[0072] It should be understood that this is merely an illustrative description of the implementation methods of the sensing modules in the system architecture and does not constitute any limitation. In specific embodiments, if the first or second sensing module is a camera module, it can be any of the following types: visible light camera module, infrared camera module, or thermal imaging camera module. Alternatively, the first or second sensing module can also be other types of radar modules, such as any of the following types: millimeter-wave radar module, ultrasonic radar module, or infrared radar module. Alternatively, the first or second sensing module can also be other types of sensing modules besides camera modules or radar modules. In other embodiments, the system architecture may also include a greater number of sensing modules, such as a third sensing module, a fourth sensing module, etc. This application does not specifically limit the number of sensing modules or the type or specifications of each sensing module.

[0073] In order to achieve spatial registration between different sensing modules, in one implementation, the spatial structure of each sensing module can be configured, for example, the first sensing module and the second sensing module can share the same receiving optical component.

[0074] Taking the first sensing module and the second sensing module as a combination of a camera module and a lidar module as an example, the camera module and the lidar module can adopt a coaxial receiving architecture or a non-coaxial (e.g., a side-axis) receiving architecture.

[0075] As shown in the architecture of Figure 2A(a), the transmitter (Tx), receiver (Rx), and camera module of the LiDAR module share the same lens to detect target objects in the external environment, which is a coaxial receiving architecture. As shown in the architecture of Figure 2A(b), the transmitter of the LiDAR module uses an independent transmitting lens, while the receiver and camera module share the same receiving lens to detect target objects, which is a rangefinder receiving architecture. Based on the detection method using the same receiving lens in architectures (a) and (b), the scanning area of ​​the camera module and the field of view (FOV) of the LiDAR module overlap, achieving spatial registration between the LiDAR module and the camera module. For example, as shown in Figure 2B(a), the FOV of the LiDAR module is within the scanning area of ​​the camera module, and this FOV is the overlapping detection area of ​​the camera module and the LiDAR module. Or, as shown in Figure 2B(b), the scanning area of ​​the camera module is within the FOV of the LiDAR module, and this scanning area of ​​the camera module is the overlapping detection area of ​​the camera module and the LiDAR module. Or, for example, as shown in Figure 2B(c), the scanning area of ​​the camera module completely overlaps with the FOV of the lidar module.

[0076] Based on the spatial registration of the first and second sensing modules, the control device 11 in Figure 1 can be used to register the working timing (e.g., scanning timing) between the first and second sensing modules, thereby minimizing the time difference in the acquisition of sensing data by the first and second sensing modules and improving their synchronous detection capability. For example, in a combination of a camera module and a LiDAR module for the first and second sensing modules, after spatiotemporal registration, the 3D point cloud data obtained by the LiDAR module and the 2D image data obtained by the camera module can be cross-checked, improving the reliability of sensor detection. Simultaneously, a color depth image (RGB-Depth, RGBD) can be generated based on the 3D point cloud data and the 2D image data, providing rich color and depth information to help the autonomous driving system better perceive and understand the surrounding environment, thereby improving driving safety and reliability.

[0077] In practical implementation, the control device 11 in Figure 1 can be integrated into the first sensing module 12 or the second sensing module 13. The first sensing module 12 and the second sensing module 13 can belong to the same sensing device or different sensing devices. For ease of distinction, the control device controlling the first sensing module can be referred to as the first control device, and the control device controlling the second sensing module can be referred to as the second control device. The sensing device to which the first sensing module belongs can be referred to as the first sensing device, and the sensing device to which the second sensing module belongs can be referred to as the second sensing device. As shown in Figure 3A, the architecture of the detection system can include sensing devices, which can include a first sensing module and a second sensing module. The first sensing module can include a first control device, and the second sensing module can include a second control device. Alternatively, as shown in Figure 3B, the architecture of the detection system can include a first sensing device and a second sensing device. The first sensing device can include a first sensing module, which can include a first control device, and the second sensing device can include a second sensing module, which can include a second control device.

[0078] Alternatively, the control device 11 in Figure 1 can be a device independent of the first sensing module 12 and the second sensing module 13, and the control device 11 belongs to the same sensing device as the first sensing module 12 and the second sensing module 13. As shown in Figure 3C, the architecture of the detection system can include a sensing device, which can include a first sensing module, a control device, and a second sensing module. The control device can control both the first sensing module and the second sensing module.

[0079] Alternatively, the control device 11 in Figure 1 can be deployed in the sensing device where the first sensing module 12 is located, and the control device 11 is independent of the first sensing module 12. Alternatively, the control device 11 can be deployed in the sensing device where the second sensing module 13 is located, and the control device 11 is independent of the second sensing module 13. As shown in Figure 3D, the architecture of the detection system may include a first sensing device and a second sensing device. The first sensing device may include a first sensing module and a first control device, and the second sensing device may include a second control device and a second sensing module.

[0080] Alternatively, different sensing devices in the detection system architecture can adopt different structures. As shown in Figure 3E, the detection system architecture may include a first sensing device and a second sensing device. The first sensing device may include a first sensing module, and the first sensing module may include a first control device. The second sensing device may include a second control device and a second sensing module. Alternatively, the structures of the first sensing device and the second sensing device can be interchanged, and this application embodiment does not specifically limit this.

[0081] In any of the architectures shown in Figures 3A-3E, the first sensing module can be used to detect the external environment and obtain first sensing data. During the detection process, the first control device can be used to control the operating state of the first sensing module to perform spatiotemporal registration between the first sensing module and the second sensing module. The second sensing module can be used to detect the external environment and obtain second sensing data. During the detection process, the second control device can be used to control the operating state of the second sensing module to perform spatiotemporal registration between the second sensing module and the first sensing module. After subsequent processing, the first sensing data and / or the second sensing data can realize relevant functions.

[0082] For example, as shown in Figure 3F, in a vehicle, the detection system can adopt the architecture shown in Figure 3D. The autonomous driving system or intelligent driver assistance system can perform fusion processing and analysis based on the first perception data provided by the first sensing device and the second perception data provided by the second sensing device, thereby realizing the vehicle's autonomous driving function or intelligent driver assistance function. Alternatively, as shown in Figure 3F, in a vehicle, the detection system can adopt the architecture shown in Figure 3C. The autonomous driving system or intelligent driver assistance system can perform fusion processing and analysis based on the first perception data provided by the first sensing module and the second perception data provided by the second sensing module, thereby realizing the vehicle's autonomous driving function or intelligent driver assistance function.

[0083] It should be understood that Figures 3A-3G are merely illustrative examples of the implementation methods of each sensing module and its corresponding control device, and do not constitute any limitation. Different dashed arrows represent different control logics and do not constitute any limitation on the information interaction methods or content.

[0084] When implementing the control method of this application embodiment, the method steps shown in FIG4 can be executed by the first control device described above:

[0085] S410: The first control device acquires the operating parameters of the second sensing module. For example, the first control device can acquire the operating parameters of the second sensing module from the second control device.

[0086] S420: The first control device controls the first sensing module according to the working parameters of the second sensing module to achieve scanning timing registration between the first sensing module and the second sensing module.

[0087] Alternatively, the method steps shown in Figure 5 can be executed by the second control device described above:

[0088] S510: The second control device acquires the operating parameters of the first sensing module. For example, the second control device can acquire the operating parameters of the first sensing module from the first control device.

[0089] S520: The second control device controls the second sensing module according to the working parameters of the first sensing module, so as to realize the scanning timing registration between the second sensing module and the first sensing module.

[0090] In the method shown in Figure 4, taking the second sensing module as a camera module and the first sensing module as a lidar module as an example, as shown in Figure 6, the operating parameters of the second sensing module may include at least one of the following parameters of the camera module: exposure time, sensitivity, frame rate, and line scan start interval. When implementing S410, the first control device can acquire at least one of the above parameters of the camera module and implement S420 to control the lidar module, thereby achieving scanning timing registration between the lidar module and the camera module.

[0091] Alternatively, taking the second sensing module as a LiDAR module and the first sensing module as a camera module as an example, as shown in Figure 7, the operating parameters of the second sensing module can include at least one of the following parameters of the LiDAR module: frame rate, number of shots, transmission power, interval per shot, line scan start interval, scan angle interval, FOV area size or region of interest (ROI) location, and range. When implementing S410, the first control device can acquire at least one of the following parameters of the LiDAR module and implement S420 to control the camera module to achieve scanning timing registration between the camera module and the LiDAR module. The meaning of each operating parameter will be explained in conjunction with examples below, and will not be elaborated here.

[0092] In an optional implementation, before S420, the first control device may also acquire first indication information, which may, for example, indicate the second scanning timing of the second sensing module. When S420 is implemented, the first control device can control the first sensing module according to the operating parameters of the second sensing module and / or the first indication information to achieve registration between the first and second scanning timings of the first sensing module. Similarly, before S420, the second control device may also acquire second indication information, which may indicate the first scanning timing of the first sensing module. When S420 is implemented, the second control device can control the second sensing module according to the operating parameters of the first sensing module and / or the second indication information to achieve registration between the second and first scanning timings of the second sensing module.

[0093] Taking the method executed by the first control device as an example, in a specific implementation, in one example, the first control device can determine (or deduce) the second scanning timing of the second sensing module by analyzing the operating parameters of the second sensing module. For example, the step of the first control device acquiring the first indication information may specifically include: determining the first indication information based on the operating parameters of the second sensing module. Thus, in this example, even if the first and second sensing modules are provided by different manufacturers, the first control device can deduce the operating state of the second sensing module by analyzing its operating parameters. During continuous scanning detection, if the operating state of the second sensing module changes, the first control device can adaptively control the first sensing module to achieve registration between the first scanning timing of the first sensing module and the second scanning timing of the second sensing module at the current moment. This example does not limit whether the first and second sensing modules are provided by the same manufacturer.

[0094] In another example, the second scan timing of the second sensing module can be determined (or derived) by the analysis of the operating parameters of the second sensing module by the second control device associated with the second sensing module. The second control device can instruct the second scan timing of the second sensing module to the first control device, so that the first control device can directly control the first sensing module based on the second scan timing, i.e., without the first control device deriving the second scan timing of the second sensing module. For example, the second control device can send first instruction information to the first control device. The step of the first control device obtaining the first instruction information can include: receiving the first instruction information from the second control device of the second sensing module. Thus, by interacting with the first instruction information, regardless of whether the first sensing device and the second sensing device belong to the same equipment manufacturer, they can know the other party's operating status based on the received first instruction information. If the other party's operating status changes, the sensing modules within the same sensing device can be adaptively controlled to achieve scan timing registration of different sensing modules.

[0095] It should be understood that, in the embodiments of this application, the transmission and reception of operating parameters and / or first indication information may, for example, involve the second sensing module sending its own operating parameters and / or first indication information to the first control device through the second control device, and correspondingly, the first control device receiving the operating parameters and / or first indication information from the second control device. Alternatively, for example, the second sensing module itself may send its own operating parameters and / or first indication information to the first control device, and correspondingly, the first control device receiving the operating parameters and / or first indication information from the second sensing module. Similarly, in the method shown in FIG5, the first sensing module or the first control device associated with the first sensing module may send the operating parameters and / or first indication information of the first sensing module to the second control device. The embodiments of this application do not specifically limit the interaction method of operating parameters and / or first indication information. The operating parameters and / or first indication information of different sensing modules may be transmitted and received in real time, periodically, or when the operating state of the corresponding sensing module changes, or when a relevant request is received. The embodiments of this application do not specifically limit the timing of the transmission and reception of operating parameters and / or first indication information.

[0096] For example, taking the deployment of the first sensing device and the second sensing device in the same vehicle as an example, the first sensing device and the second sensing device can detect the external environment of the vehicle after the autonomous driving function or intelligent assisted driving function of the vehicle is activated. During the detection process, the second sensing device can send the working parameters of the second sensing module to the first control device in the first sensing device in real time or periodically, or send the working parameters of the second sensing module and the first indication information to the first control device in the first sensing device in real time or periodically, so that the first control device can adaptively control the first sensing module in the first sensing device according to the working parameters of the second sensing module and / or the first indication information, so as to realize the scanning timing registration of different sensing modules. Alternatively, during the detection process, the first sensing device can send the operating parameters of the first sensing module to the second control device in the second sensing device in real time or periodically, or send the operating parameters of the first sensing module and the second indication information to the second control device in the second sensing device in real time or periodically, so that the second control device can adaptively control the second sensing module in the second sensing device according to the operating parameters of the first sensing module and / or the second indication information, so as to achieve scanning timing registration of different sensing modules.

[0097] In optional embodiments, the first and second sensing devices can also interact with each other via a request and response mechanism, exchanging operating parameters and / or corresponding indication information. For example, during the process of the first sensing device detecting the external environment of the vehicle, it can send a first request to the second sensing device to request the acquisition of operating parameters and / or scan timing information of the second sensing module in the second sensing device. The second control device in the second sensing device can send a first response to the first sensing device based on the received first request. This first response may include, for example, the operating parameters of the second sensing module and / or first indication information, which indicates the scan timing of the second sensing module. Accordingly, the first control device in the first sensing device can adaptively control the first sensing module based on some or all of the information in the received first response to achieve scan timing registration of different sensing modules. Alternatively, for example, during the process of the second sensing device detecting the external environment of the vehicle, it can send a second request to the first sensing device to request the acquisition of operating parameters and / or scan timing information of the first sensing module. The first control device in the first sensing device can send a second response message to the second sensing device based on the received second request. This second response message may include, for example, the operating parameters of the first sensing module and / or second indication information, which may indicate the scanning timing of the first sensing module. Correspondingly, the second control device in the second sensing device can adaptively control the second sensing module based on some or all of the information in the received second response message to achieve scanning timing registration of different sensing modules. This application embodiment does not specifically limit the interaction method of the operating parameters and / or the first indication information.

[0098] To facilitate understanding, the scanning timing registration principle between different sensing modules is explained below with examples, along with the specific implementation details of the method shown in Figure 4. The implementation details of the method shown in Figure 5 are similar to those of the method shown in Figure 4, and they can be referred to each other; they will not be differentiated or elaborated upon further below.

[0099] Taking the first sensing module and the second sensor module as a combination of a camera module and a lidar module as an example, since the camera module adopts a passive detection sensing method, if the camera module uses a line scanning method and obtains each frame of image data by scanning line by line, then based on the working timing (t), the reception time of each line of data of the camera module can overlap, as shown in Figure 8(a). Here, the exposure duration, also called the exposure time, is the duration of light sensitivity for each row of pixels, usually measured in microseconds (μs). The line scan start interval is the time difference between the start times of two consecutive line scans in a single frame detection. Since the reception time of the camera's line data can overlap, the single frame detection duration of the camera is less than n times the exposure duration, where n is the number of rows and is an integer greater than 1.

[0100] The lidar module employs an active detection sensing method. If the lidar module also uses line scanning to obtain point cloud data for each frame through line-by-line scanning, then each line of detection by the lidar module requires transmitting a detection signal and receiving an echo signal before starting the next line of detection. Therefore, the reception time of each line of data in the lidar module does not overlap, as shown in Figure 8(b). The single-line detection duration is the time difference between the transmitted and received signals in one line of detection. The line scan start interval is the time difference between the start times of two consecutive line scans. Since the reception times of the lidar's line data do not overlap, the line scan start interval is almost equal to the single-line detection duration, and the single-frame detection duration of the lidar is almost equal to m times the single-line detection duration, where m is the number of lines and is an integer greater than 1. It should be understood that the values ​​of m and n are independent; they can be the same or different.

[0101] To achieve timing alignment between the camera module and the LiDAR module, in one example, the timing of a single-frame scan by the camera module can be controlled to be as consistent as possible with that of the LiDAR module. For example, as shown in Figure 9, each scan line of the LiDAR module can be set to maintain spatiotemporal alignment (or alignment according to a predetermined reference point) with the corresponding scan line of the camera module. For instance, the midpoint of the first scan line of the LiDAR module can be aligned with the midpoint of the first scan line of the camera module, the midpoint of the second scan line of the LiDAR module with the midpoint of the fifth scan line of the camera module, the midpoint of the third scan line of the LiDAR module with the midpoint of the ninth scan line of the camera module, and so on. This minimizes the time difference between the LiDAR module acquiring a single frame of point cloud data and the camera module acquiring a single frame of image data.

[0102] In practical implementation, the initial scanning timing of the LiDAR module and the camera module can be registered through design. For example, by designing the same frame rate and frame interval for the LiDAR module and the camera module, the initial scanning timing of the LiDAR module and the camera module can be registered. The frame interval and frame rate are reciprocals of each other; the initial frame rate could be, for example, 10 Hz, and the initial frame interval could be, for example, 0.1 seconds.

[0103] Typically, camera modules can proactively and automatically adjust their operating parameters according to changes in the scene. For example, the exposure parameters of a camera module (such as exposure time and / or ISO) can respond to environmental changes to obtain properly exposed image data. Changes in the exposure parameters of a camera module may lead to changes in the module's operating state, such as changes in the frame interval. Or, for example, if a camera module is deployed on a vehicle, its frame rate and other parameters need to adapt to changes in vehicle speed to match the vehicle's autonomous driving system or intelligent driver assistance system's perception speed of changes in the surrounding environment, thereby improving driving safety and comfort.

[0104] Changes in the operating state of the camera module may cause its scanning timing to differ from the initial scanning timing. In this case, the first control device can implement the control method of this application embodiment to adaptively control the LiDAR module according to the operating parameters of the camera module, so that the LiDAR module changes its own operating state to re-register with the changed scanning timing of the camera module, thereby maintaining the obtained point cloud data and image data to achieve optimal spatiotemporal registration as much as possible.

[0105] Similarly, if the LiDAR module also possesses the ability to proactively adjust its operating parameters according to scene changes, such as the LiDAR module's autofocus capability or subframe switching capability, then changes in the LiDAR module's operating state may cause its scanning timing to differ from the initial scanning timing. In this case, the first control device can also implement the control method of this application embodiment to adaptively control the camera module based on the LiDAR module's operating parameters, causing the camera module to change its operating state to re-register with the LiDAR module's changed scanning timing, thereby maintaining the acquired point cloud data and image data to achieve optimal spatiotemporal registration as much as possible.

[0106] Taking the operating parameters of a camera module, including at least one of the following parameters, as an example: exposure time, sensitivity, frame rate, and line scan start interval; and taking the operating parameters of a LiDAR module, including at least one of the following parameters, as an example: frame rate, number of shots, transmission power, interval per shot, line scan start interval, scan angle interval, scan area size, and range, in different scenarios, the first control device can control another sensor module based on one or more operating parameters of one sensor module to achieve re-registration of the scanning timing of the two sensor modules. The combinations of different operating parameter variations and their possible applicable scenarios are shown in Table 1 below.

[0107] Table 1

[0108] In Table 1, the symbols "+" indicate an increase, "-" indicate a decrease, and "O" indicate no change. The camera module and the LiDAR module use the same scanning principle; for example, both may use line scanning or full-area scanning.

[0109] The meanings of some parameters in Table 1 are supplemented as follows:

[0110] The number of laser radar shots refers to the number of laser beams fired in the same detection cycle.

[0111] The interval between each shot of a lidar is the time interval between multiple shots in the same detection.

[0112] The scanning angle interval of a lidar is the angular interval between two adjacent laser scanning points, usually expressed in degrees (°), also known as angular resolution.

[0113] The range of a lidar is: the maximum distance at which the lidar can measure distances.

[0114] For ease of understanding, the control method of the embodiments of this application is illustrated below based on Table 1.

[0115] Example 1: The control device controls the LiDAR module according to the changes in the exposure parameters of the camera module, so as to achieve the scanning timing registration between the LiDAR module and the camera module.

[0116] In this embodiment, the exposure parameters of the camera module may include exposure time and / or ISO sensitivity. When shooting in low-light environments, to obtain sufficient light, the ISO sensitivity of the camera module is usually adjusted and the exposure time is extended, or a large aperture is used to increase the amount of light entering the camera. Conversely, when shooting in well-lit environments, to prevent overexposure, the ISO sensitivity is usually reduced and the exposure time is shortened, or a small aperture is used to reduce the amount of light entering the camera.

[0117] In Example 1, if the second sensing module is a camera module and the first sensing module is a LiDAR module, the exposure time of the camera module can be positively correlated with the frame interval of the LiDAR module, and the photosensitivity of the camera module can be negatively correlated with the frame interval of the LiDAR module. During the continuous detection process of the camera module and the LiDAR module, as the scene / environment changes, if the exposure time of the camera module increases and / or the photosensitivity decreases, the first control device needs to control the frame interval of the LiDAR module to increase in order to achieve scanning timing registration between the LiDAR module and the camera module. If the exposure time of the camera module decreases and / or the photosensitivity increases, the first control device needs to control the frame interval of the LiDAR module to decrease in order to achieve scanning timing registration between the LiDAR module and the camera module.

[0118] As shown in Figure 10, continuous frame detection is performed from left to right based on time t. When the second sensing module is a camera module, the frame interval is the detection time difference between two consecutive frames of image data. When the first sensing module is a LiDAR module, the frame interval is the detection time difference between two consecutive frames of point cloud data. Taking a continuous 5-frame detection from left to right as an example, during the first 1-3 frames, in a well-lit scene, the camera module can detect a single frame of image data with an exposure time of 1. When the scene / environment becomes darker, during the 4th frame detection, in order to obtain properly exposed image data, the camera module may automatically increase the exposure time or may also increase the ISO. For example, increasing the exposure time could be from a single frame exposure time of 1 to a single frame exposure time of 2. At this time, the first control device can, for example, adjust the line scanning start time of the current frame (the 4th frame) of the lidar module to increase the frame interval of the 3rd frame of the lidar accordingly, for example, increase frame interval 1 to frame interval 2, so that the lidar module actively aligns with the spatiotemporal alignment point of the exposure time of the camera module, ensuring that the scanning timing of the current frame of the lidar module is registered with the scanning timing of the current frame of the camera module, thereby achieving temporal and spatial alignment between the two.

[0119] It should be understood that in Example 1, the frame interval of the LiDAR module can be increased by adjusting the start time of the line scan of the current frame, or by controlling the LiDAR to pause scanning for a first duration, so that the scanning timing of the LiDAR module is re-registered with the scanning timing of the camera module. The first duration is the time difference between the frame interval of the current frame of the LiDAR module and the frame interval of the previous frame. Thus, when the exposure time of the camera module increases, the frame interval of the LiDAR module is increased by adaptively controlling it to ensure that each scan line of the LiDAR module and the corresponding scan line of the camera module are always spatiotemporally aligned (aligned according to a certain agreed reference point), thereby re-registering the scanning timing of the LiDAR module with the scanning timing of the camera module.

[0120] For example, if the initial frame rate of both the camera module and the LiDAR module is 10 Hz and the frame interval is 0.1 seconds, when the scene / environment in Example 1 becomes darker, if the exposure time of the camera module increases by 0.02 seconds (the time difference between exposure time 2 and exposure time 1), in order to increase the frame interval of the LiDAR module, the control device can control the current frame point cloud data of the LiDAR module to be delayed by 0.01 seconds, that is, the scanning start time is delayed by 0.01 seconds, so as to realize the scanning timing re-registration of the LiDAR module and the camera module.

[0121] Therefore, in Example 1 above, when the first and second sensing modules use a combination of a camera module and a lidar module, the camera module's exposure parameters will change in response to environmental changes, causing the camera module's working state to change. The control device can control the lidar module's line scanning start time according to the changes in the camera module's exposure parameters, thereby adaptively changing the lidar module's frame interval so that the lidar module's scanning sequence can be re-registered with the camera module's scanning sequence.

[0122] It should be understood that Example 1 above is merely an example of changes in the operating parameters of the sensing module caused by ambient light intensity and does not constitute any limitation. In an optional embodiment, under low light conditions, the first control device may also reduce the emission power of the LiDAR module according to the exposure parameters of the camera module to reduce power consumption. Under strong light conditions, the first control device may also increase the emission power of the LiDAR module according to the exposure parameters of the camera module to enhance the distance measurement capability of the LiDAR module. In an optional embodiment, under low light conditions, the first control device may also reduce the frame rate of the camera module by decreasing at least one of the frame rate, number of shots, or emission power of the LiDAR module, which will not be elaborated further here.

[0123] Example 2: The control device controls the LiDAR module according to the changes in the line scanning start interval or frame rate of the camera module, so as to achieve scanning timing registration between the LiDAR module and the camera module.

[0124] For example, when a camera module is deployed on a vehicle, its frame rate, line scan start interval, exposure parameters, and other parameters need to be adapted to changes in vehicle speed. By changing the working state of the camera module, the vehicle's autonomous driving system or intelligent driver assistance system can be adjusted to perceive changes in the surrounding environment, thereby improving driving safety and comfort.

[0125] Specifically, as vehicle speed increases, increasing the frame rate of the camera module helps the vehicle's autonomous driving system or intelligent driver assistance system perceive changes in the surrounding environment more quickly, thereby improving driving safety and comfort. Conversely, when vehicle speed decreases, the frame rate needs to be adaptively reduced to slow down the perception speed of the vehicle's autonomous driving system or intelligent driver assistance system.

[0126] When the frame rate of the camera module increases, the camera module may also optionally increase its sensitivity and shorten the exposure time per frame. Therefore, the detection time per frame of the camera module may be significantly shortened. If the LiDAR module is a solid-state LiDAR and a corresponding reduction in the detection time per frame is required, this can be achieved by controlling one or more of the following parameters of the LiDAR module: number of shots, scanning angle interval, FOV area size, and range.

[0127] With time t represented by the horizontal axis, FOV represented by the vertical axis, and the slope representing the line scanning speed, as shown in Figure 11(a), solid lines and solid boxes of different thicknesses are used. Under the condition of scanning timing registration of the LiDAR module (represented as L) and the camera module (represented as C), the scanning speeds of the two are basically the same. For example, the FOV size of the LiDAR module is smaller than the scanning area size of the camera module, and the single frame duration of the overlapping area is equal to the single frame scanning duration of the LiDAR module.

[0128] As the vehicle speed increases, the frame rate of the camera module adaptively increases, and the line scanning speed increases accordingly. As shown in Figure 11(b), the slope of the dashed and solid diagonal lines of different thicknesses increases due to the increased line scanning speed (faster scanning), resulting in a delayed start and earlier end of C-detection. To align the scanning timing of the LiDAR module with that of the camera module, the control device can perform at least one of the following controls: reduce the number of LiDAR module shots, or reduce the angular resolution (scanning angle interval) of the LiDAR module, or reduce the FOV size of the LiDAR module, to speed up scanning or reduce the frame interval of the LiDAR module. For example, this results in a delayed start and earlier end of L-detection, a shorter single-frame duration in the overlapping area of ​​the camera module and the LiDAR module, and a smaller FOV size.

[0129] Reducing the number of laser radar modules will reduce the laser radar's range finding capability, and reducing the resolution of the laser radar modules will reduce the laser radar's target detection capability. Therefore, in some optional embodiments, the first control device may also choose to perform the following operations: reduce the FOV of the laser radar module, reduce the number of laser radar modules, and increase the transmission power to improve the range finding capability.

[0130] In some alternative implementations, after reducing the FOV size, the first control device may also choose to adjust the position of the ROI of the LiDAR so that the LiDAR module can detect areas that are more important to the autonomous driving system or intelligent driving system.

[0131] Therefore, through Example 2 above, in the combination of camera module and lidar module, when the vehicle speed increases and causes changes in the frame rate or exposure parameters of the camera module, the working state of the camera module will change, such as the change in the duration of a single frame. The first control device can selectively control the number of shots, the emission power, the resolution, or the FOV size of the lidar module, thereby adaptively changing the frame interval of the lidar module so that the scanning sequence of the lidar module and the scanning sequence of the camera module are re-registered.

[0132] Example 3: The control device controls the camera module according to the changes in the FOV size and / or ROI position of the LiDAR module, so as to achieve the scanning timing registration between the camera module and the LiDAR module.

[0133] For example, some LiDAR modules have focusing capabilities, enabling more intelligent detection by adjusting the scanning sequence to focus on scanning the area of ​​interest. When the ROI of the LiDAR module changes, the camera module also needs to adjust its scanning sequence to re-register with the LiDAR module's scanning sequence.

[0134] As shown in Figure 12(a), with solid lines of different thicknesses, when the scanning timing of the LiDAR module (represented as L) and the camera module (represented as C) is registered, their scanning speeds are basically the same. For example, the FOV size of the LiDAR module is smaller than the scanning area size of the camera module, and the single-frame duration of the overlapping area is equal to the single-frame scanning duration of the LiDAR module.

[0135] If the FOV of the LiDAR module shrinks laterally and / or vertically, and the ROI position shifts, it will not only cause changes in the single-frame scanning speed of the LiDAR module, but also changes in the single-frame timing, such as a delay in the single-frame scan start time. As shown by the dashed and solid diagonal lines of different thicknesses in Figure 12(b), when the LiDAR module shrinks its FOV and focuses on a certain point (i.e., the ROI position changes), while other operating parameters remain unchanged, the single-frame scanning time of the LiDAR module decreases accordingly (i.e., the single-frame scanning speed increases), and the number of scan lines also decreases. To register with the LiDAR module, the camera module needs to increase the single-frame scanning speed and adjust the single-frame scan start time to re-register with the scanning timing of the LiDAR module. Increasing the single-frame scanning speed can be achieved, for example, by reducing one or more of the following parameters of the LiDAR module: number of shots, line scan start interval, FOV area size, and range.

[0136] Therefore, through Example 3 above, in the combination of camera module and lidar module, when lidar module refocuses, the working state of lidar module will change, such as changes in single-frame scanning speed and single-frame scanning start time. The control device can adaptively adjust the single-frame scanning speed and single-frame scanning start time of camera module so that the scanning timing of lidar module is registered with the scanning timing of camera module.

[0137] Example 4: The control device controls the camera module based on changes in the intra-frame scanning resolution of the LiDAR module, so as to achieve scanning timing registration between the camera module and the LiDAR module.

[0138] For example, when some LiDAR modules switch their operating parameters within a frame (called subframe switching), such as when the scanning angle interval (or resolution) changes within a single frame, the camera module also needs to change its scanning timing to re-register with the scanning timing of the LiDAR module.

[0139] As shown in Figure 13(a), with solid lines of different thicknesses, when the scanning timing of the LiDAR module (represented as L) and the camera module (represented as C) is registered, their scanning speeds are basically the same. For example, the FOV size of the LiDAR module is smaller than the scanning area size of the camera module, and the single-frame duration of the overlapping area is equal to the single-frame scanning duration of the LiDAR module.

[0140] The single-frame point cloud detected by the lidar module may include at least two regions of interest (ROIs), and the single-frame image detected by the camera module includes at least two subframes corresponding to each of the at least two ROIs. If the scanning resolution or other parameters of the lidar module change within a single frame, the camera module will also change accordingly, so that the camera module is always spatiotemporally registered with the lidar module.

[0141] For example, if one or more of the following parameters associated with the first ROI in at least two ROIs of the LiDAR module change accordingly: increased scanning angle interval, decreased number of scans, decreased line scan start interval, and decreased range, then the first control device needs to control the camera module to accelerate the scanning speed of the subframe corresponding to the first ROI, when the line scan start interval of the camera module is adjustable, in order to achieve scanning timing registration between the first sensing module and the LiDAR module. As shown in Figure 13(b), the dashed and solid diagonal lines of different thicknesses maintain the original scanning speed in the subframe corresponding to the high-resolution ROI region, and accelerate the scanning speed in the subframe corresponding to the low-resolution ROI region. Alternatively, the first control device needs to control the camera module to pause scanning other subframes after scanning the subframe corresponding to the first ROI, when the line scan start interval of the camera module is not adjustable, in order to achieve scanning timing registration between the camera module and the LiDAR module. As shown in Figure 13(c), the solid diagonal lines of different thicknesses pause scanning after the LiDAR module rapidly scans the subframe corresponding to the low-resolution ROI region, to wait for the camera module.

[0142] Therefore, through Example 4 above, when the working parameters of the LiDAR module change within a single frame (referred to as frame-level change), the first control device adaptively adjusts the working parameters of the camera module within a single frame or controls the working state of the camera module within a single frame, so that the camera module is always spatiotemporally registered with the LiDAR module, so that the camera module always outputs the best possible perception result with the LiDAR module.

[0143] It should be understood that the above examples are merely illustrations of how the working states of different sensing modules change and do not constitute any limitation. In specific implementations, the re-registration of the scanning timing of different sensing modules can be achieved through any possible control method, which will not be elaborated here.

[0144] It should be understood that the control method provided in this application can also be extended to any information system that requires accurate detection of objects. It should be understood that all technical solutions that use the control scheme provided in this application to achieve accurate detection are within the scope of protection of this application, and will not be listed individually here.

[0145] According to the control scheme provided in the embodiments of this application, this application also provides a control device, including at least one processor and an interface circuit. The interface circuit is used to provide data or code instructions to at least one processor, and the at least one processor is used to implement the method executed by the control device (including the first control device or the second control device) through logic circuits or executing code instructions.

[0146] According to the control scheme provided in the embodiments of this application, this application also provides a detection system, including a control device, a first sensing module and a second sensing module, wherein the first sensing module and the second sensing module share the same receiving optical component, and the control device is used to implement the method performed by the control device (including the first control device or the second control device) described above.

[0147] In one possible design, the control device can be integrated into the first sensing module, or the control device can be integrated into the second sensing module. Alternatively, the control device can belong to the same sensing device as the first sensing module but be independent of the first sensing module; or the control device can belong to the same sensing device as the second sensing module but be independent of the second sensing module. Alternatively, the control device can belong to the same sensing device as both the first and second sensing modules but be independent of both. This application does not specifically limit the deployment relationship between the control device and the various sensing modules.

[0148] According to the control scheme provided in the embodiments of this application, this application also provides a terminal device, including the control device described above. Examples of terminal devices include, but are not limited to: smart home devices (such as televisions, robot vacuum cleaners, smart lamps, audio systems, smart lighting systems, appliance control systems, home background music systems, home theater systems, intercom systems, video surveillance, etc.), smart transportation equipment (such as automobiles, ships, drones, trains, freight cars, trucks, etc.), smart manufacturing equipment (such as robots, industrial equipment, smart logistics, smart factories, etc.), and smart terminals (mobile phones, computers, tablets, PDAs, desktop computers, headphones, speakers, wearable devices, in-vehicle devices, virtual reality devices, augmented reality devices, etc.).

[0149] According to the control scheme provided in the embodiments of this application, this application also provides a computer-readable storage medium storing a computer program, which, when run, executes the method performed by the control device as described above.

[0150] According to the control scheme provided in the embodiments of this application, this application also provides a computer program product that, when the computer program product is run on a processor, implements the method executed by the control device as described above.

[0151] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0152] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0153] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0157] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A control method, characterized in that, Applied to a first control device, the method includes: Obtain the operating parameters of the second sensing module; Based on the operating parameters of the second sensing module, the first sensing module is controlled to achieve scanning timing registration between the first sensing module and the second sensing module, wherein the second sensing module and the first sensing module share the same receiving optical component.

2. The method according to claim 1, characterized in that, The method further includes: Obtain first indication information, wherein the first indication information indicates the second scanning timing of the second sensing module; The step of controlling the first sensing module according to the operating parameters of the second sensing module to achieve scanning timing registration between the first sensing module and the second sensing module includes: Based on the operating parameters of the second sensing module and / or the first indication information, the first sensing module is controlled to achieve registration of the first scanning timing sequence of the first sensing module with the second scanning timing sequence.

3. The method according to claim 2, characterized in that, The acquisition of the first indication information includes: The first indication information is determined based on the operating parameters of the second sensing module; or... Receive the first instruction information from the second control device of the second sensing module.

4. The method according to any one of claims 1-3, characterized in that, The second sensing module includes a camera module, and the operating parameters of the second sensing module include at least one of the following parameters of the camera module: exposure time, sensitivity, frame rate, and line scan start interval; or, The second sensing module includes a lidar module, and the operating parameters of the second sensing module include at least one of the following parameters of the lidar module: frame rate, number of shots, transmission power, interval per shot, line scan start interval, scan angle interval, field of view (FOV) area size or region of interest (ROI) position, and range.

5. The method according to claim 4, characterized in that, The second sensing module is the camera module. The step of controlling the first sensing module according to the operating parameters of the second sensing module to achieve scanning timing registration between the first and second sensing modules includes at least one of the following: If the exposure time of the camera module increases and / or the photosensitivity decreases, the frame interval of the first sensing module is increased to achieve scanning timing registration between the first sensing module and the camera sensing module. If the line scan start interval of the camera module is reduced, the frame interval of the first sensing module is reduced to achieve scanning timing registration between the first sensing module and the camera sensing module. If the frame rate of the camera module increases, the frame interval of the first sensing module is reduced to achieve scanning timing registration between the first sensing module and the camera sensing module.

6. The method according to claim 5, characterized in that, The first sensing module is a lidar module, wherein, Increasing the frame interval of the first sensing module includes increasing one or more of the following parameters of the lidar module: Number of shots, interval between shots, line scan interval, FOV area size, range; or, Decreasing the frame interval of the first sensing module includes increasing one or more of the following parameters of the lidar module: Frame rate, scan angle interval.

7. The method according to claim 4, characterized in that, The second sensing module is the lidar module. The step of controlling the first sensing module according to the operating parameters of the second sensing module to achieve scanning timing registration between the first and second sensing modules includes at least one of the following: If one or more of the following parameters of the lidar module decrease, the first sensing module is controlled to accelerate the single-frame scanning speed to achieve scanning timing registration between the first sensing module and the lidar module: number of shots, line scanning start interval, FOV area size, and range. If the ROI position of the lidar module moves, the start time of the single-frame scan of the first sensing module is changed to achieve scanning timing registration between the first sensing module and the lidar module. If the scanning angle interval of the lidar module increases, the first sensing module is controlled to accelerate the single-frame scanning speed to achieve scanning timing registration between the first sensing module and the lidar module.

8. The method according to claim 7, characterized in that, The first sensing module is a camera module, wherein controlling the first sensing module to accelerate the single-frame scanning speed includes: Reduce the line scan start interval or exposure time of the camera module; or, Increase the frame rate or sensitivity of the camera module.

9. The method according to claim 4, characterized in that, The second sensing module is a lidar module and the first sensing module is a camera module. The single-frame point cloud detected by the lidar module includes at least two regions of interest (ROIs), and the single-frame image detected by the camera module includes at least two subframes corresponding to each of the at least two ROIs. If one or more of the following parameters associated with the first ROI of the at least two ROIs change accordingly: increased scanning angle interval, decreased number of shots, reduced line scan start interval, or decreased measurement range. The step of controlling the first sensing module according to the operating parameters of the second sensing module to achieve scanning timing registration between the first sensing module and the second sensing module includes: If the row scanning start interval of the camera module is adjustable, the camera module is controlled to accelerate the scanning speed of the subframe corresponding to the first ROI, so as to achieve scanning timing registration between the first sensing module and the lidar module; or, If the row scanning start interval of the camera module is not adjustable, the camera module is controlled to pause scanning other subframes after scanning the subframe corresponding to the first ROI, so as to achieve scanning timing registration between the camera module and the lidar module.

10. The method according to any one of claims 4-6 and 8-9, characterized in that, The camera module is of any of the following types: Visible light camera module, infrared camera module, thermal imaging camera module.

11. A control device, characterized in that, include: The acquisition unit is used to acquire the operating parameters of the second sensing module. The control unit is used to control the first sensing module according to the operating parameters of the second sensing module, so as to realize the scanning timing registration of the first sensing module and the second sensing module, wherein the second sensing module and the first sensing module share the same receiving optical component.

12. The apparatus according to claim 11, characterized in that, The acquisition unit is also used for: Obtain first indication information, wherein the first indication information indicates the second scanning timing of the second sensing module; The control unit is used for: Based on the operating parameters of the second sensing module and / or the first indication information, the first sensing module is controlled to achieve registration of the first scanning timing sequence of the first sensing module with the second scanning timing sequence.

13. The apparatus according to claim 12, characterized in that, The acquisition unit is used for: The first indication information is determined based on the operating parameters of the second sensing module; or... Receive the first instruction information from the second control device of the second sensing module.

14. The apparatus according to any one of claims 11-13, characterized in that, The second sensing module includes a camera module, and the operating parameters of the second sensing module include at least one of the following parameters of the camera module: exposure time, sensitivity, frame rate, and line scan start interval; or, The second sensing module includes a lidar module, and the operating parameters of the second sensing module include at least one of the following parameters of the lidar module: frame rate, number of shots, transmission power, interval per shot, line scan start interval, scan angle interval, FOV area size or ROI position of interest, and range.

15. The apparatus according to claim 14, characterized in that, The second sensing module is the camera module, and the control unit is used to perform at least one of the following: If the exposure time of the camera module increases and / or the photosensitivity decreases, the frame interval of the first sensing module is increased to achieve scanning timing registration between the first sensing module and the camera sensing module. If the line scan start interval of the camera module is reduced, the frame interval of the first sensing module is reduced to achieve scanning timing registration between the first sensing module and the camera sensing module. If the frame rate of the camera module increases, the frame interval of the first sensing module is reduced to achieve scanning timing registration between the first sensing module and the camera sensing module.

16. The apparatus according to claim 15, characterized in that, The first sensing module is a lidar module, wherein, The control unit controls the increase of the frame interval of the first sensing module, including controlling the increase of one or more of the following parameters of the lidar module: Number of shots, interval between shots, line scan interval, FOV area size, range; or, The control unit controls the reduction of the frame interval of the first sensing module, including controlling the increase of one or more of the following parameters of the lidar module: Frame rate, scan angle interval.

17. The apparatus according to claim 14, characterized in that, The second sensing module is the lidar module, and the control unit is used to perform at least one of the following: If one or more of the following parameters of the lidar module decrease, the first sensing module is controlled to accelerate the single-frame scanning speed to achieve scanning timing registration between the first sensing module and the lidar module: number of shots, line scanning start interval, FOV area size, and range. If the ROI position of the lidar module moves, the start time of the single-frame scan of the first sensing module is changed to achieve scanning timing registration between the first sensing module and the lidar module. If the scanning angle interval of the lidar module increases, the first sensing module is controlled to accelerate the single-frame scanning speed to achieve scanning timing registration between the first sensing module and the lidar module.

18. The apparatus according to claim 17, characterized in that, The first sensing module is a camera module, wherein the control unit controls the first sensing module to accelerate the single-frame scanning speed, including: Reduce the line scan start interval or exposure time of the camera module; or, Increase the frame rate or sensitivity of the camera module.

19. The apparatus according to claim 14, characterized in that, The second sensing module is a lidar module and the first sensing module is a camera module. The single-frame point cloud detected by the lidar module includes at least two regions of interest (ROIs), and the single-frame image detected by the camera module includes at least two subframes corresponding to each of the at least two ROIs. If one or more of the following parameters associated with the first ROI of the at least two ROIs change accordingly: increased scanning angle interval, decreased number of shots, decreased line scan start interval, or decreased range, the control unit is used to: If the row scanning start interval of the camera module is adjustable, the camera module is controlled to accelerate the scanning speed of the subframe corresponding to the first ROI, so as to achieve scanning timing registration between the first sensing module and the lidar module; or, If the row scanning start interval of the camera module is not adjustable, the camera module is controlled to pause scanning other subframes after scanning the subframe corresponding to the first ROI, so as to achieve scanning timing registration between the camera module and the lidar module.

20. The apparatus according to any one of claims 11-19, characterized in that, The control device is integrated into the first sensing module; or... The control device and the first sensing module belong to the same sensing device, and the control device is independent of the first sensing module; or... The control device belongs to the same sensing device as the first sensing module and the second sensing module, and the control device is independent of the first sensing module and the second sensing module.

21. The apparatus according to any one of claims 14-16, 18-19, characterized in that, The camera module is of any of the following types: Visible light camera module, infrared camera module, thermal imaging camera module.

22. A control device, characterized in that, It includes at least one processor and an interface circuit, the interface circuit being used to provide data or code instructions to the at least one processor, the at least one processor being used to implement the method as described in any one of claims 1-10 through logic circuits or executing code instructions.

23. A detection system, characterized in that, The device includes a control unit, a first sensing module, and a second sensing module, wherein the first sensing module and the second sensing module share the same receiving optical component, and the control unit is used to implement the method as described in any one of claims 1-10.

24. A terminal device, characterized in that, Includes the control device as described in any one of claims 11-21.

25. A computer-readable storage medium, characterized in that, The computer-readable medium stores program code that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-10.

26. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-10.

Images