Long-distance synchronous acquisition method and system for plurality of ordinary and dynamic vision sensors

Abstract

Embodiments of the present application provide a long-distance synchronous acquisition method and system for a plurality of ordinary and dynamic vision sensors. The long-distance synchronous acquisition system for a plurality of ordinary and dynamic vision sensors comprises a plurality of ordinary vision sensors, a plurality of dynamic vision sensors, and a control processor. The method is applied to the control processor, and the method comprises: determining a target ordinary vision sensor and determining a target dynamic vision sensor; acquiring a first synchronization signal of the target ordinary vision sensor, performing first delay compensation processing on the first synchronization signal, and then sending the first synchronization signal to the remaining ordinary vision sensors; performing second delay compensation processing on the first synchronization signal, and then sending the first synchronization signal to each dynamic vision sensor; and receiving a second synchronization signal sent by the target dynamic vision sensor, performing third delay compensation processing on the second synchronization signal, and then sending the second synchronization signal to the remaining dynamic vision sensors, so as to perform data synchronous acquisition, thereby improving the accuracy and synchronism of image data acquisition by the plurality of vision sensors.

Description

Long-distance synchronous acquisition method and system for multiple ordinary and dynamic vision sensors Technical Field

[0001] This application relates to the field of image acquisition technology, and in particular to a method and system for long-distance synchronous acquisition of multiple ordinary and dynamic vision sensors. Background Technology

[0002] In advanced driver assistance systems (ADAS), the surrounding environment needs to be observed, requiring multiple vision sensors to simultaneously acquire signals from different directions for subsequent processing and decision-making. Currently, AAS primarily relies on ordinary vision sensors. Ordinary vision sensors suffer from several drawbacks: low frame rates, resulting in blurring and trailing when dealing with fast-moving objects; and relatively narrow dynamic ranges, leading to poor recognition in strong or low light conditions. Dynamic Vision Sensors (DVS) are a novel event-based sensor that triggers events by detecting changes in the brightness of each pixel. When the brightness change of a pixel reaches a certain threshold, the sensor outputs an event containing pixel coordinates, occurrence time, and polarity. A positive polarity indicates a brightness increase exceeding a high threshold, while a negative polarity indicates a brightness decrease below a low threshold. DVS offers advantages such as microsecond-level temporal resolution and low-latency output, along with a dynamic range of up to 120dB, significantly higher than the 60dB of traditional vision sensors, making it suitable for perception in extreme scenarios such as strong and low light. While DVS can capture scene change information, it lacks scene texture information. Therefore, in autonomous driving assistance systems, the combination of multiple ordinary vision sensors and dynamic vision sensors can complement each other's advantages and disadvantages. It can not only collect complete grayscale information of images from multiple directions, but also solve the problems of blurriness or high latency when capturing fast-moving objects, as well as the problem of unclear recognition under strong light or low light conditions.

[0003] In advanced driver assistance systems (ADAS), vision sensors are typically mounted on the roof or sides of the vehicle, while the autonomous driving domain controller that controls the cameras is usually located in the trunk at the rear, a distance of several meters. To achieve high-bandwidth, long-distance transmission of data from multiple ordinary vision sensors, the common practice is to use serial-to-deserialization schemes like GMSL or FPD-LINK. However, dynamic vision sensor data uses a custom message format, completely different from the standard RAW / RGB / YUV message formats of ordinary vision sensors, and the frame length is not fixed. Using the same mode as ordinary vision sensors would lead to packet errors. Therefore, solving the problem of lossless long-distance transmission of dynamic vision sensor data has become a challenge.

[0004] For combined arrays of multiple ordinary vision sensors and multiple dynamic vision sensors, each sensor has its own local clock source. Without time synchronization, inconsistencies in the timestamps of data recorded by different sensors at the same point in time will hinder subsequent information processing and decision-making. However, in related technologies, there is currently no suitable solution for time synchronization of combined arrays of multiple ordinary vision sensors and multiple dynamic vision sensors. Summary of the Invention

[0005] This application provides a method and system for long-distance synchronous acquisition of multiple ordinary and dynamic vision sensors, which can improve the synchronization of image data acquisition by multiple ordinary vision sensors and multiple dynamic vision sensors.

[0006] To achieve the above objectives, a first aspect of this application proposes a long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors. The multiple ordinary and dynamic vision sensors include multiple ordinary vision sensors, multiple dynamic vision sensors, and a control processor. The method is applied to the control processor and includes:

[0007] Determine the target general vision sensor from among the plurality of general vision sensors, and determine the target dynamic vision sensor from among the plurality of dynamic vision sensors;

[0008] Acquire the first synchronization signal of the target ordinary vision sensor, and send the first synchronization signal to the other ordinary vision sensors after performing a first delay compensation process;

[0009] After performing a second delay compensation process on the first synchronization signal, it is sent to the external event trigger pin of the target dynamic vision sensor;

[0010] The system receives a second synchronization signal from the target dynamic vision sensor, performs a third delay compensation process on the second synchronization signal, and then sends it to the other dynamic vision sensors so that the multiple ordinary vision sensors and the multiple dynamic vision sensors can synchronously acquire data.

[0011] In some embodiments, each of the general vision sensors is connected to the control processor via a serializer, a high-speed coaxial cable, and a deserializer. Acquiring the first synchronization signal of the target general vision sensor includes:

[0012] The first synchronization signal is obtained after being emitted by the target ordinary vision sensor and sequentially processed by the target ordinary serializer, the high-speed coaxial cable, and the target ordinary deserializer. The target ordinary serializer is the serializer connected to the target ordinary vision sensor, and the target ordinary deserializer is the deserializer connected to the target ordinary vision sensor.

[0013] In some embodiments, the step of sending the first synchronization signal to the other ordinary vision sensors after performing a first delay compensation process includes:

[0014] The first serialization delay time generated when the first synchronization signal passes through the target ordinary serializer and the first deserialization delay time generated when it passes through the target ordinary deserializer are obtained.

[0015] A first delay compensation processing time is generated based on the first serialization delay time and the first deserialization delay time;

[0016] The first synchronization signal is subjected to a first delay compensation process based on the first delay compensation processing time, and the first synchronization signal after the first delay compensation processing is sent to the other ordinary vision sensors.

[0017] In some embodiments, generating a first delay compensation processing time based on the first serialization delay time and the first deserialization delay time includes:

[0018] The second serialization delay time generated when the first synchronization signal passes through other ordinary serializers and the second deserialization delay time generated when passing through other ordinary deserializers are obtained. The other ordinary serializers are the serializers connected to other ordinary vision sensors, and the other ordinary deserializers are the deserializers connected to other ordinary vision sensors.

[0019] Obtain the first synchronization time period for the first synchronization signal generated by the target ordinary vision sensor;

[0020] The first delay compensation processing time is obtained by subtracting the first serialization delay time, the second serialization delay time, the first deserialization delay time, and the second deserialization delay time sequentially from the first synchronization time period.

[0021] In some embodiments, each of the dynamic vision sensors is connected to the control processor via a serializer, a high-speed coaxial cable, and a deserializer. The step of sending the first synchronization signal to each of the dynamic vision sensors after performing a second delay compensation process includes:

[0022] The third serialization delay time generated when the first synchronization signal passes sequentially through the serializer connected to the dynamic vision sensor and the third deserialization delay time generated when the dynamic vision sensor is connected to the deserializer are obtained.

[0023] A second delay compensation processing time is generated based on the first synchronization time period, the first serialization delay time, the first deserialization delay time, the third serialization delay time, and the third deserialization delay time;

[0024] The first synchronization signal is subjected to a second delay compensation process based on the second delay compensation processing time, and the first synchronization signal after the second delay compensation processing is sent to each of the dynamic vision sensors.

[0025] In some embodiments, the step of sending the second synchronization signal to the remaining dynamic vision sensors after undergoing a third delay compensation process includes:

[0026] Obtain the changing pulse phase of the remaining dynamic vision sensors when they receive the second synchronization signal;

[0027] The third delay compensation processing time is generated based on the changed pulse phase;

[0028] The second synchronization signal is subjected to a third delay compensation process based on the third delay compensation processing time, and the second synchronization signal after the third delay compensation processing is sent to the other dynamic vision sensors.

[0029] In some embodiments, the generation of a third delay compensation processing time based on the changed pulse phase includes:

[0030] Obtain the phase difference of the second synchronization signal between the target dynamic vision sensor and the other dynamic vision sensors;

[0031] When the changing pulse phase is delayed, the second synchronization time period of the target dynamic vision sensor generating the second synchronization signal is obtained, and the third delay compensation processing time is obtained based on the difference between the second synchronization time period and the phase difference of the second synchronization signal.

[0032] When the changing pulse phase is ahead of the pulse phase, the third delay compensation processing time is obtained based on the phase difference of the second synchronization signal.

[0033] In some embodiments, the serialization and deserialization of the dynamic vision sensor are implemented based on the Tunnel mode of the GMSL chip.

[0034] In some embodiments, the plurality of conventional vision sensors, the plurality of dynamic vision sensors, and the control processor transmit data via a single COAX high-speed COAX cable.

[0035] To achieve the above objectives, a second aspect of this application provides a long-distance synchronous acquisition system for multiple ordinary and dynamic vision sensors. The multiple ordinary and dynamic vision sensors include multiple ordinary vision sensors, multiple dynamic vision sensors, and a control processor. The device is applied to the control processor and includes:

[0036] A target sensor determination module is used to determine a target general vision sensor from a plurality of general vision sensors, and to determine a target dynamic vision sensor from a plurality of dynamic vision sensors;

[0037] The first delay compensation module is used to acquire the first synchronization signal of the target ordinary vision sensor, and send the first synchronization signal to the other ordinary vision sensors after performing a first delay compensation process.

[0038] The second delay compensation module is used to send the first synchronization signal to each of the dynamic vision sensors after performing a second delay compensation process.

[0039] The third delay compensation module is used to receive the second synchronization signal emitted by the target dynamic vision sensor, and send the second synchronization signal to the other dynamic vision sensors after performing the third delay compensation processing, so that the multiple ordinary vision sensors and the multiple dynamic vision sensors can perform data synchronous acquisition.

[0040] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the long-distance synchronous acquisition method of multiple ordinary and dynamic vision sensors as described in the first aspect.

[0041] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the long-distance synchronous acquisition method of multiple ordinary and dynamic vision sensors described in the first aspect.

[0042] The present application proposes a long-distance synchronous acquisition method and system for multiple ordinary and dynamic vision sensors. The multiple ordinary and dynamic vision sensors include multiple ordinary vision sensors, multiple dynamic vision sensors, and a control processor. The method is applied to the control processor and includes: first, determining a target ordinary vision sensor from among the multiple ordinary vision sensors, and determining a target dynamic vision sensor from among the multiple dynamic vision sensors; then, acquiring a first synchronization signal from the target ordinary vision sensor, performing a first delay compensation process on the first synchronization signal, and sending it to the remaining ordinary vision sensors; next, performing a second delay compensation process on the first synchronization signal, and sending it to the external event trigger pin of each dynamic vision sensor; finally, receiving a second synchronization signal from the target dynamic vision sensor, performing a third delay compensation process on the second synchronization signal, and sending it to the remaining dynamic vision sensors, thereby enabling the multiple ordinary vision sensors and the multiple dynamic vision sensors to acquire data synchronously. This application employs multiple conventional vision sensors and multiple dynamic vision sensors to acquire image data simultaneously. This leverages the advantages of different types of vision sensors, enabling the acquisition of scene texture information, resolving issues of blurriness or high latency when capturing fast-moving objects, and addressing unclear identification under strong or weak light conditions, thereby improving the accuracy of data acquisition. Furthermore, a first delay compensation process is applied to the first synchronization signal between the multiple conventional vision sensors, a second delay compensation process is applied to the first synchronization signal between the conventional and dynamic vision sensors, and a third delay compensation process is applied to the second synchronization signal between the multiple dynamic vision sensors. This allows for corresponding delay compensation processing for different situations and synchronization signals, enabling multiple vision sensors to acquire data simultaneously, thus improving the accuracy and synchronization of image data acquisition by the multiple vision sensors.

[0043] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0044] Figure 1 is a schematic diagram of the structure of a visual acquisition system provided in an embodiment of this application.

[0045] Figure 2 is a flowchart of a long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors provided in another embodiment of this application.

[0046] Figure 3 is a schematic diagram of the process of transmitting a first synchronization signal from a target ordinary vision sensor to other ordinary vision sensors according to another embodiment of this application.

[0047] Figure 4 is a flowchart of step 202 in Figure 2.

[0048] Figure 5 is a flowchart of step 402 in Figure 4.

[0049] Figure 6 is a schematic diagram of the time curve of a first synchronization signal after a first delay compensation process according to another embodiment of this application.

[0050] Figure 7 is a schematic diagram of the information acquisition triggering synchronization of a dynamic vision sensor according to another embodiment of this application.

[0051] Figure 8 is a schematic diagram of the process of transmitting a first synchronization signal from a target ordinary vision sensor to a dynamic vision sensor according to another embodiment of this application.

[0052] Figure 9 is a flowchart of step 203 in Figure 2.

[0053] Figure 10 is a schematic diagram of the process of transmitting a second synchronization signal from a target dynamic vision sensor to other dynamic vision sensors according to another embodiment of this application.

[0054] Figure 11 is a flowchart of step 204 in Figure 2.

[0055] Figure 12 is a flowchart of step 1102 in Figure 11.

[0056] Figure 13 is a schematic diagram of the signal waveform curve of a dynamic vision sensor receiving a second synchronization signal according to another embodiment of this application.

[0057] Figure 14 is a schematic diagram of the structure of a long-distance synchronous acquisition system for multiple ordinary and dynamic vision sensors provided in another embodiment of this application.

[0058] Figure 15 is a schematic diagram of the hardware structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0062] In advanced driver assistance systems (ADAS), the surrounding environment needs to be observed, requiring multiple vision sensors to simultaneously acquire signals from different directions for subsequent processing and decision-making. Currently, AAS primarily relies on ordinary vision sensors. Ordinary vision sensors suffer from several drawbacks: low frame rates, resulting in blurring and trailing when dealing with fast-moving objects; and relatively narrow dynamic ranges, leading to poor recognition in strong or low light conditions. Dynamic Vision Sensors (DVS) are a novel event-based sensor that triggers events by detecting changes in the brightness of each pixel. When the brightness change of a pixel reaches a certain threshold, the sensor outputs an event containing pixel coordinates, occurrence time, and polarity. A positive polarity indicates a brightness increase exceeding a high threshold, while a negative polarity indicates a brightness decrease below a low threshold. DVS offers advantages such as microsecond-level temporal resolution and low-latency output, along with a dynamic range of up to 120dB, significantly higher than the 60dB of traditional vision sensors, making it suitable for perception in extreme scenarios such as strong and low light. While DVS can capture scene change information, it lacks scene texture information. Therefore, in autonomous driving assistance systems, the combination of multiple ordinary vision sensors and dynamic vision sensors can complement each other's advantages and disadvantages. It can not only collect complete grayscale information of images from multiple directions, but also solve the problems of blurriness or high latency when capturing fast-moving objects, as well as the problem of unclear recognition under strong light or low light conditions.

[0063] In advanced driver assistance systems (ADAS), vision sensors are typically mounted on the roof or sides of the vehicle, while the autonomous driving domain controller that controls the cameras is usually located in the trunk at the rear, a distance of several meters. To achieve high-bandwidth, long-distance transmission of data from multiple ordinary vision sensors, the common practice is to use serial-to-deserialization schemes like GMSL or FPD-LINK. However, dynamic vision sensor data uses a custom message format, completely different from the standard RAW / RGB / YUV message formats of ordinary vision sensors, and the frame length is not fixed. Using the same mode as ordinary vision sensors would lead to packet errors. Therefore, solving the problem of lossless long-distance transmission of dynamic vision sensor data has become a challenge.

[0064] For combined arrays of multiple ordinary vision sensors and multiple dynamic vision sensors, each sensor has its own local clock source. Without time synchronization, inconsistencies in the timestamps of data recorded by different sensors at the same point in time will hinder subsequent information processing and decision-making. However, in related technologies, there is currently no suitable solution for time synchronization of combined arrays of multiple ordinary vision sensors and multiple dynamic vision sensors.

[0065] To improve the synchronization of image data acquisition by multiple ordinary vision sensors and multiple dynamic vision sensors, this application embodiment employs multiple ordinary vision sensors and multiple dynamic vision sensors to acquire image data simultaneously. This leverages the advantages of different types of vision sensors, enabling the acquisition of scene texture information, resolving issues of blurriness or high latency when capturing fast-moving objects, and addressing unclear identification under strong or weak light conditions, thereby improving the accuracy of data acquisition. Furthermore, a first delay compensation process is applied to the first synchronization signal between multiple ordinary vision sensors, a second delay compensation process is applied to the first synchronization signal between ordinary and dynamic vision sensors, and a third delay compensation process is applied to the second synchronization signal between multiple dynamic vision sensors. This allows for corresponding delay compensation processing for different situations and synchronization signals, enabling multiple vision sensors to acquire data simultaneously, thereby improving the accuracy and synchronization of image data acquisition by multiple vision sensors.

[0066] The following will further describe the long-distance synchronous acquisition method and system for multiple ordinary and dynamic vision sensors provided in the embodiments of this application. First, the vision acquisition system provided in the embodiments of this application will be described. Referring to Figure 1, it is a schematic diagram of the structure of a vision acquisition system provided in the embodiments of this application. As shown in Figure 1, the vision acquisition system is provided with multiple ordinary vision sensors, multiple dynamic vision sensors, and a control processor. Each ordinary vision sensor or each dynamic vision sensor is connected to the control processor through a GMSL serializer, a high-speed COAX cable, and a GMSL deserializer. The control processor can be an FPGA or a CPLD.

[0067] Because the message format of dynamic vision sensors is completely different from the standard message formats such as RAW / RGB / YUV of ordinary vision sensors, the GMSL chip's Tunnel mode is used for serialization and deserialization during the serialization process via the GMSL serializer connected to the dynamic vision sensors to ensure data integrity and correctness. Furthermore, to reduce sensor cabling, a single high-speed COAX cable is used to connect the Vsync signal of the ordinary vision sensors, the SYN and TRIG signals of the dynamic vision sensors to the FPGA / CPLD, resolving the connection issue. Serialization and deserialization are then routed through the GMSL control channel, enabling the data to share a single COAX cable with the sensors.

[0068] After acquiring data, the vision sensor (including ordinary vision sensor and dynamic vision sensor) converts the data into GMSL serial data through the GMSL serializer via the MIPI CSI2 interface. The GMSL serial data is then transmitted to the GMSL deserializer via a high-speed COAX cable. The GMSL deserializer then deserializes the GMSL serial data into MIPI CSI2 signals and sends them to the control processor.

[0069] Because the message format of the dynamic vision sensor is completely different from the standard message formats such as RAW / RGB / YUV of ordinary vision sensors, this embodiment uses the Tunnel mode of the GMSL chip for serial deserialization to ensure data integrity and correctness. Furthermore, to reduce sensor cabling, a high-speed COAX cable is used to solve the connection problem between the Vsync signal of the ordinary vision sensor, the SYN and TRIG signals of the dynamic vision sensor, and the FPGA / CPLD. Serial deserialization is performed via the GMSL control channel, allowing data to share a single COAX cable with the sensor data.

[0070] Conventional vision sensors, also known as traditional image sensors, typically refer to sensors that capture images at a fixed frame rate, such as CCD (charge-coupled device) or CMOS (complementary metal-oxide-semiconductor) sensors. Dynamic vision sensors, on the other hand, are a new type of event-based vision sensor. Unlike traditional sensors, they do not capture the entire image at a fixed frame rate; instead, they generate data only when pixel brightness changes. Conventional vision sensors are suitable for routine image capture tasks, while dynamic vision sensors excel in applications requiring high temporal resolution, high dynamic range, and low latency. Both can be used complementaryly in areas such as advanced driver assistance systems (ADAS) and robot vision to achieve better environmental perception and decision-making.

[0071] An FPGA (Field Programmable Gate Array) is a configurable semiconductor device that contains programmable logic blocks, input / output blocks, and internal interconnects. FPGAs allow developers to design and modify hardware logic to meet specific application requirements without changing the physical hardware.

[0072] A CPLD (Complex Programmable Logic Device) is a relatively simple programmable logic device that uses an EEPROM-based memory to store logic configurations.

[0073] COAX cable (Coaxial cable) is a common type of transmission line, consisting of a central conductor (inner conductor) and an outer conductive layer (outer conductor), with the two layers separated by an insulating material.

[0074] A GMSL serializer is an integrated circuit that converts parallel data (such as video signals) into a high-speed serial data stream. This conversion allows data to be transmitted over long distances via a single coaxial or twisted-pair cable without significantly degrading signal quality. GMSL serializers are typically used in conjunction with GMSL deserializers for data transmission and reception.

[0075] A GMSL deserializer is the counterpart to a GMSL serializer. It receives high-speed serial data streams and converts them back to their original parallel data format. In video signal transmission, this means that the deserializer restores the received serial video data into a parallel video signal that can be processed by display devices or image processing systems.

[0076] The following describes in detail the long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors in the embodiments of this application. Referring to Figure 2, an optional flowchart of the long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors provided in the embodiments of this application is shown. The method in Figure 2 may include, but is not limited to, steps 201 to 204. It is also understood that this embodiment does not specifically limit the order of steps 201 to 204 in Figure 2, and the order of steps can be adjusted or certain steps can be reduced or added according to actual needs. The long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors provided in the embodiments of this application can be applied to the control processor of a vision acquisition system.

[0077] Step 201: Determine the target general vision sensor from multiple general vision sensors, and determine the target dynamic vision sensor from multiple dynamic vision sensors.

[0078] Step 201 will be described in detail below.

[0079] In some embodiments, in response to an image data acquisition request from a vision acquisition system, the aim is to improve the synchronization of data acquisition from multiple vision sensors.

[0080] First, one ordinary vision sensor needs to be selected from multiple ordinary vision sensors as the target ordinary vision sensor. That is, one ordinary vision sensor out of M ordinary vision sensors is set to Master mode, while the other ordinary vision sensors remain in Slave mode. Similarly, one dynamic vision sensor is selected from multiple dynamic vision sensors as the target dynamic vision sensor. That is, one dynamic vision sensor out of N dynamic vision sensors is set to Master mode, while the other dynamic vision sensors remain in Slave mode. As shown in Figure 1, ordinary vision sensor number 1 is selected as the target ordinary vision sensor, and dynamic vision sensor number 1 is selected as the target dynamic vision sensor.

[0081] Step 202: Obtain the first synchronization signal of the target ordinary vision sensor, and send the first synchronization signal to the other ordinary vision sensors after performing the first delay compensation processing.

[0082] Step 202 will be described in detail below.

[0083] In some embodiments, after determining the target general vision sensor and the target dynamic vision sensor, in order to ensure the synchronization of data acquisition between multiple general vision sensors and between the general vision sensor and multiple dynamic vision sensors, the target general vision sensor outputs a first synchronization signal Vsync, which is then transmitted to the control processor via a target general serializer (i.e., the GMSL serializer connected to the target general vision sensor as shown in FIG1), a high-speed coaxial cable (i.e., the high-speed COAX cable shown in FIG1), and a target general deserializer (i.e., the GMSL deserializer connected to the target general vision sensor as shown in FIG1). The control processor uses the first synchronization signal Vsync to synchronize the time between multiple general vision sensors and between the target general vision sensor and multiple dynamic vision sensors.

[0084] The following describes the synchronization process for multiple general vision sensors. To achieve synchronized data acquisition from multiple general vision sensors, a first delay compensation process is performed on the first synchronization signal emitted by the target general vision sensor before it is sent to the other general vision sensors. Referring to Figure 3, it is a schematic diagram of the process of transmitting the first synchronization signal from the target general vision sensor to other general vision sensors according to an embodiment of this application. As shown in Figure 3, taking two general vision sensors as an example, after the target general vision sensor emits the first synchronization signal Vsync, the first synchronization signal Vsync will sequentially pass through the target general serializer (i.e., the GMSL serializer connected to general vision sensor No. 1 as shown in Figure 1) and the target general deserializer (i.e., the GMSL deserializer connected to general vision sensor No. 1 as shown in Figure 1) to the control processor; then, after the control processor performs the first delay compensation process on the first synchronization signal Vsync, it sequentially passes through other general deserializers (i.e., the GMSL deserializer connected to general vision sensor No. 2 as shown in Figure 1) and other general serializers (i.e., the GMSL serializer connected to general vision sensor No. 2 as shown in Figure 1) to the second general vision sensor.

[0085] The following section will further describe how to perform the first delay compensation process on the first synchronization signal.

[0086] Referring to Figure 4, the first synchronization signal is sent to the other ordinary vision sensors after undergoing a first delay compensation process, including the following steps 401 to 403.

[0087] Step 401: Obtain the first serialization delay time generated when the first synchronization signal passes through the target ordinary serializer and the first deserialization delay time generated when it passes through the target ordinary deserializer.

[0088] Step 402: Generate the first delay compensation processing time based on the first serialization delay time and the first deserialization delay time.

[0089] Steps 401 to 402 are described in detail below.

[0090] In some embodiments, a processing delay is unavoidable during the first synchronization signal's passage through the target serializer and deserializer. For example, if the first synchronization signal is connected to the GPIO of the GMSL serializer / deserializer chip, and the GPIO is configured as an input, after detecting a signal change at the input terminal, the GPIO output terminal with the same ID as the input terminal on the other side will copy the input signal, as if the two sides were directly connected. This copying process generates a delay in the microsecond range. However, the GMSL chip can be configured in delay compensation mode to ensure that the delay is a fixed value, thereby ensuring that the delay time of the synchronization signal each time it passes through the GMSL serializer and deserializer is fixed and predictable.

[0091] Therefore, after the control processor receives the first synchronization signal emitted by the target ordinary vision sensor, it acquires the first serialization delay time t0 generated when the first synchronization signal passes through the target ordinary serializer and the first deserialization delay time t1 generated when the target ordinary deserializer passes through the first serialization delay time t0 and the first deserialization delay time t1. Next, based on the first serialization delay time t0 and the first deserialization delay time t1, a first delay compensation processing time for performing the first delay compensation processing on each of the remaining ordinary vision sensors is generated, as described below.

[0092] Referring to Figure 5, a first delay compensation processing time is generated based on a first serialization delay time and a first deserialization delay time, including the following steps 501 to 503.

[0093] Step 501: Obtain the second serialization delay time generated when the first synchronization signal passes through other ordinary serializers and the second deserialization delay time generated when passing through other ordinary deserializers.

[0094] Step 502: Obtain the first synchronization time period of the first synchronization signal generated by the target ordinary vision sensor.

[0095] Step 503: Subtract the first serialization delay time, the second serialization delay time, the first deserialization delay time, and the second deserialization delay time sequentially from the first synchronization time period to obtain the first delay compensation processing time.

[0096] Steps 501 to 503 are described in detail below.

[0097] In some embodiments, in addition to obtaining the first serialization delay time t0 and the first deserialization delay time t1, for each of the remaining ordinary vision sensors, the second serialization delay time t2 generated when the first synchronization signal passes through other ordinary serializers and the second deserialization delay time t3 generated when the first synchronization signal passes through other ordinary serializers in sequence during transmission to the ordinary vision sensor corresponding to that ordinary vision sensor are obtained.

[0098] At the same time, it is also necessary to obtain the first synchronization time period ΔT of the target ordinary vision sensor generating the first synchronization signal, that is, the time interval between generating two consecutive first synchronization signals, which is also the frame period of the target ordinary vision sensor.

[0099] Then, by subtracting the first serialization delay time t0, the second serialization delay time t2, the first deserialization delay time t1, and the second deserialization delay time t3 in sequence from the first synchronization time period ΔT, the first delay compensation processing time Δt1 = ΔT - t0 - t1 - t2 - t3 between the target ordinary vision sensor and each other ordinary vision sensor can be obtained.

[0100] Step 403: Perform first delay compensation processing on the first synchronization signal based on the first delay compensation processing time, and send the first synchronization signal after the first delay compensation processing to the other ordinary vision sensors.

[0101] Step 403 will be described in detail below.

[0102] In some embodiments, after obtaining the first delay compensation processing time Δt1 corresponding to each of the other ordinary vision sensors, the control processor performs a first delay compensation processing on the first synchronization signal Vsync according to the first delay compensation processing time Δt1, and sends the first synchronization signal after the first delay compensation processing to the other corresponding ordinary vision sensors in sequence through the GMSL deserializer and GMSL serializer connected to the other ordinary vision sensors, thereby ensuring the phase synchronization of all ordinary vision sensors (i.e., rising edge or falling edge phase alignment), and thus enabling synchronous data information acquisition.

[0103] Referring to Figure 6, it is a schematic diagram of the time curve of a first synchronization signal after a first delay compensation process according to an embodiment of this application. As shown in Figure 6, taking the output of the first synchronization signal Vsync by ordinary vision module 1 operating in Master mode as an example, and finally sent to the Vsync input terminal of ordinary vision module 2 operating in Slave mode, the data flow of the first synchronization signal is shown in Figure 3. As shown in Figure 6, the Vsync signal output by ordinary vision sensor 1 operating in Master mode is delayed by t0 after passing through the GMSL serializer of module 1, delayed by t1 after passing through the deserializer of module 1, and then after delay compensation in the control processor, the pulse phase is shifted back by Δt1. After being delayed by t2 after passing through the deserializer of module 2, and finally delayed by t3 after passing through the serializer of module 2, the rising edge of the signal sent to the Vsync pin of ordinary vision sensor 2 operating in Slave mode is aligned with T+ΔT, where ΔT is one Vsync signal cycle. To ensure that the rising edge phase of the pulse received by the Vsync sensor #2 is the same as the rising edge phase of the pulse sent by the Vsync sensor #1, the first delay compensation time of the FPGA / CLPD is Δt1, which is Δt1 = ΔT - t0 - t1 - t2 - t3.

[0104] It is understandable that the delay values ​​of GPIOs with different IDs are configurable within a certain delay range. That is, the delay values ​​t0-t3 in Figure 6 can all be configured via registers, with the smallest granularity of the delay value being in the microsecond range. Given that the Vsync signal period ΔT and the serial-to-deserial delays t0-t3 are known, the total delay value of the first delay compensation time Δt1 can be calculated. Internally, the FPGA / CLPD can adjust the Vsync phase using local high-frequency clock pulse counting to perform delay compensation.

[0105] Through steps 401 to 403 and steps 501 to 503 above, using a GMSL serializer and a GMSL deserializer with a fixed processing delay, the processing delay generated when the first synchronization signal is sent from the target ordinary vision sensor to each of the other ordinary vision sensors is predetermined. Combined with the first synchronization time period for generating the first synchronization signal, the first delay compensation time corresponding to phase synchronization between each of the other ordinary vision sensors and the target ordinary vision sensor can be accurately determined. Thus, the first delay compensation time can be used to accurately realize the synchronous image data acquisition of multiple ordinary vision sensors, thereby improving the synchronization of data acquisition by multiple ordinary and dynamic vision sensors.

[0106] Step 203: After performing a second delay compensation process on the first synchronization signal, send it to the external event trigger pin of each dynamic vision sensor.

[0107] Step 203 will be described in detail below.

[0108] In some embodiments, to achieve time synchronization between ordinary vision sensors and dynamic vision sensors, the Vsync signal of the ordinary vision sensor operating in Master mode needs to be sent to the external event trigger pin TRIG of N dynamic vision sensors through a control processor. Referring to Figure 7, it is a schematic diagram of the acquisition information trigger synchronization of a dynamic vision sensor according to an embodiment of this application. As shown in Figure 7, the TRIG signal of the dynamic vision sensor is an external event trigger signal. When the TRIG signal is set as an input, as shown in Figure 7, when the TRIG signal detects a rising or falling edge (with accuracy down to the microsecond level), the dynamic vision sensor generates a TRIG event. This event records the timestamp of the change in the external TRIG signal, such as [T2, T0] or [T3, T1]. The time difference between T2 and T0 (or T3 and T1) is one frame period of an ordinary vision sensor (e.g., 30Hz or 60Hz). During the frame period, the dynamic vision sensor generates multiple positive polarity (increased brightness) and negative polarity (decreased brightness) event information representing brightness changes. This event information includes timestamps, coordinates, and polarity. Therefore, by aligning the timestamps of external trigger events recorded by the dynamic vision sensor with the rising or falling edge of the Vsync signal of each frame of the ordinary vision sensor, time synchronization between ordinary vision sensors and dynamic vision sensors of different sensor types can be achieved.

[0109] Referring to Figure 8, it is a schematic diagram of the process of transmitting a first synchronization signal from a target ordinary vision sensor to a dynamic vision sensor according to an embodiment of this application. As shown in Figure 8, taking an ordinary vision sensor and a dynamic vision sensor as an example, after the target ordinary vision sensor sends out the first synchronization signal Vsync, the first synchronization signal Vsync will sequentially pass through the target ordinary serializer (i.e., the GMSL serializer connected to the No. 1 ordinary vision sensor as shown in Figure 1) and the target ordinary deserializer (i.e., the GMSL deserializer connected to the No. 1 ordinary vision sensor as shown in Figure 1) to the control processor; then, after the control processor performs a second delay compensation process on the first synchronization signal Vsync, it will sequentially send it to the external event trigger pin TRIG of the No. 1 dynamic vision sensor through the GMSL deserializer connected to the No. 1 dynamic vision sensor and the GMSL serializer connected to the No. 1 dynamic vision sensor.

[0110] The following section will further describe how to perform the second delay compensation process on the first synchronization signal.

[0111] Referring to Figure 9, the first synchronization signal is sent to each dynamic vision sensor after undergoing a second delay compensation process, including the following steps 901 to 903.

[0112] Step 901: Obtain the third serialization delay time generated when the first synchronization signal passes through the serializer connected to the dynamic vision sensor and the third deserialization delay time generated when it passes through the deserializer connected to the dynamic vision sensor.

[0113] Step 902: Generate a second delay compensation processing time based on the first synchronization time period, the first serialization delay time, the first deserialization delay time, the third serialization delay time, and the third deserialization delay time.

[0114] Step 903: Perform second delay compensation processing on the first synchronization signal based on the second delay compensation processing time, and send the first synchronization signal after the second delay compensation processing to the target dynamic vision sensor.

[0115] Steps 901 to 903 are described in detail below.

[0116] In some embodiments, similar to the first delay compensation process described in steps 401 to 403 above, when performing the second delay compensation process for the first synchronization signal Vsync, it is necessary to obtain in advance the third serialization delay time t4 generated when the first synchronization signal passes through the GMSL serializer connected to each dynamic vision sensor and the third deserialization delay time t5 generated when the first synchronization signal passes through the GMSL deserializer connected to each dynamic vision sensor.

[0117] Then, by subtracting the first serialization delay time t0, the third serialization delay time t4, the first deserialization delay time t1, and the third deserialization delay time t5 sequentially from the first synchronization time period ΔT, we can obtain the second delay compensation processing time Δt2 = ΔT - t0 - t1 - t4 - t5 between the target ordinary vision sensor and each dynamic vision sensor.

[0118] Then, the control processor performs second delay compensation processing on the first synchronization signal Vsync according to the second delay compensation processing time Δt2, and sends the first synchronization signal after the second delay compensation processing to the external event trigger pin TRIG of each corresponding dynamic vision sensor through the GMSL deserializer and GMSL serializer connected to each vision sensor in sequence. This ensures the time synchronization association between the target ordinary vision sensor and each dynamic vision sensor (i.e., the rising edge or falling edge of the Vsync transmitter of the target ordinary vision sensor is phase-aligned with the rising edge or falling edge of the Vsync signal received by the external event trigger pin TRIG of the dynamic vision sensor), thereby enabling synchronous data information acquisition.

[0119] Through steps 901 to 903 above, using a GMSL serializer and GMSL deserializer with a fixed processing delay, the processing delay generated when the first synchronization signal is sent from the target ordinary vision sensor to the target dynamic vision sensor is predetermined. Combined with the first synchronization time period for generating the first synchronization signal, the second delay compensation time can be accurately determined. Thus, by combining the second delay compensation time with the timestamp recorded by the external trigger event of the dynamic vision sensor and the rising or falling edge alignment of the first synchronization signal of each frame of the ordinary vision sensor, time synchronization between ordinary vision sensors and dynamic vision sensors of different sensor types can be achieved, thereby improving the synchronization of data acquisition by multiple ordinary and dynamic vision sensors.

[0120] Step 204: Receive the second synchronization signal emitted by the target dynamic vision sensor, and send the second synchronization signal to the other dynamic vision sensors after performing the third delay compensation processing, so that multiple dynamic vision sensors can collect data synchronously.

[0121] Step 204 is described in detail below.

[0122] In some embodiments, to ensure the synchronization of data acquisition by multiple dynamic vision sensors, the target dynamic vision sensor operating in Master mode outputs a second synchronization signal SYN, which is sent to the remaining dynamic vision sensors via the control processor, thereby achieving data acquisition synchronization among the multiple dynamic vision sensors. Specifically, the target dynamic vision sensor sends the second synchronization signal SYN to the SYN input pins of N-1 dynamic vision sensors operating in Slave mode via the FPGA / CPLD. The second synchronization signal SYN serves as the reference clock for timing events of the dynamic vision sensors.

[0123] In some embodiments, after the ordinary GPIO of the control processor FPGA / CPLD receives the second synchronization signal SYN, the control processor FPGA / CPLD performs a third delay compensation process on the second synchronization signal SYN.

[0124] Referring to Figure 10, it is a schematic diagram of the process of transmitting a second synchronization signal from a target dynamic vision sensor to other dynamic vision sensors according to an embodiment of this application. As shown in Figure 10, taking two dynamic vision sensors as an example, after the target dynamic vision sensor sends out the second synchronization signal SYN, the second synchronization signal SYN will sequentially pass through the target dynamic serializer (i.e., the GMSL serializer connected to dynamic vision sensor No. 1 as shown in Figure 1) and the target dynamic deserializer (i.e., the GMSL deserializer connected to dynamic vision sensor No. 1 as shown in Figure 1) to the control processor; then, after the control processor performs a third delay compensation process on the second synchronization signal SYN, it will sequentially send it to the remaining dynamic vision sensors through other dynamic deserializers (i.e., the GMSL deserializers connected to other dynamic vision sensors as shown in Figure 1) and other dynamic serializers (i.e., the GMSL serializers connected to other dynamic vision sensors as shown in Figure 1).

[0125] The following will further describe how to perform a third delay compensation process on the second synchronization signal. Referring to Figure 11, the second synchronization signal is sent to the remaining dynamic vision sensors after undergoing the third delay compensation process, including the following steps 1101 to 1103.

[0126] Step 1101: Obtain the changing pulse phase when the other dynamic vision sensors receive the second synchronization signal.

[0127] Step 1102: Generate the third delay compensation processing time based on the changing pulse phase.

[0128] Steps 1101 to 1102 are described in detail below.

[0129] For the phase alignment of the second synchronization signal SYN between dynamic vision sensors, the third delay compensation processing time corresponding to its phase compensation calculation method differs from the calculation methods for the first and second delay compensation processing times described above. This is because, compared to the first synchronization time period (typically tens of milliseconds) for a regular vision sensor to generate the first synchronization signal Vsync, the time period for a dynamic vision sensor to generate the second synchronization signal SYN is very short (typically on the microsecond level). Furthermore, the minimum granularity of the fixed delay settings in the chips of the GMSL serializer and GMSL deserializer is also on the microsecond level, with delays configurable from several microseconds to hundreds of microseconds. The values ​​of t0+t1+t2+t3 in the formula for calculating the first delay compensation processing time already exceed the time period of a single second synchronization signal SYN. ​​Therefore, for the synchronization of multiple dynamic vision sensors, it is only necessary to ensure that the SYN signals of the target dynamic vision sensor and the other dynamic vision sensors are phase aligned.

[0130] Therefore, to ensure phase alignment of the SYN signals of the target dynamic vision sensor and the other dynamic vision sensors, it is first necessary to acquire the phase changes of the pulses received by the other dynamic vision sensors when they receive the second synchronization signal SYN. ​​It is understandable that after the GPIO delays of the serializer and deserializer are set, the phase difference of the SYN signals between the target dynamic vision sensor and the other dynamic vision sensors is also fixed. Therefore, it is only necessary to measure the phase changes of the pulses and the phase difference between the target dynamic vision sensor and the other dynamic vision sensors during debugging, which is sufficient for practical applications.

[0131] Then, based on the different pulse phase changes between the target dynamic vision sensor and the other dynamic vision sensors, a third delay compensation processing time corresponding to each of the other dynamic vision sensors is generated, as described below.

[0132] In some embodiments, a third delay compensation processing time is generated based on the changing pulse phase, including steps 1201 to 1203.

[0133] Step 1201: Obtain the phase difference of the second synchronization signal between the target dynamic vision sensor and the other dynamic vision sensors.

[0134] Step 1202: When the changing pulse phase is delayed, the second synchronization time period of the second synchronization signal generated by the target dynamic vision sensor is obtained, and the third delay compensation processing time is obtained based on the difference between the second synchronization time period and the phase difference of the second synchronization signal.

[0135] Step 1203: When the changing pulse phase is ahead of the pulse phase, the third delay compensation processing time is obtained based on the phase difference of the second synchronization signal.

[0136] Steps 1201 to 1202 are described in detail below.

[0137] In some embodiments, the second synchronization signal phase difference ΔT_dly between the target dynamic vision sensor and each of the remaining dynamic vision sensors is first obtained.

[0138] Then, for each of the remaining dynamic vision sensors, when the change pulse phase between the target dynamic vision sensor and the dynamic vision sensor is delayed, the second synchronization time period ΔTp of the second synchronization signal generated by the target dynamic vision sensor is further obtained, and based on the difference between the second synchronization time period and the phase difference of the second synchronization signal, the third delay compensation processing time is obtained as Δt3=ΔTp-ΔT_dly.

[0139] When the phase of the changing pulse between the target dynamic vision sensor and the dynamic vision sensor is ahead of the pulse phase, the third delay compensation processing time is obtained based on the phase difference of the second synchronization signal as Δt3=ΔT_dly.

[0140] Referring to Figure 13, it is a schematic diagram of the signal waveform curve of a dynamic vision sensor receiving a second synchronization signal according to an embodiment of this application. As shown in Figure 13, taking the synchronization SYN signal waveforms of three dynamic vision sensors as an example, compared with the waveform of the synchronization signal SYN of target dynamic vision sensor 1 (i.e., channel 1), the waveform of the synchronization signal SYN of dynamic vision sensor 2 is delayed (i.e., the rising edge pulse phase of the changing pulse is delayed), and the phase difference of the second synchronization signal between them is ΔT_dly1; conversely, the waveform of the synchronization signal SYN of dynamic vision sensor 3 is advanced (i.e., the rising edge pulse phase of the changing pulse is advanced), and the phase difference of the second synchronization signal between sensor 3 and sensor 1 is ΔT_dly2. Then, the corresponding third delay compensation processing time Δt3 is calculated according to the corresponding situation.

[0141] Step 1103: Perform third delay compensation processing on the second synchronization signal based on the third delay compensation processing time, and send the second synchronization signal after the third delay compensation processing to the other dynamic vision sensors.

[0142] Step 1103 will be described in detail below.

[0143] In some embodiments, after obtaining the third delay compensation processing time Δt3 corresponding to each of the remaining dynamic vision sensors, the control processor performs third delay compensation processing on the second synchronization signal SYN according to each third delay compensation processing time Δt3, and sends the second synchronization signal SYN after the third delay compensation processing to each corresponding dynamic vision sensor in sequence through the GMSL deserializer and GMSL serializer connected to the remaining dynamic vision sensors, thereby ensuring the phase synchronization of multiple dynamic vision sensors (i.e., rising edge or falling edge phase alignment), and thus enabling synchronous data information acquisition.

[0144] Through steps 1101 to 1103 and steps 1201 to 1203 above, based on the changing pulse phase situation between the target dynamic vision sensor and each of the other dynamic vision sensors, as well as the phase difference of the second synchronization signal, the third delay compensation processing time used to compensate for the phase synchronization between the target dynamic vision sensor and each of the other dynamic vision sensors is accurately determined. Thus, the third delay compensation time can be used to accurately realize the synchronous image data acquisition of multiple dynamic vision sensors together, thereby improving the synchronization of data acquisition by multiple ordinary and dynamic vision sensors.

[0145] In some embodiments, after all ordinary and dynamic vision sensors in the vision acquisition system are phase-synchronized according to a first synchronization signal or a second synchronization signal, they will be simultaneously acquired for data synchronization, thereby effectively improving the accuracy and synchronization of image data acquisition by multiple vision sensors. The multi-channel ordinary and dynamic vision sensors provided in this embodiment can be applied in long-distance transmission scenarios such as autonomous driving; through the synchronization signals of ordinary and dynamic vision sensors and external event trigger signals, microsecond-level timestamp synchronization of multiple ordinary and dynamic vision sensors under long-distance transmission is achieved, and the synchronization signals, external event trigger signals and sensor data share a single high-speed cable, simplifying hardware design and saving cable costs.

[0146] The present application proposes a long-distance synchronous acquisition method and system for multiple ordinary and dynamic vision sensors. The method is applied to a control processor and includes: first, determining a target ordinary vision sensor from multiple ordinary vision sensors, and determining a target dynamic vision sensor from multiple dynamic vision sensors; then, acquiring a first synchronization signal emitted by the target ordinary vision sensor and sequentially processed by a target ordinary serializer for serialization, a high-speed coaxial cable, and a target ordinary deserializer for deserialization; acquiring a first serialization delay time and a first deserialization delay time generated when the first synchronization signal sequentially passes through the target ordinary serializer; and acquiring the delay time generated when the first synchronization signal sequentially passes through other ordinary serializers. The second serialization delay time generated by the other common deserializers is used to obtain the second deserialization delay time generated by the other common deserializers. The other common deserializers are those connected to other common vision sensors, and the other common deserializers are those connected to other common vision sensors. The first synchronization time period of the target common vision sensor generating the first synchronization signal is obtained. Based on the first synchronization time period, the first serialization delay time, the second serialization delay time, the first deserialization delay time, and the second deserialization delay time are subtracted sequentially to obtain the first delay compensation processing time. The first synchronization signal is then subjected to first delay compensation processing based on the first delay compensation processing time, and the first synchronization signal after first delay compensation processing is sent to the other common vision sensors. Next, the first... The synchronization signal is processed by a series of events: a third serialization delay time (generated when the synchronization signal passes through a serializer connected to the dynamic vision sensor) and a third deserialization delay time (generated when the synchronization signal passes through a deserializer connected to the dynamic vision sensor). Based on the first synchronization time period, the first serialization delay time, the first deserialization delay time, the third serialization delay time, and the third deserialization delay time, a second delay compensation processing time is generated. The first synchronization signal is then processed using this second delay compensation processing time, and the processed first synchronization signal is sent to each dynamic vision sensor. Finally, the second synchronization signal emitted by the target dynamic vision sensor is received, and the changing pulse phase of the pulses received by the other dynamic vision sensors is acquired to obtain the target dynamic... The second synchronization signal phase difference between the visual sensor and the other dynamic visual sensors is used to obtain the second synchronization time period of the target dynamic visual sensor generating the second synchronization signal when the changing pulse phase is delayed. Based on the difference between the second synchronization time period and the phase difference of the second synchronization signal, the third delay compensation processing time is obtained. When the changing pulse phase is advanced, the third delay compensation processing time is obtained based on the phase difference of the second synchronization signal. The second synchronization signal is then subjected to third delay compensation processing based on the third delay compensation processing time, and the second synchronization signal after the third delay compensation processing is sent to the other dynamic visual sensors so that multiple ordinary visual sensors and multiple dynamic visual sensors can perform data synchronous acquisition.

[0147] This application employs multiple conventional vision sensors and multiple dynamic vision sensors simultaneously for image data acquisition. This leverages the advantages of different types of vision sensors, enabling the acquisition of scene texture information and resolving issues such as blurriness or high latency when capturing fast-moving objects, as well as unclear recognition under strong or low light conditions, thereby improving the accuracy of data acquisition. Furthermore, by utilizing a GMSL serializer and GMSL deserializer with configurable fixed processing delays, the processing delay generated when the first synchronization signal is sent from the target conventional vision sensor to each of the other conventional vision sensors is predetermined. Combined with the first synchronization time period for generating the first synchronization signal, the first delay compensation time corresponding to phase synchronization between each of the other conventional vision sensors and the target conventional vision sensor can be accurately determined. This first delay compensation time allows for precise simultaneous image data acquisition by multiple conventional vision sensors, further improving the synchronization of data acquisition. Moreover, by utilizing a GMSL serializer and GMSL deserializer with configurable fixed processing delays, the first synchronization signal is predetermined when the first synchronization signal is sent from the target conventional vision sensor to the target dynamic vision sensor. The processing delay generated during the process, combined with the first synchronization time period for generating the first synchronization signal, allows for precise determination of the second delay compensation time. This second delay compensation time, combined with the timestamp recorded by the external trigger event of the dynamic vision sensor and the alignment of the rising or falling edge of the first synchronization signal for each frame of the ordinary vision sensor, enables time synchronization between ordinary and dynamic vision sensors of different sensor types. This improves the synchronization of data acquisition from multiple ordinary and dynamic vision sensors. Furthermore, considering the changing pulse phase between the target dynamic vision sensor and each of the other dynamic vision sensors, as well as the phase difference of the second synchronization signal, a third delay compensation processing time is precisely determined to compensate for phase synchronization between the target dynamic vision sensor and each of the other dynamic vision sensors. This third delay compensation time allows for precise simultaneous image data acquisition from multiple dynamic vision sensors, further improving the synchronization of data acquisition from multiple ordinary and dynamic vision sensors. This enables multiple vision sensors to acquire data simultaneously, further enhancing the accuracy and synchronization of image data acquisition from multiple vision sensors.

[0148] This application embodiment also provides a long-distance synchronous acquisition system for multiple ordinary and dynamic vision sensors, which can realize the above-mentioned long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors. Referring to FIG14, the device 1400 includes:

[0149] The target sensor determination module 1410 is used to determine the target general vision sensor from a plurality of general vision sensors, and to determine the target dynamic vision sensor from a plurality of dynamic vision sensors.

[0150] The first delay compensation module 1420 is used to acquire the first synchronization signal of the target ordinary vision sensor, and send the first synchronization signal to the other ordinary vision sensors after performing the first delay compensation processing.

[0151] The second delay compensation module 1430 is used to send the first synchronization signal to the target dynamic vision sensor after performing second delay compensation processing.

[0152] The third delay compensation module 1440 is used to receive the second synchronization signal emitted by the target dynamic vision sensor, and send the second synchronization signal to the other dynamic vision sensors after performing the third delay compensation processing, so that multiple ordinary vision sensors and multiple dynamic vision sensors can perform data synchronous acquisition.

[0153] In some embodiments, the first delay compensation module 1420 is further configured to:

[0154] The first synchronization signal is acquired after being emitted by the target ordinary vision sensor and sequentially processed by the target ordinary serializer, the high-speed coaxial cable, and the target ordinary deserializer.

[0155] In some embodiments, the first delay compensation module 1420 is further configured to:

[0156] The first serialization delay time generated when the first synchronization signal passes through the target ordinary serializer and the first deserialization delay time generated when it passes through the target ordinary deserializer are obtained.

[0157] The first delay compensation processing time is generated based on the first serialization delay time and the first deserialization delay time;

[0158] The first synchronization signal is subjected to a first delay compensation process based on the first delay compensation processing time, and the first synchronization signal after the first delay compensation processing is sent to the other ordinary vision sensors.

[0159] In some embodiments, the first delay compensation module 1420 is further configured to:

[0160] The second serialization delay time generated when the first synchronization signal passes through other ordinary serializers and the second deserialization delay time generated when passing through other ordinary deserializers are obtained. Other ordinary serializers are serializers connected to other ordinary vision sensors, and other ordinary deserializers are deserializers connected to other ordinary vision sensors.

[0161] Acquire the first synchronization time period of the first synchronization signal generated by the target's ordinary vision sensor;

[0162] The first delay compensation processing time is obtained by subtracting the first serialization delay time, the second serialization delay time, the first deserialization delay time, and the second deserialization delay time sequentially from the first synchronization time period.

[0163] In some embodiments, the second delay compensation module 1430 is further configured to:

[0164] The third serialization delay time generated when the first synchronization signal passes through the target dynamic serializer and the third deserialization delay time generated when passing through the target dynamic deserializer are obtained.

[0165] The second delay compensation processing time is generated based on the first serialization delay time, the first deserialization delay time, the third serialization delay time, and the third deserialization delay time;

[0166] The first synchronization signal is subjected to second delay compensation processing based on the second delay compensation processing time, and the first synchronization signal after the second delay compensation processing is sent to the target dynamic vision sensor.

[0167] In some embodiments, the third delay compensation module 1440 is further configured to:

[0168] Acquire the changing pulse phase when the other dynamic vision sensors receive the second synchronization signal;

[0169] The third delay compensation processing time is generated based on the changing pulse phase;

[0170] The second synchronization signal is subjected to third delay compensation processing based on the third delay compensation processing time, and the second synchronization signal after the third delay compensation processing is sent to the other dynamic vision sensors.

[0171] In some embodiments, the third delay compensation module 1440 is further configured to:

[0172] Acquire the phase difference of the second synchronization signal between the target dynamic vision sensor and the other dynamic vision sensors;

[0173] When the changing pulse phase is delayed, the second synchronization time period of the second synchronization signal generated by the target dynamic vision sensor is obtained, and the third delay compensation processing time is obtained based on the difference between the second synchronization time period and the phase difference of the second synchronization signal.

[0174] When the changing pulse phase is ahead of the pulse phase, the third delay compensation processing time is obtained based on the phase difference of the second synchronization signal.

[0175] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, the specific implementation of the long-distance synchronous acquisition system of multiple ordinary and dynamic vision sensors is basically the same as the specific implementation of the long-distance synchronous acquisition method of multiple ordinary and dynamic vision sensors, and will not be repeated here.

[0176] In this embodiment, the long-distance synchronous acquisition method and system for multiple ordinary and dynamic vision sensors employs multiple ordinary vision sensors and multiple dynamic vision sensors to acquire image data simultaneously. This leverages the advantages of different types of vision sensors, enabling the acquisition of scene texture information and resolving issues such as blurriness or high latency when capturing fast-moving objects, as well as unclear recognition under strong or weak light conditions, thereby improving the accuracy of data acquisition. Furthermore, by utilizing a GMSL serializer and GMSL deserializer with configurable fixed processing delays, the processing delay generated by the first synchronization signal being sent from the target ordinary vision sensor to each of the other ordinary vision sensors is predetermined. Combined with the first synchronization time period for generating the first synchronization signal, the first delay compensation time corresponding to phase synchronization between each of the other ordinary vision sensors and the target ordinary vision sensor can be accurately determined. This first delay compensation time allows for precise simultaneous image data acquisition by multiple ordinary vision sensors, further improving the synchronization of data acquisition. Moreover, by utilizing a GMSL serializer and GMSL deserializer with configurable fixed processing delays, the processing delay generated by the first synchronization signal being sent from the target ordinary vision sensor is predetermined. The processing delay generated when the signal is sent to the target dynamic vision sensor, combined with the first synchronization time period for generating the first synchronization signal, allows for precise determination of the second delay compensation time. This second delay compensation time, combined with the timestamp recorded by the external trigger event of the dynamic vision sensor and the rising or falling edge alignment of the first synchronization signal for each frame of the ordinary vision sensor, enables time synchronization between ordinary and dynamic vision sensors of different sensor types, thereby improving the synchronization of data acquisition from multiple ordinary and dynamic vision sensors. Furthermore, considering the changing pulse phase between the target dynamic vision sensor and each of the other dynamic vision sensors, as well as the phase difference of the second synchronization signal, a third delay compensation processing time is precisely determined to compensate for phase synchronization between the target dynamic vision sensor and each of the other dynamic vision sensors. This third delay compensation time allows for precise simultaneous image data acquisition from multiple dynamic vision sensors, further improving the synchronization of data acquisition from multiple ordinary and dynamic vision sensors. This enables multiple vision sensors to acquire data simultaneously, further enhancing the accuracy and synchronization of image data acquisition from multiple vision sensors.

[0177] This application also provides an electronic device, including:

[0178] At least one memory;

[0179] At least one processor;

[0180] At least one program;

[0181] The program is stored in a memory, and the processor executes the at least one program to implement the long-distance synchronous acquisition method of multiple ordinary and dynamic vision sensors described above in this application. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and autonomous driving domain controllers.

[0182] Please refer to Figure 15, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes:

[0183] The processor 1501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0184] The memory 1502 can be implemented using NOR FLASH, NAND FLASH, EEPROM, or other similar formats. The memory 1502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1502 and is called and executed by the processor 1501 to execute the long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors according to the embodiments of this application.

[0185] The input / output interface 1503 is used to implement information input and output;

[0186] The communication interface 1504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0187] Bus 1505 transmits information between various components of the device (e.g., processor 1501, memory 1502, input / output interface 1503, and communication interface 1504);

[0188] The processor 1501, memory 1502, input / output interface 1503 and communication interface 1504 are connected to each other within the device via bus 1505.

[0189] This application embodiment also provides a storage medium, which is a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described long-distance synchronous acquisition method of multiple ordinary and dynamic vision sensors.

[0190] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0191] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0192] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0193] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0195] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0196] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0197] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0198] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0199] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0200] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0201] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A long-distance synchronous acquisition method using multiple ordinary and dynamic vision sensors, characterized in that, The method is applied to a control processor, and the method includes: Determining a target from multiple conventional vision sensors and determining a target from multiple dynamic vision sensors; Acquire the first synchronization signal of the target ordinary vision sensor, and send the first synchronization signal to the other ordinary vision sensors after performing a first delay compensation process; After the first synchronization signal undergoes a second delay compensation process, it is sent to the external event trigger pin of each of the dynamic vision sensors; The system receives a second synchronization signal from the target dynamic vision sensor, performs a third delay compensation process on the second synchronization signal, and then sends it to the other dynamic vision sensors so that the multiple ordinary vision sensors and the multiple dynamic vision sensors can synchronously acquire data.

2. The long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors according to claim 1, characterized in that, Each of the conventional vision sensors is connected to the control processor via a serializer, a high-speed coaxial cable, and a deserializer. Acquiring the first synchronization signal of the target conventional vision sensor includes: The first synchronization signal is obtained after being emitted by the target ordinary vision sensor and sequentially processed by the target ordinary serializer, the high-speed coaxial cable, and the target ordinary deserializer. The target ordinary serializer is the serializer connected to the target ordinary vision sensor, and the target ordinary deserializer is the deserializer connected to the target ordinary vision sensor.

3. The long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors according to claim 2, characterized in that, The step of sending the first synchronization signal to the other ordinary vision sensors after performing a first delay compensation process includes: The first serialization delay time generated when the first synchronization signal passes through the target ordinary serializer and the first deserialization delay time generated when it passes through the target ordinary deserializer are obtained. A first delay compensation processing time is generated based on the first serialization delay time and the first deserialization delay time; The first synchronization signal is subjected to a first delay compensation process based on the first delay compensation processing time, and the first synchronization signal after the first delay compensation processing is sent to the other ordinary vision sensors.

4. The long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors according to claim 3, characterized in that, The step of generating a first delay compensation processing time based on the first serialization delay time and the first deserialization delay time includes: The second serialization delay time generated when the first synchronization signal passes through other ordinary serializers and the second deserialization delay time generated when passing through other ordinary deserializers are obtained. The other ordinary serializers are the serializers connected to other ordinary vision sensors, and the other ordinary deserializers are the deserializers connected to other ordinary vision sensors. Obtain the first synchronization time period for the first synchronization signal generated by the target ordinary vision sensor; The first delay compensation processing time is obtained by subtracting the first serialization delay time, the second serialization delay time, the first deserialization delay time, and the second deserialization delay time sequentially from the first synchronization time period.

5. The long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors according to claim 3, characterized in that, Each of the dynamic vision sensors is connected to the control processor via a serializer, a high-speed coaxial cable, and a deserializer. The step of sending the first synchronization signal to each of the dynamic vision sensors after performing a second delay compensation process includes: The third serialization delay time generated when the first synchronization signal passes sequentially through the serializer connected to the dynamic vision sensor and the third deserialization delay time generated when the dynamic vision sensor is connected to the deserializer are obtained. A second delay compensation processing time is generated based on the first synchronization time period, the first serialization delay time, the first deserialization delay time, the third serialization delay time, and the third deserialization delay time; The first synchronization signal is subjected to a second delay compensation process based on the second delay compensation processing time, and the first synchronization signal after the second delay compensation processing is sent to each of the dynamic vision sensors.

6. The long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors according to claim 2, characterized in that, The step of sending the second synchronization signal to the remaining dynamic vision sensors after performing a third delay compensation process includes: Obtain the changing pulse phase of the remaining dynamic vision sensors when they receive the second synchronization signal; The third delay compensation processing time is generated based on the changed pulse phase; The second synchronization signal is subjected to a third delay compensation process based on the third delay compensation processing time, and the second synchronization signal after the third delay compensation processing is sent to the other dynamic vision sensors.

7. The long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors according to claim 6, characterized in that, The process of generating a third delay compensation time based on the changed pulse phase includes: Obtain the phase difference of the second synchronization signal between the target dynamic vision sensor and the other dynamic vision sensors; When the changing pulse phase is delayed, the second synchronization time period of the target dynamic vision sensor generating the second synchronization signal is obtained, and the third delay compensation processing time is obtained based on the difference between the second synchronization time period and the phase difference of the second synchronization signal. When the changing pulse phase is ahead of the pulse phase, the third delay compensation processing time is obtained based on the phase difference of the second synchronization signal.

8. The long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors according to claim 2, characterized in that, The serial conversion and deserialization processes of the dynamic vision sensor are implemented based on the Tunnel mode of the GMSL chip.

9. The long-distance synchronous acquisition method for multiple ordinary and dynamic vision sensors according to claim 1, characterized in that, Data transmission between the multiple conventional vision sensors, the multiple dynamic vision sensors, and the control processor is achieved via a single high-speed COAX cable.

10. A long-distance synchronous acquisition system for multiple ordinary and dynamic vision sensors, characterized in that, The multi-channel general and dynamic vision sensor includes multiple general vision sensors, multiple dynamic vision sensors, and a control processor. The device is applied to the control processor and includes: A target sensor determination module is used to determine a target general vision sensor from a plurality of general vision sensors, and to determine a target dynamic vision sensor from a plurality of dynamic vision sensors; The first delay compensation module is used to acquire the first synchronization signal of the target ordinary vision sensor, and send the first synchronization signal to the other ordinary vision sensors after performing a first delay compensation process. The second delay compensation module is used to send the first synchronization signal to each of the dynamic vision sensors after performing a second delay compensation process. The third delay compensation module is used to receive the second synchronization signal emitted by the target dynamic vision sensor, and send the second synchronization signal to the other dynamic vision sensors after performing the third delay compensation processing, so that the multiple ordinary vision sensors and the multiple dynamic vision sensors can perform data synchronous acquisition.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the long-distance synchronous acquisition method of multiple ordinary and dynamic vision sensors as described in any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the long-distance synchronous acquisition method of multiple ordinary and dynamic vision sensors as described in any one of claims 1 to 9.

Images