Data alignment method, system, and vehicle-mounted terminal

Abstract

A data alignment method, a system, and a vehicle-mounted terminal. The data alignment method comprises: acquiring world time information and selecting a constraint mode for a trigger signal; on the basis of the world time information and the constraint mode, outputting the trigger signal to corresponding sensing devices, so that the sensing devices acquire preliminary sensing data on the basis of the trigger signal; and on the basis of the constraint mode, selecting a corresponding alignment mode to perform alignment processing on the preliminary sensing data, so as to acquire final sensing data. The present application solves the problem that a back-end sensing fusion application cannot obtain accurate sensing information due to mismatch between sensing content of sensing devices.

Description

A data alignment method, system and vehicle terminal Technical Field

[0001] This application belongs to the field of intelligent assisted driving technology, specifically relating to a data alignment method, system, and vehicle terminal. Background Technology

[0002] Advanced driver assistance systems (ADAS) and autonomous driving systems utilize various sensors, such as millimeter-wave radar (Radar), lidar (Lidar), and cameras, to detect the surrounding environment in order to perceive the driving environment and perform planning and control. However, the frame rates perceived by different types of sensors are not entirely the same; for example, radar typically operates at 20Hz, lidar at 10Hz, and cameras at 30Hz. The start time of detection for each sensor is determined by itself and is not synchronized with each other. Even if the electronic control unit (ECU) synchronizes the time of perception for each sensor, the time information acquired by the ECU is often the time when the perception data arrives at the ECU. The content perceived by each sensor is mismatched, resulting in inaccurate perception information for backend perception fusion applications. This hinders the realization of advanced driver assistance applications and, to some extent, impedes the rapid development of autonomous driving. Summary of the Invention

[0003] This application proposes a data alignment method, system, and vehicle terminal, which aims to provide an effective solution for the inability to obtain accurate sensing data due to the mismatch of sensing frequencies of various sensing devices, and to perform alignment processing on the sensing data collected by multi-mode sensing devices.

[0004] Specifically, this application proposes a data alignment method, comprising: acquiring world time information and selecting a constraint method for a trigger signal; outputting a trigger signal to a corresponding sensing device according to the world time information and the constraint method, so that each sensing device acquires preliminary sensing data based on the trigger signal; and performing alignment processing on each preliminary sensing data by selecting a corresponding alignment method based on the constraint method to obtain final sensing data.

[0005] In the above technical solution, acquiring the world time information ensures that the multimodal sensing devices operate on a unified time basis, guaranteeing the consistency of the sensing data. Alignment processing of each sensing data point based on the alignment method corresponding to the selected trigger signal effectively aligns the data sensed by the multimodal sensing devices, ensuring the accuracy and reliability of data alignment. The constraint-based trigger signal effectively improves data acquisition efficiency and enhances the collaborative working efficiency of the sensing devices.

[0006] As one implementation method, the constraint method includes at least a first constraint method and a second constraint method; the first constraint method specifically includes taking the maximum interval time of the number of sensing frames in the sensing device as a first trigger time interval, and outputting a trigger signal based on the first trigger time interval.

[0007] The first constraint method selects the maximum interval between sensing frames in the sensing device as the trigger time, which avoids wasting computing resources or storage space due to excessively frequent triggering and reduces unnecessary data collection. Triggering the signal based on the maximum interval time can solve the problem of asynchronous triggering timing between sensing devices due to differences in frame rates. Selecting the maximum frame interval time as the trigger period ensures that the trigger signals of all sensing devices are synchronized, reducing data inconsistencies caused by device differences. Triggering the signal through the first constraint method can reduce data misalignment and timing errors, ensuring the consistency and accuracy of data collection.

[0008] Furthermore, the second constraint method specifically includes setting the trigger interval of each sensing device to the minimum interval of the number of sensing frames in the sensing device as the second trigger interval, setting the trigger interval of each sensing device to a multiple of the minimum time of the second trigger interval, and outputting a trigger signal based on the second trigger interval.

[0009] By selecting the minimum interval between sensing frames in the sensing devices as the second trigger interval, sensing data can be acquired more quickly, making it suitable for scenarios requiring more frequent responses and real-time data acquisition. Setting the trigger interval for each sensing device as a multiple of the minimum interval ensures consistency in their trigger cycles, improving coordination and synchronization. Appropriate multiplier adjustments ensure all sensing devices collect data at uniform time intervals, thereby enhancing the robustness of sensory data acquisition.

[0010] Furthermore, the output trigger signal to the corresponding sensing device includes at least a first triggering method and a second triggering method; the first triggering method specifically includes transmitting the trigger signal to each sensing device through the triggering interface of each sensing device, so as to control each sensing device to trigger a sensing operation at a preset time based on the trigger signal; the second triggering method specifically includes transmitting the trigger signal to each sensing device based on a time synchronization allocation method, so as to control each sensing device to trigger a sensing operation at a preset time based on the trigger signal.

[0011] The first triggering method transmits the trigger signal through the trigger interface of each sensing device. Each sensing device has an independent trigger interface, improving the independence of the sensing data acquisition process. Each sensing device is triggered in an independent timing sequence, making it suitable for scenarios requiring independent operation of each sensing device. This first triggering method enhances the flexibility of sensing data acquisition. If a sensing device malfunctions while receiving or executing a trigger, other sensing devices can still operate independently, improving the fault tolerance of sensing data acquisition.

[0012] By employing the second triggering method, and through time synchronization allocation, the triggering operation of each sensing device can be performed at a predetermined precise time. This is suitable for applications requiring high-precision time control, avoiding triggering errors caused by asynchronous clocks of sensing devices. It ensures the consistency and accuracy of the sensed data.

[0013] Furthermore, each sensing device acquires preliminary sensing data based on the trigger signal, including: when the sensing device receives the trigger signal, it transmits a sensing signal and records the number of transmitted sensing frames; the sensing device receives the sensing echo signal and converts the sensing echo signal into environmental data; and the environmental data and the number of sensing frames are packaged and processed to obtain the preliminary sensing data.

[0014] By recording the number of sensing frames from each sensing device, timestamps can be provided for the sensing data, enabling precise management of each frame and ensuring consistency in data processing and analysis, preventing data loss or sequence errors. Packaging the environmental data and sensing frame counts makes data transmission and storage more efficient, improving data transfer efficiency. Recording the sensing frame counts also ensures that data collected at the same time point can be correlated and synchronized, avoiding data errors caused by clock differences, ensuring that each frame of sensing data is completely recorded, and guaranteeing the integrity and reliability of the sensing data.

[0015] Furthermore, the alignment method includes at least a frame-by-frame alignment method and a minimum interval alignment method; the frame-by-frame alignment method specifically includes: if the constraint method is a first constraint method, then the preliminary sensing data acquired by each sensing device is sorted according to the number of sensing frames, and the sorted preliminary sensing data is aligned frame-by-frame based on the number of sensing frames.

[0016] By aligning the sorted initial sensing data frame by frame, precise alignment of data from various sensing devices is ensured during data fusion, thus resolving the time asynchrony issue caused by inconsistent sampling frequencies of different sensors and improving the timeliness and accuracy of the sensing data. Frame-by-frame alignment coordinates the working rhythm of each sensing device, allowing data from different sources to be processed and fused simultaneously. This synchronization helps eliminate systematic errors caused by time differences, thereby improving overall system performance.

[0017] Furthermore, the minimum interval alignment method specifically includes: if the constraint method is the second constraint method, then the number of sensing frames with the smallest time interval among all sensing devices is used to calculate and obtain the alignment frame number of the sensing devices, and the preliminary sensing data is aligned according to the alignment frame number; wherein, obtaining the alignment frame number of the sensing devices specifically includes calculating based on the number of sensing frames with the smallest time interval, a multiple of the smallest time interval, and a preset value to obtain an integer quotient, and using the integer quotient as the alignment frame number.

[0018] By aligning the sensed data using the minimum interval alignment method, the temporal consistency and accuracy of the sensed data are improved. The number of aligned frames is calculated using a multiple of the minimum time interval and a preset value, reducing data errors. This minimizes data deviations caused by time errors between different sensing devices, ensuring the accuracy of the sensed data.

[0019] Furthermore, the alignment process for each preliminary sensing data also includes: determining whether there are any unaligned sensing frames in the aligned preliminary sensing data; if so, inserting data frames into the preliminary sensing data for alignment; otherwise, ignoring them.

[0020] By inserting data frames into the initial sensing data for alignment, the effective alignment and integrity of the initial sensing data are ensured. Inserting data frames to align misaligned sensing data improves the accuracy and reliability of the sensing data alignment process.

[0021] Furthermore, after aligning the preliminary sensing data, the method further includes: extracting features from the preliminary sensing data based on a preset feature extraction algorithm to obtain multiple feature data; and fusing the multiple feature data based on a preset sensing fusion algorithm to obtain the final sensing data.

[0022] By extracting features from the aligned preliminary sensing data, key information can be effectively extracted and redundant data removed, thereby improving data accuracy. Furthermore, feature data fusion processing integrates data from different sensing devices, resulting in more comprehensive and accurate final sensing data. This overcomes the limitations of a single sensing device and improves sensing accuracy.

[0023] Based on the same inventive concept, this application also proposes a data alignment system, which includes at least multiple vehicle-mounted sensing devices, a global navigation satellite system, and a control unit; the multiple vehicle-mounted sensing devices are used to collect preliminary sensing data of the environment; the global navigation satellite system is used to acquire world time information and transmit it to the control unit; the control unit is used to constrain the multiple vehicle-mounted sensing devices according to the world time information and to perform alignment processing on the preliminary sensing data collected by the multiple vehicle-mounted sensing devices.

[0024] Furthermore, the control unit includes at least a time controller, a memory, and a processor; the time controller is used to determine the constraint method of the trigger signal based on the world time information, and output the trigger signal to each vehicle-mounted sensing device based on the constraint method; the memory is used to store computer instructions for each functional module that performs data alignment processing on the preliminary sensing data collected by each vehicle-mounted sensing device based on the world time information and the constraint method; the processor communicates with the memory and is used to execute the computer instructions of each functional module in the memory.

[0025] Furthermore, the functional module includes at least: a data perception module and a data alignment module; the data perception module includes computer instructions for acquiring preliminary perception data transmitted by each vehicle-mounted perception device; the data alignment module includes computer instructions for selecting a corresponding alignment method based on the constraint method to perform alignment processing on each preliminary perception data and acquire final perception data.

[0026] Furthermore, the data alignment module includes at least a first alignment module and a second alignment module; the first alignment module is configured to, when the constraint method is the first constraint method, sort the preliminary sensing data according to the number of sensing frames, and perform frame-by-frame alignment processing on the sorted preliminary sensing data based on the number of sensing frames.

[0027] The second alignment module includes a function for calculating and obtaining an alignment frame number based on the number of sensing frames with the smallest time interval when the constraint method is the second constraint method, and performing alignment processing on the preliminary sensing data according to the alignment frame number.

[0028] Furthermore, the functional module may also include a data fusion module, which includes computer instructions for extracting features from each preliminary sensing data based on a preset feature extraction algorithm to obtain multiple feature data; and for fusing the multiple feature data based on a preset sensing fusion algorithm to obtain the final sensing data.

[0029] Based on the same inventive concept, this application also proposes an in-vehicle terminal, which includes at least a data alignment system. When the data alignment system is running, it implements the data alignment method to perform data alignment on preliminary sensing data collected by multiple in-vehicle sensing devices.

[0030] Compared with the prior art, this application has at least the following beneficial effects:

[0031] This application provides an effective solution to the problem of inaccurate sensing data caused by mismatched frequencies of sensing data collected by various sensing devices. Based on acquired world time information and selected constraint methods, it outputs trigger signals to control the timing of data collection by the sensing devices, ensuring that multiple sensing devices operate on a unified time basis and guaranteeing data consistency. By aligning the sensing data according to the selected alignment method corresponding to the trigger signals, the data sensed by the multi-mode sensing devices can be effectively aligned, ensuring the accuracy and reliability of data alignment. The constraint-based trigger signals effectively improve data acquisition efficiency and enhance the collaborative working efficiency of the sensing devices. Attached Figure Description

[0032] Figure 1 is a flowchart illustrating a data alignment method according to an embodiment of this application.

[0033] Figure 2 is a schematic diagram of the first constraint trigger signal shown in an embodiment of this application.

[0034] Figure 3 is a schematic diagram of the second constraint trigger signal shown in an embodiment of this application.

[0035] Figure 4 is a schematic diagram of a data alignment system shown in an embodiment of this application.

[0036] Figure 5 is a schematic diagram of the control unit shown in an embodiment of this application.

[0037] Figure 6 is a schematic diagram of the memory structure shown in an embodiment of this application.

[0038] Figure 7 is a schematic diagram of the structure of the vehicle terminal shown in an embodiment of this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0040] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices. Example 1:

[0041] Please refer to Figure 1. The data alignment method mainly includes steps S100 to S300.

[0042] Step S100 includes acquiring world time information and selecting a constraint method for the trigger signal. This can primarily be achieved by connecting a GNSS (Global Navigation Satellite System) to an ECU (Electronic Control Unit), through which the ECU acquires the world time information from the GNSS. The ECU and GNSS are mainly connected via a serial communication interface. For example, the ECU and GNSS can be connected using an RS232 (Recommended Standard 232) communication interface.

[0043] Step S200 includes: outputting a trigger signal to the corresponding sensing device according to the world time information and constraint method, so that each sensing device acquires preliminary sensing data based on the trigger signal. Each sensing device may include at least a millimeter-wave radar, a lidar, and a camera. By sending the trigger signal to the millimeter-wave radar, lidar, and camera, the device is controlled to collect current environmental data. The millimeter-wave radar, lidar, and camera may be one or more. Those skilled in the art can select other sensing devices according to the actual situation, and are not limited to this. By outputting the trigger signal to the sensing device, the timing of data acquisition by each sensing device is controlled.

[0044] Step S300 includes: selecting a corresponding alignment method based on the constraint method to align each preliminary perception data to obtain final perception data. The constraint method mainly includes a first constraint method and a second constraint method, and the alignment method mainly includes frame-by-frame alignment and minimum interval alignment. When the first constraint method is selected, the frame-by-frame alignment method is used to align each perception data; when the second constraint method is selected, the minimum interval alignment method is used to align each perception data. After aligning the preliminary perception data, the process may further include fusing the aligned preliminary perception data to obtain more comprehensive final perception data. This final perception data can be input into the intelligent driving assistance system and the autonomous driving system. Since the final perception data is obtained by fusing aligned preliminary perception data collected from multiple sensing devices, the limitations of a single sensing device are avoided. Based on this aligned and fused final perception data, the autonomous driving system can accurately identify and track pedestrians, other vehicles, traffic signs, traffic lights, etc., on the road, and calculate the distance, speed, and direction of movement to the target, thereby performing more accurate path planning. Meanwhile, by aligning and fusing the initial perception data collected by multiple sensing devices, the autonomous driving system can adapt to complex driving environments through the final perception data, such as driving conditions in adverse weather conditions like fog, haze, rain, and snow, thus ensuring safe driving.

[0045] In some embodiments, the constraint method includes at least a first constraint method and a second constraint method; the first constraint method specifically includes using the maximum interval time of the number of sensing frames in the sensing device as a first trigger time interval, and outputting a trigger signal based on the first trigger time interval.

[0046] The sensing devices mentioned above, taking millimeter-wave radar, lidar, and a camera as examples, refer to Figure 2. The sensing frame rate of the millimeter-wave radar is 20Hz, the lidar is 10Hz, and the camera is 30Hz. That is, the camera has the shortest frame rate interval, and the lidar has the longest frame rate interval. Therefore, the lidar's frame rate interval is used as the trigger signal for the first trigger time interval.

[0047] Preferably, the second constraint method specifically includes setting the trigger interval of each sensing device to the minimum interval of the number of sensing frames in the sensing device as the second trigger interval, setting the trigger interval of each sensing device to a multiple of the minimum time of the second trigger interval, and outputting a trigger signal based on the second trigger interval.

[0048] Please refer to Figure 3, taking millimeter-wave radar, lidar, and a camera as examples of data collection devices. The camera has the shortest frame interval; therefore, the trigger intervals for the millimeter-wave radar and lidar are set as a multiple of the camera's shortest frame interval, and the camera's time interval is used as the second trigger interval to output the trigger signal.

[0049] Preferably, the output trigger signal to the corresponding sensing device includes at least a first triggering method and a second triggering method; the first triggering method specifically includes transmitting the trigger signal to each sensing device through the triggering interface of each sensing device, so as to control each sensing device to trigger a sensing operation at a preset time based on the trigger signal; the second triggering method specifically includes transmitting the trigger signal to each sensing device based on a time synchronization allocation method, so as to control each sensing device to trigger a sensing operation at a preset time based on the trigger signal.

[0050] Each sensing device is equipped with an independent trigger interface. When using the first triggering method, the trigger signal is transmitted to the corresponding sensing device through the trigger interface, thereby controlling the sensing device to trigger sensing operations. For example, a trigger signal is transmitted to the millimeter-wave radar device through its trigger interface, controlling the transmitter and receiver of the millimeter-wave radar to enter working mode, thereby collecting environmental data. When using the second triggering method, trigger signals are transmitted to each sensing device simultaneously based on a time-synchronized allocation method. For example, trigger signals are simultaneously output to the millimeter-wave radar, lidar, and camera devices, simultaneously controlling the millimeter-wave radar, lidar, and camera devices to enter working mode and collect environmental data.

[0051] Preferably, the acquisition of preliminary sensing data by each sensing device based on the trigger signal includes: when the sensing device receives the trigger signal, it transmits a sensing signal and records the number of transmitted sensing frames; the sensing device receives a sensing echo signal and converts the sensing echo signal into environmental data; and the environmental data and the number of sensing frames are packaged together to obtain the preliminary sensing data.

[0052] Taking a sensing device employing millimeter-wave radar as an example, the millimeter-wave radar can be connected to the ECU. There can be one or more millimeter-wave radars. When the controller in the millimeter-wave radar receives a trigger signal, it transmits millimeter-wave signals and records the number of millimeter-wave frames. It receives echo information, converts it into environmental data, packages the environmental data and the number of millimeter-wave frames to obtain preliminary sensing data, and transmits this preliminary sensing data to the ECU via CAN bus or vehicle Ethernet communication.

[0053] Taking a sensing device using LiDAR as an example, the LiDAR can be connected to the ECU. There can be one or more LiDARs. When the controller of the LiDAR receives a trigger signal, it emits infrared laser and records the number of emission frames. It receives the return light information and converts it into environmental data. The controller of the LiDAR packages the environmental data and the number of emission frames to obtain the preliminary sensing data, and transmits the preliminary sensing data to the ECU through the vehicle Ethernet communication method.

[0054] Taking a sensing device employing a camera as an example, the camera can be connected to an ECU. There can be one or more camera devices. When the camera receives a trigger signal, it activates exposure and records the number of exposure frames. The sensing information is packaged with the number of exposure frames to obtain the preliminary sensing data, which is then transmitted to the ECU via a serial-to-deserializer communication method.

[0055] Optionally, the alignment method includes at least frame-by-frame alignment and minimum interval alignment; the frame-by-frame alignment method specifically includes: if the constraint method is the first constraint method, then sorting the preliminary sensing data acquired by each sensing device according to the number of sensing frames, and performing frame-by-frame alignment processing on the sorted preliminary sensing data based on the number of sensing frames.

[0056] Specifically, the initial sensing data is sorted according to the number of sensing frames in the initial sensing data collected by each sensing device. The sorting method can be either ascending or descending, without restriction. This ensures that subsequent frame-by-frame alignment can be performed in the correct order.

[0057] Optionally, the minimum interval alignment method specifically includes: if the constraint method is the second constraint method, then the number of sensing frames with the smallest time interval among the sensing devices is calculated to obtain the alignment frame number of the sensing devices, and the preliminary sensing data is aligned according to the alignment frame number; wherein, obtaining the alignment frame number of the sensing devices specifically includes calculating based on the number of sensing frames with the smallest time interval, a multiple of the smallest time interval, and a preset value to obtain an integer quotient, and using the integer quotient as the alignment frame number.

[0058] When receiving sensing data transmitted from each sensing device, if the second constraint method is used when outputting trigger signals to each sensing device, then the minimum interval alignment method is adopted, and the calculation is based on the sensing frame number with the smallest time interval among all sensing devices. The specific calculation method is as follows:

[0059]

[0060] The calculated integer quotient is used as the number of alignment frames, and the perceived data is aligned according to the number of alignment frames.

[0061] For example, taking millimeter-wave radar, lidar, and cameras as examples, the time interval between the sensing frames of the camera is the smallest. If the sensing frame count of the camera is 9, and the time interval between the sensing frames of the millimeter-wave radar and the minimum time interval of the camera is 2, then the alignment frame count of the millimeter-wave radar is 4. If the time interval between the sensing frames of the lidar and the minimum time interval of the camera is 3, then the alignment frame count of the lidar is 3.

[0062] Optionally, the alignment process for each preliminary sensing data further includes: determining whether there are any unaligned sensing frames in the aligned preliminary sensing data; if so, inserting data frames into the preliminary sensing data for alignment; otherwise, ignoring them.

[0063] If there are unaligned sensing frames in the aligned sensing data, for example, if there are unaligned sensing frames between the camera's sensing data and the sensing data of other sensing devices, the sensing data can be aligned by inserting data frames into the camera's sensing data.

[0064] Optionally, after aligning the preliminary sensing data, the method further includes: extracting features from the preliminary sensing data based on a preset feature extraction algorithm to obtain multiple feature data; and fusing the multiple feature data based on a preset sensing fusion algorithm to obtain the final sensing data.

[0065] The feature extraction algorithm can employ image processing algorithms, point cloud processing algorithms, time series analysis algorithms, etc., without limitation. Image processing algorithms can extract features such as object edges, color distribution, and shape characteristics of targets in a scene from preliminary perception data collected by camera equipment. Point cloud processing algorithms can extract features such as spatial location and shape of targets in a scene from point cloud data collected by LiDAR or other radar equipment. The perception fusion algorithm can be a weighted fusion algorithm, probabilistic fusion algorithm, etc., without limitation. Deep learning algorithms can be used to construct a data fusion model, which is then used to fuse different feature data.

[0066] Example 2:

[0067] Referring to Figure 4, this application also proposes a data alignment system, comprising at least multiple vehicle-mounted sensing devices 100, a global navigation satellite system 300, and a control unit 400. The multiple vehicle-mounted sensing devices 100 are used to collect preliminary environmental sensing data. The global navigation satellite system 300 is used to acquire world time information and transmit it to the control unit 400. The control unit 400 is used to constrain the multiple vehicle-mounted sensing devices 100 according to the world time information and to perform alignment processing on the preliminary sensing data collected by the multiple vehicle-mounted sensing devices 100. This final sensing data can be input into intelligent driving assistance systems and autonomous driving systems. Since the final sensing data is obtained by fusing preliminary sensing data collected and aligned by multiple sensing devices, the limitations of a single sensing device are avoided. Based on this aligned and fused final sensing data, the autonomous driving system can accurately identify and track pedestrians, other vehicles, traffic signs, traffic lights, etc., on the road, and calculate the distance, speed, and direction of movement to the target, thereby performing more accurate path planning. Meanwhile, by aligning and fusing the initial perception data collected by multiple sensing devices, the autonomous driving system can adapt to complex driving environments through the final perception data, such as driving conditions in adverse weather conditions like fog, haze, rain, and snow, thus ensuring safe driving.

[0068] Further, referring to Figure 5, the control unit 400 includes at least a time controller 410, a memory 420, and a processor 430; the time controller 410 is used to determine the constraint method of the trigger signal based on the world time information, and output the trigger signal to each vehicle-mounted sensing device 100 based on the constraint method; referring to Figure 6, the memory 420 is used to store computer instructions for each functional module that performs data alignment processing on the preliminary sensing data collected by each vehicle-mounted sensing device 100 based on the world time information and the constraint method; the processor 430 communicates with the memory and is used to execute the computer instructions of each functional module in the memory.

[0069] The memory 420 includes, but is not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), and CD-ROM (Compact Disc Read-Only Memory); the processor 430 includes a central processing unit or a device or module with processing capabilities.

[0070] Furthermore, referring to Figure 6, the functional module includes at least: a data perception module 421 and a data alignment module 422; the data perception module 421 includes computer instructions for acquiring preliminary perception data transmitted by each vehicle-mounted perception device 100; the data alignment module 422 includes computer instructions for selecting a corresponding alignment method based on the constraint method to perform alignment processing on each preliminary perception data and acquire final perception data.

[0071] The data alignment module 422 includes at least a first alignment module 4221 and a second alignment module 4222. The first alignment module 4221 is configured to, when the constraint method is a first constraint method, sort the preliminary sensing data according to the number of sensing frames, and perform frame-by-frame alignment processing on the sorted preliminary sensing data based on the number of sensing frames. The second alignment module 4222 is configured to, when the constraint method is a second constraint method, calculate and obtain an alignment frame number based on the number of sensing frames with the smallest time interval, and perform alignment processing on the preliminary sensing data according to the alignment frame number.

[0072] Furthermore, the functional module may also include a data fusion module 423, which includes computer instructions for extracting features from each preliminary sensing data based on a preset feature extraction algorithm to obtain multiple feature data; and for fusing the multiple feature data based on a preset sensing fusion algorithm to obtain the final sensing data.

[0073] Referring to Figure 7, this application also proposes an in-vehicle terminal, which includes at least a data alignment system 200. When the data alignment system 200 is running, it implements the data alignment method as described in Embodiment 1 to perform data alignment on the preliminary sensing data collected by multiple in-vehicle sensing devices 100.

[0074] The plurality of vehicle-mounted sensing devices 100 may further include a first vehicle-mounted sensing device 110, a second vehicle-mounted sensing device 120, and a third vehicle-mounted sensing device 130. The first vehicle-mounted sensing device 110 may include, but is not limited to, one or more millimeter-wave radar devices. The second vehicle-mounted sensing device 120 may include, but is not limited to, one or more lidar devices. The third vehicle-mounted sensing device 130 may include, but is not limited to, one or more cameras. The control unit 400 performs alignment and fusion processing on the sensing data collected by the first vehicle-mounted sensing device 110, the second vehicle-mounted sensing device 120, and the third vehicle-mounted sensing device 130, and outputs the data to other systems for fusion applications. For example, the final sensing data obtained after alignment and fusion processing is input into an autonomous driving system, and the autonomous driving system performs functions such as path planning based on the final sensing data.

[0075] In summary, this application provides an effective solution to the problem of inaccurate sensing data caused by the mismatch in the frequency of sensing data collected by various sensing devices. Based on the acquired world time information and the selected constraint method, it outputs trigger signals to control the timing of data collection by the sensing devices, ensuring that multiple sensing devices operate on a unified time basis and guaranteeing the consistency of the sensing data. By aligning the sensing data according to the alignment method corresponding to the selected trigger signal, the data sensed by the multi-mode sensing devices can be effectively aligned, ensuring the accuracy and reliability of data alignment. The constraint-based trigger signal effectively improves the efficiency of data acquisition and enhances the collaborative working efficiency of the sensing devices.

[0076] In the several embodiments provided in this application, it will be understood that each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0077] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application for those skilled in the art.

Claims

1. A data alignment method, comprising: Obtain world time information and select the constraint method for triggering the signal (S100). Based on the world time information and constraint method, a trigger signal is output to the corresponding sensing device so that each sensing device can acquire preliminary sensing data based on the trigger signal (S200). And, based on the constraint method, select the corresponding alignment method to perform alignment processing on each preliminary sensing data to obtain the final sensing data (S300).

2. The data alignment method according to claim 1, wherein the constraint method includes at least a first constraint method and a second constraint method; The first constraint method specifically includes taking the maximum interval of the number of sensing frames in the sensing device as the first trigger time interval, and outputting a trigger signal based on the first trigger time interval.

3. The data alignment method according to claim 2, wherein the second constraint method specifically includes setting the trigger interval of each sensing device to the minimum interval of the number of sensing frames in the sensing device as the second trigger interval, setting the trigger interval of each sensing device to the minimum time multiple of the second trigger interval, and outputting a trigger signal based on the second trigger interval.

4. The data alignment method according to claim 1, wherein the output trigger signal to the corresponding sensing device includes at least a first triggering method and a second triggering method; The first triggering method specifically includes transmitting the trigger signal to each sensing device through the trigger interface of each sensing device, so as to control each sensing device to trigger a sensing operation at a preset time based on the trigger signal; The second triggering method specifically includes transmitting the trigger signal to each sensing device based on a time synchronization allocation method, so as to control each sensing device to trigger a sensing operation at a preset time based on the trigger signal.

5. The data alignment method according to claim 1, wherein each sensing device acquires preliminary sensing data based on the trigger signal, comprising: When the sensing device receives the trigger signal, it transmits a sensing signal and records the number of sensing frames transmitted. The sensing device receives the sensing echo signal and converts the sensing echo signal into environmental data. The environmental data and the number of perception frames are packaged and processed to obtain the preliminary perception data.

6. The data alignment method according to claim 5, wherein the alignment method includes at least frame-by-frame alignment and minimum interval alignment; The frame-by-frame alignment method specifically includes: If the constraint method is the first constraint method, the preliminary sensing data acquired by each sensing device is sorted according to the number of sensing frames, and the sorted preliminary sensing data is aligned frame by frame based on the number of sensing frames.

7. The data alignment method according to claim 6, wherein the minimum interval alignment method specifically includes: If the constraint method is the second constraint method, then the number of sensing frames with the smallest time interval among all sensing devices is used to calculate the number of aligned frames of the sensing devices, and the preliminary sensing data is aligned according to the number of aligned frames. Specifically, obtaining the alignment frame number of the sensing device includes calculating based on the sensing frame number of the minimum time interval, a multiple of the minimum time interval, and a preset value to obtain an integer quotient, and using the integer quotient as the alignment frame number.

8. The data alignment method according to claim 6, wherein the alignment processing of each preliminary sensing data further includes: Determine whether there are any unaligned sensing frames in the preliminary sensing data after alignment processing. If so, insert data frames into the preliminary sensing data for alignment processing. Otherwise, ignore it.

9. The data alignment method according to claim 8, further comprising, after aligning each preliminary sensing data, the following: Based on a preset feature extraction algorithm, features are extracted from each of the preliminary sensing data to obtain a variety of feature data; The multiple feature data are fused based on a preset perception fusion algorithm to obtain the final perception data.

10. A data alignment system comprising at least a plurality of vehicle-mounted sensing devices (100), a global navigation satellite system (300), and a control unit (400); wherein the plurality of vehicle-mounted sensing devices (100) are used to collect preliminary sensing data of the environment; The global navigation satellite system (300) is used to acquire world time information and transmit it to the control unit (400). The control unit (400) is used to constrain the various vehicle-mounted sensing devices (100) according to the world time information and to perform alignment processing on the preliminary sensing data collected by the various vehicle-mounted sensing devices (100).

11. The data alignment system according to claim 10, wherein the control unit (400) includes at least a time controller (410), a memory (420), and a processor (430). The time controller (410) is used to determine the constraint method of the trigger signal based on the world time information, and output the trigger signal to each vehicle-mounted sensing device (100) based on the constraint method. The memory (420) is used to store computer instructions for each functional module that performs data alignment processing on the preliminary sensing data collected by each vehicle-mounted sensing device (100) based on the world time information and constraint method. The processor (430) communicates with the memory and executes computer instructions for each functional module in the memory.

12. The data alignment system according to claim 11, wherein the functional module includes at least: Data sensing module (421) and data alignment module (422); The data sensing module (421) includes computer instructions for acquiring preliminary sensing data transmitted by each vehicle-mounted sensing device (100); The data alignment module (422) includes computer instructions for selecting a corresponding alignment method based on the constraint method to perform alignment processing on each preliminary sensing data and obtain the final sensing data.

13. The data alignment system according to claim 12, wherein the data alignment module (422) includes at least a first alignment module (4221) and a second alignment module (4222). The first alignment module (4221) includes a function for sorting the preliminary sensing data according to the number of sensing frames when the constraint method is the first constraint method, and performing frame-by-frame alignment processing on the sorted preliminary sensing data based on the number of sensing frames. The second alignment module (4222) includes a function for calculating and obtaining an alignment frame number based on the number of sensing frames with the smallest time interval when the constraint method is the second constraint method, and performing alignment processing on the preliminary sensing data according to the alignment frame number.

14. The data alignment system according to claim 13, wherein the functional module may further include a data fusion module (423), the data fusion module (423) comprising: a computer instruction for extracting features from each preliminary sensing data based on a preset feature extraction algorithm to obtain multiple feature data; and for fusing the multiple feature data based on a preset sensing fusion algorithm to obtain the final sensing data.

15. A vehicle-mounted terminal, the vehicle-mounted terminal comprising at least a data alignment system (200), the data alignment system (200) implementing the data alignment method as described in claim 1 during operation to perform data alignment on preliminary sensing data collected by a plurality of vehicle-mounted sensing devices (100).

Images