Point cloud resolving method, electronic device, and storage medium

Abstract

Disclosed are a point cloud resolving method, an electronic device, and a storage medium, the point cloud resolving method comprising: acquiring a first echo sequence signal captured by LiDAR; acquiring visual information of a heterogeneous sensor; and using the visual information in combination with the first echo sequence signal and echo information of each echo in the first echo sequence signal to determine a second echo sequence signal for point cloud generation.

Description

Point cloud solving method, electronic device and storage medium TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data acquisition, in particular to a point cloud solving method, an electronic device and a storage medium. BACKGROUND

[0002] In related technologies, in a LiDAR system, the principle of the point cloud generation process is that an optical emitting device emits a light beam with a certain scattering angle, the light beam hits the surface of an object in the environment to produce reflected light, an optical receiving device captures the time-intensity curve waveform of the reflected light, and sends it to a point cloud generation unit for echo-point cloud solving. In the point cloud generation step, the point cloud solving unit analyzes a series of echo peak and valley values, echo peak and valley times, half-peak values, half-peak times and other information in the operation device (including but not limited to FPGA / ASIC / DSP) according to the echo of the optical receiving device provided by the underlying signal processing unit, calculates the distance of one or more echoes, and converts it into a point cloud, which is output to the downstream point cloud processing algorithm.

[0003] The inventor found that the defect of the related technology is that only one or more optical devices can be used to infer the situation of the light path reflection object at the same or different time, and there is no more information input.

[0004] Based on the above information input limitation, the point cloud solving unit cannot well infer which echoes are light path interference (including but not limited to: rain, fog, front vehicle exhaust, dust raising) and which echoes are actual objects only according to echo time, echo intensity, echo slope and other information. Especially in the case where the echo intensity of the light path interference is greater than the reflection intensity of the object in the actual environment, only all information of the echo sequence can be discarded. Due to the above information input limitation, the point cloud solving unit cannot well infer the actual situation of the object in the environment corresponding to a series of echoes, so as to generate accurate point clouds. SUMMARY

[0005] Embodiments of the present application provide a point cloud solving method, an electronic device and a storage medium, which are used to at least solve one of the above technical problems.

[0006] In a first aspect, embodiments of the present application provide a point cloud solving method, comprising: acquiring a first echo sequence signal captured by a LiDAR; acquiring visual information of a heterogenous sensor; and determining a second echo sequence signal for point cloud generation by using the visual information in combination with the first echo sequence signal and echo information of each echo in the first echo sequence signal.

[0007] In a second aspect, an embodiment of the present application provides an electronic device, comprising at least one processor, and a memory connected with the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the point cloud solving methods described above.

[0008] In a third aspect, an embodiment of the present application provides a storage medium, wherein the storage medium stores one or more programs including execution instructions, and the execution instructions are readable and executable by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to perform any of the point cloud solving methods described above.

[0009] In a fourth aspect, an embodiment of the present application further provides a computer program product, comprising a computer program stored on a storage medium, and the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer is enabled to perform any of the point cloud solving methods described above.

[0010] In a fifth aspect, an embodiment of the present application further provides a movable platform comprising the electronic device as described in the second aspect.

[0011] The method of the present application determines the echo property by using the visual information of the heterogenous sensor, in combination with one or more information such as echo time, echo intensity, echo slope, etc., and determines which one or which ones of a series of echoes to use for point cloud generation, thereby providing a higher density and higher precision point cloud output. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0013] FIG. 1 is a flowchart of a point cloud solving method according to an embodiment of the present application;

[0014] FIG. 2 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0016] Please refer to FIG. 1, which shows a flow chart of a point cloud solving method according to an embodiment of the present application. The point cloud solving method can exist in a computer in the form of an algorithm, and in subsequent embodiments, it can be referred to as a point cloud solving device, such as a program code or a software, etc., which is not limited in the present application. The point cloud solving method can be used in a laser radar system or a sensor containing a laser radar, or a computing device connected to the sensor, which is not limited in the present application.

[0017] As shown in FIG. 1, in step 101, a first echo sequence signal captured by a laser radar is obtained;

[0018] In step 102, visual information of a heterogenous sensor is obtained;

[0019] In step 103, a second echo sequence signal for point cloud generation is determined by using the visual information in combination with the first echo sequence signal and echo information of each echo in the first echo sequence signal.

[0020] In the present embodiment, for step 101, the point cloud solving device obtains the first echo sequence signal captured by the laser radar. For example, the first echo sequence signal can be an echo sequence signal provided by a laser radar optical receiving device, including but not limited to peak-to-valley value, peak-to-valley time, half-peak value and half-peak time, etc. of the echo. Wherein, each measurement has multiple echoes, which is called echo sequence, and each single echo in the echo sequence has time information, intensity information, slope information, etc. of the echo, which are collectively referred to as echo information.

[0021] Then, for step 102, the point cloud solving device obtains visual information of a heterogenous sensor in the same or adjacent time window as the first echo sequence signal. Wherein, the visual information includes semantics, texture and depth, etc., and the heterogenous sensor can be a visual sensor, a millimeter wave radar, an infrared sensor, etc. which can obtain visual information.

[0022] Finally, for step 103, the point cloud solving device uses the visual information in combination with the first echo sequence signal and the echo information of each echo in the first echo sequence signal to remove the interference echoes in the first echo sequence signal, and determines the second echo sequence signal for point cloud generation.

[0023] The method of the embodiment determines the second echo sequence signal for point cloud generation by using the visual information to combine the first echo sequence signal and the echo information of each echo in the first echo sequence signal, so that higher-precision point cloud output can be achieved.

[0024] In some optional embodiments, the visual information includes semantic information and / or depth information, and determining the second echo sequence signal for point cloud generation by using the visual information to combine the first echo sequence signal and the echo information of the first echo sequence signal includes: the point cloud solving device selects one or more echoes in the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; determines whether the current echo is a light path interference echo by using the semantic information and / or depth information of the heterogenous sensor in the same time period as the current echo; if the current echo is a light path interference echo, removes the light path interference echo by using the semantic information and / or depth information to combine the echo information of the current echo; and determines the signal after removing all the light path interference echoes in the first echo sequence signal as the second echo sequence signal for point cloud generation.

[0025] In one specific example, the point cloud solving device selects one or more echoes according to the first echo sequence signal, which represent potential distance measurements on the light path, wherein the characteristic of the laser radar sensor is that each potential distance measurement on the light path is reflected as an echo on the data, and if there are multiple objects on the light path, there will be multiple echoes; then determines whether there is rain, fog, dust, or front vehicle exhaust by using the semantic information provided by the heterogenous sensor such as the monocular sensor in the same time window, and if there is light path interference, removes the interference echoes in the first echo sequence signal by combining the echo information, and selects the echoes of the actual measured objects in the environment; if there is a binocular sensor, removes the interference echoes in the first echo sequence signal by combining the semantic information and depth information provided by the heterogenous sensor in the same time window, and the echo information, and selects the echoes of the actual measured objects in the environment, so as to perform higher-precision point cloud generation. Among them, in addition to the echoes of the target objects, some non-target objects sometimes also scatter the same frequency electromagnetic waves into the radar receiving system to form clutter, which is called radial interference echo. The radial interference echo is one of the important factors that pollute the radar echo data, and seriously affects the quantitative application of radar data, so it is necessary to filter out the light path interference echoes. Among them, the detection capability of the laser radar for interference echoes can be improved by one or more technical means such as multi-echo detection, wavelength optimization, signal processing, multi-sensor fusion, dynamic adjustment of scanning mode, environment modeling, hardware improvement, and machine learning. These methods can be used alone or in combination, depending on the application scenario and requirements.

[0026] In some optional embodiments, the visual information includes semantic information and texture information, and determining the second echo sequence signal for point cloud generation by combining the first echo sequence signal and echo information of the first echo sequence signal with the visual information includes: selecting one or more echoes in the first echo sequence signal; determining whether a potential object on a light path of the current echo is only partially in a light beam exit angle by using the semantic information and the texture information of the heterogenous sensor in the same time period as the selected current echo, in particular, if a certain light beam spot range includes multiple semantic targets, it can be considered that the object is only partially in the light beam exit angle; if the potential object on the light path of the current echo is only partially in the light beam exit angle, providing multiple actual measurement distances in the current echo by combining the semantic information and the texture information with echo information of the current echo; and determining the first echo sequence signal and the multiple actual measurement distances of at least one echo as the second echo sequence signal for point cloud generation.

[0027] In one specific example, the point cloud solving device selects one or more echoes in the first echo sequence signal, the first echo sequence signal including multiple echoes; then determines whether a potential object on a light path of the current echo is only partially in a light beam exit angle by using the semantic information and the texture information of the heterogenous sensor in the same time period as the selected current echo, wherein the laser radar light spot is of a certain size, such as a circle, and there may be a case where the left half hits the first target and the right half hits another target, thereby providing distance measurement of two targets in one spot, and thus only partially in the light beam exit angle; then, if the potential object on the light path of the current echo is only partially in the light beam exit angle, multiple actual measurement distances are provided in the current echo by combining the semantic information and the texture information with echo information of the current echo, for example, if the light spot hits multiple objects, multiple echoes are provided, and target points can be generated for multiple echoes different in time sequence along the effectiveness of the multiple echoes (echo time-transmit time=photon flight time=distance x2), thereby providing multiple actual distance measurements. Finally, the first echo sequence signal and the multiple actual measurement distances of at least one echo are determined as the second echo sequence signal for point cloud generation, thereby generating a more abundant point cloud.

[0028] In some optional embodiments, the visual information includes semantic information and texture information, and the second echo sequence signal for point cloud generation is determined by combining the visual information, the first echo sequence signal, and echo information of the first echo sequence signal, including: selecting one or more echoes in the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; determining whether a potential object on the optical path of the current echo is only partially in the light exit angle by using the semantic information and the texture information of the heterogenous sensor in the same time period as the selected current echo; if the potential object on the optical path of the current echo is only partially in the light exit angle, providing a single actual measurement distance in the current echo by combining the semantic information and the texture information with the echo information of the current echo, and performing a corresponding offset on the exit angle of the optical path of the current echo according to the semantic information and the texture information; and determining the first echo sequence signal, the single actual measurement distance of the at least one echo, and the offset corresponding to the exit angle of the at least one echo as the second echo sequence signal for point cloud generation. Wherein, for providing a single or multiple actual measurement distances, it is mainly determined according to the objects in the spot range. If the semantic information in the spot range prompts that it is an actual object, including a person, a vehicle, a signboard, etc., a single measurement distance is provided; if there are multiple objects in the semantic information, such as a person in the foreground and a building in the background, multiple measurement distances can be provided. As for how to offset specifically, the following examples are given for reference by those skilled in the art. In the case of a larger spot, such as a person on the left half and a building on the right half according to the semantic information, the spot can be divided into two parts, the angle of the person is adjusted to the left half, and the angle of the building is adjusted to the right half, such as a spot of 0.2 degrees, which can be split into left and right 0.1 degrees, and assigned to different objects, so that the original resolution is only 0.2 degrees, and after subdivision, the resolution can be 0.1 degrees.

[0029] In one specific example, the point cloud solving device selects one or more echoes in the first echo sequence signal, wherein the first echo sequence signal includes a plurality of echoes. Then, using the semantic information and the texture information of the heterogenous sensor in the same time period as the selected current echo, such as a vision sensor, a millimeter wave radar, an infrared sensor, etc., it is determined whether the potential object on the light path of the current echo is only partially in the light exit angle, i.e., a plurality of target distance measurements are provided within one spot. Then, if the potential object on the light path of the current echo is only partially in the light exit angle, a single actual measurement distance is provided in the current echo using the semantic information and the texture information combined with the echo information of the current echo, and the exit angle of the light path of the current echo is correspondingly offset according to the semantic information and the texture information, so that a certain echo can be selected from the plurality of echoes for point cloud generation, and a single distance measurement can be obtained; finally, the first echo sequence signal, the single actual measurement distance of the at least one echo, and the offset corresponding to the exit angle of the at least one echo are determined as the second echo sequence signal for point cloud generation, so as to perform more accurate point cloud generation.

[0030] In some optional embodiments, the visual information includes semantic information and texture information, and the second echo sequence signal for point cloud generation is determined by combining the first echo sequence signal and echo information of the first echo sequence signal with the visual information, including: selecting one or more echoes in the first echo sequence signal, wherein the first echo sequence signal includes a plurality of echoes; then, determining whether the potential object on the optical path of the current echo contains a low-reflective material by using the semantic information and the texture information of the heterogenous sensor in the same time period as the selected current echo; if the potential object on the optical path of the current echo contains a low-reflective material, providing a plurality of distances corresponding to the low-reflective material and the normal reflective material in the echo information of the current echo by combining the semantic information and the texture information with the echo information of the current echo; and finally, determining the plurality of distances corresponding to the low-reflective material and the normal reflective material of at least one echo of the first echo sequence signal as the second echo sequence signal for point cloud generation, thereby performing higher-precision point cloud generation. The low-reflective material refers to a material with weak surface light reflection ability. Low-reflective materials reduce light reflection through surface roughness, high light absorption, or special coatings, and are widely used in optical, construction, automotive, and electronic fields. When a laser radar detects a low-reflective material, it can improve the detection effect by increasing laser power, optimizing receiver sensitivity, using multi-echo detection, selecting appropriate wavelengths, optimizing data processing algorithms, shortening detection distance, multi-beam scanning, and environmental light compensation. These methods can be used alone or in combination, depending on the application scenario and requirements. In the embodiments of the present application, multiple distances are given for low-reflective materials because glass and other low-reflective materials can affect the physical properties of the echo; without vision, other echoes can only be discarded; with vision, other echoes that penetrate glass can be recalled. The material is only used to determine whether it is a special low-reflective material such as glass, and does not change the distance. Therefore, more echo signals can be retained, and rich echo signals can help improve subsequent detection accuracy.

[0031] In some optional embodiments, the visual information is an image collected by the heterogenous sensor, and a visual feature can be extracted by a deep learning network. The second echo sequence signal for point cloud generation is determined by combining the visual information, the first echo sequence signal, and echo information of the first echo sequence signal, including: selecting one or more echoes in the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; then, inputting the echo information of the current echo and the visual feature in the same time period as the current echo into the deep learning network to obtain echo selection output by the deep learning network; and finally, taking all echo selections of the first echo sequence signal as the second echo sequence signal for point cloud generation, so as to generate a more abundant point cloud. The function of the deep learning network is to simultaneously input the laser radar raw information and the visual information, and can provide the echo selection output. The visual feature is the intermediate result after the image is input into the deep network for convolution and other operations.

[0032] In some optional embodiments, the second echo sequence signal for point cloud generation is determined by combining the visual information, the first echo series signal, and the echo information of the first echo series signal, including: extracting a bottom feature of the laser radar by a deep learning network, wherein the visual information is used as an auxiliary in the process of training the deep learning network to extract the bottom feature; selecting one or more echoes in the first echo sequence signal, wherein the first echo sequence signal includes a plurality of echoes; then, inputting the bottom feature in the same time period as the selected current echo into the deep learning network to obtain echo selection output by the deep learning network; and lastly, taking all echo selections of the first echo sequence signal as the second echo sequence signal for point clouds generation. The bottom feature refers to the intermediate result after the raw output of the laser radar sensor is input into the deep network for convolution and other operations.

[0033] In one specific example, the input of the point cloud solving unit can include an echo sequence signal provided by an optical receiving device and one or more information provided by a visual sensor in the same or adjacent time window. The echo sequence signal provided by the optical receiving device includes but is not limited to peak-to-valley value, peak-to-valley time, half-peak value, half-peak time, etc. The one or more information provided by the visual sensor in the same or adjacent time window (for example, the typical value can be within 0ms-100ms) includes but is not limited to semantics, texture, depth, etc.

[0034] The point cloud solving unit will make full use of the information provided by the visual sensor to determine the nature of the echo and generate the corresponding point cloud. The specific point cloud generation method includes but is not limited to the following rule-based method, the following rules can be executed in parallel, all in order to filter out false results and obtain more abundant information, which is not limited in this application. First, according to the echo sequence, one or more echoes representing potential distance measurements on the light path can be selected; according to the semantic information provided by the visual sensor within the same time window, it is judged whether there is rain, fog, dust, and the tail gas of the preceding vehicle. If there is light path interference, the interference echo is removed from the echo sequence combined with the echo information, and the echo of the actual measured object in the environment is selected; if there is a binocular sensor, according to the depth information provided by the visual sensor within the same time window, plus the echo information, the interference echo is removed from the echo sequence, and the echo of the actual measured object in the environment is selected to generate more accurate point clouds. The second, according to the echo sequence, one or more echoes representing potential distance measurements on the light path can be selected; according to the semantic, texture information provided by the visual sensor within the same time window, it is judged whether the potential object on the light path is only part of the light exit angle. If so, combine the echo information to provide multiple actual distance measurements in the echo sequence to generate more abundant point clouds. Among them, if the semantic information within the range of the light spot prompts that it is an actual object, including people, vehicles, signs, etc., a single measurement is provided; if there are multiple objects in the semantic information, such as a person in the foreground and a building in the background, multiple echoes can be provided. The third, according to the echo sequence, one or more echoes representing potential distance measurements on the light path can be selected; according to the semantic, texture information provided by the visual sensor within the same time window, it is judged whether the potential object on the light path is only part of the light exit angle. If so, combine the single actual distance measurement in the echo sequence according to the echo information, and according to the information provided by the visual sensor, the exit angle of the light path is offset accordingly to generate more accurate point clouds. Among them, the single measurement distance is generally the measurement distance corresponding to the nearest echo. The fourth, according to the echo sequence, one or more echoes representing potential distance measurements on the light path are selected; according to the semantic, texture information provided by the visual sensor within the same time window to judge whether the potential object on the light path has low-reflective materials such as glass. If so, combine the echo information to provide multiple distances of multiple echoes corresponding to low-reflective and normal-reflective in the echo sequence to generate more abundant point clouds.

[0035] Further, the specific point cloud generation method also includes but is not limited to the following deep learning-based method: according to the echo sequence, one or more echoes representing potential distance measurements on the optical path are selected; according to the visual features extracted by the deep learning network within the same time window, the potential echo distance and the underlying features are input into the deep learning network, and echo selection information is output to generate more accurate and rich point clouds. Alternatively, according to the echo sequence, one or more echoes representing potential distance measurements on the optical path are selected; according to the underlying features extracted by the deep learning network within the same time window, the laser radar sensor outputs echo selection information to generate more accurate and rich point clouds.

[0036] The existing technology only selects the nearest echo by default. The scheme of the embodiment of the present application may provide the nearest echo or multiple echoes containing the nearest echo after combining semantic information, so that the generated point cloud is more accurate and rich.

[0037] In some other embodiments, the embodiment of the present application also provides a non-volatile computer storage medium, which stores computer executable instructions. The computer executable instructions can execute the point cloud solving method in any of the above method embodiments.

[0038] As an implementation manner, the non-volatile computer storage medium of the present application stores computer executable instructions, which are configured to:

[0039] acquire a first echo sequence signal captured by the laser radar;

[0040] acquire visual information of the heterogenous sensor;

[0041] determine a second echo sequence signal for point cloud generation by using the visual information in combination with the first echo sequence signal and echo information of each echo in the first echo sequence signal.

[0042] The non-volatile computer readable storage medium can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required by a function. The data storage area can store data created according to the use of the point cloud solving device. In addition, the non-volatile computer readable storage medium can include a high-speed random access memory and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer readable storage medium can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the point cloud solving device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0043] This invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the above-described point cloud solving methods.

[0044] Figure 2 is a schematic diagram of the electronic device provided in an embodiment of the present invention. As shown in Figure 2, the device includes one or more processors 210 and a memory 220. Figure 2 shows an example of one processor 210. The device for the point cloud solving method may further include an input device 230 and an output device 240. The processor 210, memory 220, input device 230, and output device 240 can be connected via a bus or other means. Figure 2 shows an example of connection via a bus. The memory 220 is the aforementioned non-volatile computer-readable storage medium. The processor 210 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 220, thereby implementing the point cloud solving method described in the above embodiment. The input device 230 can receive input digital or character information and generate key signal inputs related to user settings and function control of the point cloud solving device. The output device 240 may include a display screen or other display device.

[0045] This application also provides a mobile platform, which includes a vehicle body, a power system, and electronic devices as described in the above embodiments. The power system is installed on the vehicle body to provide power; the principle and implementation of the electronic devices are consistent with those described in the above embodiments and will not be repeated here. The electronic devices may be sensors, controllers, or other computing devices installed on the mobile platform. Optionally, the mobile platform may include at least one of the following: a vehicle, a mobile robot, or an unmanned vehicle.

[0046] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0047] In one embodiment, the above-described electronic device is applied in a point cloud computing apparatus, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0048] Acquire the first echo sequence signal captured by the lidar;

[0049] Acquire visual information from heterogeneous sensors;

[0050] By combining the visual information with the first echo sequence signal and the echo information of each echo in the first echo sequence signal, a second echo sequence signal for point cloud generation is determined.

[0051] The electronic devices described in this application exist in various forms, including but not limited to:

[0052] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0053] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0054] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players (e.g., iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0055] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0056] (5) Other electronic devices with data interaction functions, such as cameras, lidar and other sensors.

[0057] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0058] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A point cloud solution method, comprising: Acquire the first echo sequence signal captured by the lidar; Acquire visual information from heterogeneous sensors; By combining the visual information with the first echo sequence signal and the echo information of each echo in the first echo sequence signal, a second echo sequence signal for point cloud generation is determined.

2. The method according to claim 1, characterized in that, The visual information includes semantic information and / or depth information. Using the visual information in combination with the first echo sequence signal and its echo information, a second echo sequence signal for point cloud generation is determined, including: Select one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; Using semantic and / or depth information from heterogeneous sensors within the same time period as the selected current echo, determine whether the current echo is an optical path interference echo; If the current echo is an optical path interference echo, the optical path interference echo is removed by combining the semantic information and / or the depth information with the echo information of the current echo. The signal obtained by removing all optical path interference echoes from the first echo sequence signal is determined as the second echo sequence signal used for point cloud generation.

3. The method according to claim 1, characterized in that, The visual information includes semantic information and texture information. Using this visual information in conjunction with the first echo sequence signal and its echo information, a second echo sequence signal for point cloud generation is determined, including: Select one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; Using semantic and texture information from heterogeneous sensors within the same time period as the selected current echo, it is determined whether only a portion of the potential objects in the optical path of the current echo are at the light emission angle. If only part of the potential object in the optical path of the current echo is at the light emission angle, multiple actual measurement distances are provided in the current echo by combining the semantic information and the texture information with the echo information of the current echo. The first echo sequence signal and multiple actual measured distances of at least one echo are used to determine the second echo sequence signal for point cloud generation.

4. The method according to claim 1, characterized in that, The visual information includes semantic information and texture information. Using this visual information in conjunction with the first echo sequence signal and its echo information, a second echo sequence signal for point cloud generation is determined, including: Select one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; Using semantic and texture information from heterogeneous sensors within the same time period as the selected current echo, it is determined whether only a portion of the potential objects in the optical path of the current echo are at the light emission angle. If only part of the potential object in the optical path of the current echo is at the light emission angle, the semantic information and the texture information are combined with the echo information of the current echo to provide a single actual measured distance in the current echo, and the emission angle of the optical path of the current echo is offset accordingly based on the semantic information and the texture information. The first echo sequence signal, the individual actual measured distance of at least one echo, and the offset corresponding to the departure angle of the at least one echo are determined as the second echo sequence signal for point cloud generation.

5. The method according to claim 4, characterized in that, The step of shifting the exit angle of the current echo's optical path according to the semantic information and the texture information includes: The position of the potential object in the optical path of the current echo is determined based on the semantic information and the texture information. The emission angle of the current echo's optical path is segmented, and a corresponding offset is made towards the position of the potential object.

6. The method according to claim 1, characterized in that, The visual information includes semantic information and texture information. Using this visual information in conjunction with the first echo sequence signal and its echo information, a second echo sequence signal for point cloud generation is determined, including: Select one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; Using semantic and texture information from heterogeneous sensors within the same time period as the selected current echo, it is determined whether potential objects in the optical path of the current echo contain low-reflectivity materials. If the potential object in the optical path of the current echo contains a low-reflectivity material, the semantic information and the texture information are combined with the echo information of the current echo to provide multiple distances corresponding to the low-reflectivity material and the normal reflectivity material, respectively. The first echo sequence signal and at least one echo at multiple distances corresponding to low-reflectivity and normal-reflectivity materials are determined as the second echo sequence signal for point cloud generation.

7. The method according to claim 1, characterized in that, The visual information is the visual features extracted from the images acquired by the heterogeneous sensor through a deep learning network. Using this visual information, combined with the first echo sequence signal and its echo information, a second echo sequence signal is determined for point cloud generation, including: Select one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; Based on the visual features within the same time period as the selected current echo, the echo information and low-level features of the current echo are input into the deep learning network to obtain the echo selection output by the deep learning network. The low-level features are obtained by feeding the original output of the lidar into the deep learning network. All echoes from the first echo sequence signal are selected as the second echo sequence signal for point cloud generation.

8. The method according to claim 1, characterized in that, Using the visual information combined with the first echo sequence signal and the echo information of the first echo sequence signal, a second echo sequence signal for point cloud generation is determined, including: The low-level features of the LiDAR are extracted through a deep learning network, wherein the visual information is used as an aid during the training of the deep learning network by extracting the low-level features. Select one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; The low-level features within the same time period as the selected current echo are input into the deep learning network to obtain the echo selection output by the deep learning network; All echoes from the first echo sequence signal are selected as the second echo sequence signal for point cloud generation.

9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 8.

10. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A portable platform comprising the electronic device as described in claim 9.

Images