Combined high resolution lidar and RGB color sensor

Abstract

Three-dimensional LiDAR scanning combines a solid-state fast scanning device such as an optical switch and a slower scanning device such as a mirror and may include a switch architecture for a large port-count optical switch to provide frame rates of 100Hz or higher with improved resolution and detection range. A controller provides adjustable scanning of the field- of-view (FOV) with respect to scan area, scan or frame rate, and resolution for a frame, detected object, or time slices of a scan. A controller combines RGB data with NIR data to match 3D images with color 2D images. A controller or computer processes point cloud data to generate vector cloud data to identify, categorize, and track objects within or beyond the FOV. Vector cloud data provides lossless compression for storage / communication of road traffic and scene data, object history, and object sharing beyond the FOV.

Description

COMBINED HIGH RESOLUTION LiDAR AND RGB COLOR SENSORCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 716,379, filed November 5, 2024, the disclosure of which is hereby incorporated in its entirety by reference herein.

[0002] This application is a continuation-in-part of U.S. Application No. 17 / 331,265, filed May 26, 2021, the disclosure of which is hereby incorporated in its entirety by reference herein.TECHNICAL FIELD

[0003] This disclosure relates to a system and method that combine high resolution 3D (three-dimensional) LiDAR measurements and 2D (two-dimensional) RGB (red-green-blue) color images.BACKGROUND

[0004] Light detection and ranging (LiDAR or lidar) sensors are commonly used to capture spatial 3D point clouds representing environments for applications including mapping, navigation, and perception, for example. Lidars actively emit laser light to illuminate objects within a field of view and measure return times of reflected light to provide precise distance data. However, traditional lidars lack color information, producing only grayscale intensity values. Passive camera sensors can capture high resolution color images with rich scene details. But standard 2D cameras lack depth information and 3D spatial relationships.

[0005] Integrating lidar and camera technologies would provide the benefits of both accurate 3D geometry from lidar combined with high fidelity color imagery from cameras. However, prior multi-modal approaches face various challenges. For example, parallax and misalignment between the sensors due to their separate perspectives result in different fields of view. Complex registration algorithms are required to try to align the separate 3D and 2D data,resulting in artifacts and limited success. In addition, the integration process forces reducing the capabilities of both sensors, limiting resolution, frame rates, and other parameters to find a common ground, which compromises the fidelity of color and depth perception.SUMMARY

[0006] Systems and methods according to the present disclosure solve various problems associated with prior approaches using an integrated design aligning the lidar and camera optics and synchronizing data captures of the lidar and camera while maintaining the full resolution and capabilities of both sensors. This enables fusing high accuracy 3D lidar spatial data with high resolution 2D camera color imagery without significant compromises to resolution, frame rates, and other parameters to enhance the fidelity of color and depth perception.

[0007] In various embodiments, an integrated color lidar apparatus aligns and synchronizes a 3D lidar sensor with a 2D camera to combine high fidelity spatial and color data using a single sensor. Dichroic mirror optics enable matching fields of view without parallax. Modular hardware components including the lidar, camera, mirror, and synchronization electronics are arranged to maintain optimal performance of both sensors. Captured 3D point clouds and 2D images can be processed independently at variable resolutions and frame rates. Software algorithms then combine the geometric and visual data to generate enhanced colored 3D outputs. The system provides efficient alignment of high accuracy lidar 3D spatial data and high resolution camera 2D color imagery without compromises, enabling advanced perception and scene understanding capabilities.

[0008] In one embodiment, a color lidar system includes a lidar sensor that emits laser light and detects reflected light to scan an environment in three dimensions and generate a 3D point cloud, a camera having an RGB sensor (such as a CMOS or similar sensor, for example) aligned with the lidar sensor that captures two dimensional color images of the environment, a dichroic mirror located between the lidar sensor and the camera to reflect the lidar laser light and transmit visible light from the environment to align the fields of view of the lidar sensor and camera without parallax, and a processing system having memory and a processor programmed to combine 3D point cloud data from the lidar sensor and 2D color image data from the camera in real time tominimize or eliminate distortion, time differences, and parallax between pixels using synchronization and calibration techniques to generate a fused 3D color point cloud. The processor may be programmed or configured to synchronize the capture of 2D color images by the camera or RGB sensor with the scanning of the environment by the lidar sensor within a synchronization module. The synchronization module may include hardware and firmware to trigger the camera shutter at set positions during a sweep or scan of a scanning galvanometric mirror of the lidar sensor. The synchronization module may be configured to provide synchronized shutter instances and scanning angles to accurately align the 2D images and 3D point cloud. In various embodiments, the lidar sensor outputs a 2D image buffer containing a high resolution image of the environment in two dimensions, and a 3D point cloud containing spatial position data of the environment in three dimensions, wherein the 2D image buffer and 3D point cloud can be accessed separately and utilized independently. In various embodiments, the frame rate and resolution of the 2D and 3D outputs are different to accommodate particular application requirements.

[0009] Embodiments of a color lidar system according to the disclosure may include a processor or computer programmed or configured to analyze 3D point cloud data to identify and locate objects within the environment. The processor may be further programmed or configured to cluster 3D points corresponding to individual objects using one or more segmentation algorithms. The processor may be further programmed or configured to analyze geometric features of objects to determine or classify objects as one of a plurality of types or classes using one or more classification algorithms. The processor may be further programmed or configured to match identified 3D objects with corresponding 2D image subsets from the aligned color camera using a correlation module. The processor may be further programmed or configured to extract color snippets segmented from the 2D images captured by the color camera based on the location and boundaries of objects within the 3D point cloud associated with the lidar. The processor may be further programmed or configured to output labels and classes of identified objects and associated high resolution 2D color images. In various embodiments, the processor is further programmed or configured to match or correlate identified 3D objects with corresponding 2D image subsets from the aligned camera using a correlation module. The processor may be further programmed or configured to identify or extract color snippets segmented from the 2D imagesbased on the location and boundaries of objects in the 3D point cloud, as well as output labels and classes of identified objects with associated high resolution 2D color images.

[0010] One or more embodiments of a color lidar system may include a dichroic mirror positioned at a 45 degree angle to the optical axes of a lidar sensor and camera. The dichroic mirror transmits visible wavelength light from the environment to the camera while reflecting the lidar sensors infrared laser light. This provides an optical path separation that eliminates parallax between the camera and lidar by aligning their fields of view along the same optical axis. The system may include a processor programmed or configured to align the field of view of the camera to the scanning field of view of the lidar sensor in a calibration module or algorithm. Coarse adjustment of the camera mounting position and orientation to center its field of view within the lidar’s scanning range may be combined with operation of calibration algorithms that fine tune the alignment by analyzing corresponding data points between the camera images and lidar 3D points of a calibration target or various other objects within the fields of view. A synchronization algorithm or module implemented by a programmed processor may determine the angle of the lidar's scanning galvanometric mirror based on rotary position encoders or laser position detectors. When the mirror reaches preset synchronized angles during its sweep, the module triggers the camera to capture an image, creating aligned frames of the lidar and camera.[OOH] In one embodiment, a color lidar system includes a lidar sensor that timestamps each laser firing and 3D frame capture event. Images captured by a color camera are timestamped upon shutter trigger and frame capture. A programmed processor executes a synchronization algorithm or module to align the timestamps of the lidar and camera data to identify matching frames and laser firing instances.

[0012] Various embodiments of a color lidar system include a lidar sensor comprised of a laser emitter, optical elements, photodetectors, timing circuitry, and output interface, a camera comprised of an image sensor, optics, frame buffer, and output interface, a dichroic mirror with spectral coatings to reflect the lidar wavelengths while passing visible light, a galvanometric scanning mirror for the lidar transmitter and receiver, a processor programmed with a synchronization module in communication with timing circuitry and a shutter trigger, and interfaces between the modular components for control signals, data transfer, and power, forexample. The system may include one or more processors of a processing system programmed or otherwise configured to perform sensor fusion algorithms that align and integrate the 3D point cloud and 2D color images into a single colored 3D point cloud, synchronization algorithms or routines that time-match frames between the lidar and camera based on capture timestamps, and calibration routines that improve alignment based on data correlations. A classification algorithm or module is executed by one or more processors to analyze geometric features of the segmented 3D point cloud data for an identified object to determine a preliminary 3D classification and score, analyze visual features of the correlated 2D image subset for the same object to determine a preliminary 2D classification and score, and integrate the 3D and 2D classification scores using a fusion algorithm to calculate an overall weighted certainty score for the object class. The classification algorithm may include class-specific algorithms that identify detailed properties of objects based on processing the correlated 2D image subsets. For example, for a traffic light object class, color classification on the 2D image determines the current state of the light (red, yellow, green, directional arrows), whereas for a sign object class, optical character recognition on the 2D image extracts text and semantic information. The 2D analysis may be performed in higher resolution to provide higher confidence in object / property identification compared to lower resolution 3D data. Various embodiments also allow for variations of scan frequency of designated object classes based on their certainty and propensity to change (e.g. traffic lights change, but traffic signs do not).

[0013] A correlation algorithm or module may be executed by one or more processors that extracts data of an identified 3D object from point cloud segmentation and object classification modules, determines the expected 2D projection of that object in the aligned camera images based on the calibration between the lidar and camera subsystems, extracts a 2D subset from the camera images that matches and encloses the 2D projection of the identified 3D object, and enables further analysis on the higher resolution 2D subsets correlated and aligned to each recognized 3D object. Embodiments may also include 3D point cloud completion algorithms that fill gaps in the lidar data and reconstruct surfaces of objects. These algorithms analyze the correlated 2D image subsets which provide texture and surface continuity information. Missing 3D points are interpolated by propagating and extending surfaces that match the 2D image textures. This leverages the 2D color data aligned to each identified object to improve reconstruction of the 3D geometry.

[0014] In various embodiments of a color lidar system according to the disclosure, the lidar sensor specifications include resolution, frame rate, field of view, and range meeting requirements for 3D scanning of the environment, the camera sensor specifications include resolution, frame rate, field of view, and color depth meeting requirements for capturing color images of the environment, and the specifications and capabilities of the lidar and camera are selected or maintained independently without reduction or limitation of one modality by the other.

[0015] In one embodiment, a color lidar system includes a lidar subsystem having a galvanometric mirror with a scanning speed that can be varied to different horizontal rates. A synchronization module triggers the color camera shutter only when the galvanometric mirror reaches the predetermined or otherwise specified angular positions, independent of scanning speed. The processing system integrates the captured 2D color data horizontally into the 3D point cloud based on the galvanometric mirror scanning rate. This enables variable horizontal resolution in the fused color point cloud reflecting the mirror speed.

[0016] One or more embodiments may provide associated significant advantages over traditional independent lidar and camera systems for capturing multidimensional color scene data. For example, fusing high resolution 2D color imagery with 3D spatial data enables more detailed 3D environmental models with realistic colors and textures. This enhanced 3D perception improves scene understanding capabilities. Various embodiments use dichroic mirror optics to accurately align the camera and lidar fields of view along the same axis without parallax, which avoids laborious error-prone software registration strategies. Triggering the camera shutter when the lidar mirror is at set points enables time-matched captures of camera and lidar data streams to provide synchronized frames. Unlike other fusion approaches, both lidar and camera retain full independent configurations for range, resolution, frame rate, etc. reducing or eliminating resolution compromises common to prior strategies. Separate outputs provide the ability to access both 2D images and 3D point clouds independently with custom parameters to provide flexibility and efficiency. Aligned 3D and 2D data streams enable sophisticated processing with advance algorithms to enable object classification, property identification, surface reconstruction, and point cloud completion, for example. Use of modular hardware such as individual lidar, camera, optical, and electronic components provides flexibility compared to fully integrated sensors.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings illustrate representative embodiments of a fused hi-res Lidar with RGB sensors according to the disclosure.

[0018] Figure 1 is a block diagram illustrating operation of embodiments of a system or method for color LiDAR having a linear RGB sensor with associated coupling optics implemented by optical fibers.

[0019] Figure 2 is a block diagram illustrating color LiDAR embodiments similar to those of Figure 1 but having RGB line sensor coupling optics implemented by at least one lens rather than optical fibers with the RGB line sensor positioned in the focal plane of the coupling optics.

[0020] Figure 3 is a block diagram illustrating color LiDAR embodiments similar to Figure2, but with a 2D matrix RGB sensor.

[0021] Figure 4 is a block diagram illustrating color LiDAR embodiments similar to Figure3, but with a 2D matrix RGB sensor positioned in front of a scanning mirror and combined into the field of view of the LiDAR through a dichroic mirror.

[0022] Figure 5 is a block diagram illustrating color LiDAR embodiments similar to Figure4, but with the 2D matrix RGB sensor positioned behind a scanning mirror with an overlapping field of view with the LiDAR.

[0023] Figure 6 illustrates an embodiment of a synchronization strategy for LiDAR and RGB data suitable for configurations similar to those of Figures 1 and 2.

[0024] Figure 7 is a flowchart illustrating a timing sequence or algorithm for the synchronization strategy illustrated in Figure 6.

[0025] Figure 8 illustrates an embodiment of a synchronization strategy for LiDAR and RGB data suitable for configurations similar to Figure 3.

[0026] Figure 9 is a flowchart illustrating a timing sequence or algorithm for the synchronization strategy illustrated in Figure 8.

[0027] Figure 10 illustrates an embodiment of a synchronization strategy for LiDAR and RGB data suitable for configurations similar to Figures 3, 4, and 5.

[0028] Figure 11 is a flowchart illustrating a time sequence or algorithm for the synchronization strategy illustrated in Figure 10.

[0029] Figure 12A illustrates representative LiDAR 3D data represented by color gradients and Figure 12B illustrates representative fused LiDAR and RGB color image data.

[0030] Figure 13A illustrates color LiDAR point cloud data and Figure 13B illustrates operation of object identification and color snippet segmentation using color LiDAR data.DETAILED DESCRIPTION

[0031] Detailed embodiments are disclosed herein; however, it is to be understood that the disclosed embodiments are merely representative and may be alternatively embodied in various forms. The figures are not necessarily to scale; some features may be exaggerated or minimized to show details of components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the claimed subject matter. Similarly, while various embodiments illustrate combinations of features associated with representative implementations, those of ordinary skill in the art will recognize that features from one or more embodiments are readily combinable to form new embodiments that may not be explicitly described or illustrated in the figures.

[0032] As used in this description, an image or related terminology is not limited to a visual representation and refers more generally to a data representation of a field of view (FOV). Different types of data, such as location / position, distance / range, intensity, polarization, speed / velocity, etc., may be collected for each measured point or pixel within the FOV to provide a multi-dimensional data array that may be processed by a controller without generating a visualrepresentation of the data. Similarly, references to a pixel do not imply or require a visual representation or display of associated data, or an area on a display screen, but refer more generally to a discrete measurement point or observation point within the FOV. Underlying discrete measurements for a particular pixel location may be referred to as sub-pixels that may be used to improve or enhance the resolution within the pixel.

[0033] Resolution is used in its broadest sense and generally refers to a number of pixels (sometimes referred to as voxels when referring to volume or three-dimensional space) per unit area or volume, with a higher resolution indicative of a higher number of pixels or data points per unit area or volume, which may include a specified FOV, portion of the FOV, or an object within or outside the FOV, for example.

[0034] A point cloud refers to a dataset that represents a three-dimensional shape or object in space. Each point represents an x, y, and z coordinate of a single point of a detected shape or object relative to a fixed or stationary reference point. Depending on the embodiment, each point may also include other data, parameters, or characteristics including color content such as RGB (red, green, blue) values and / or speed / velocity, object identification or object type / category, for example.

[0035] A vector cloud refers to a dataset that represents a change or difference of a point or group of points in a point cloud and may characterize the change or difference with respect to a speed and direction (or velocity) of a change in distance from a reference, which may be a previous position or location of the point or group of points, a fixed reference, or a moving reference.

[0036] A frame refers to a data representation of the FOV (or portion thereof) for a particular period of time, which may reflect the time required to scan the FOV (or portion thereof) at least once. The frame data may be a mathematical or statistical combination of data generated by two or more scans of the FOV (or portion thereof). For example, the frame data for a particular pixel may be a maximum, minimum, average, or other function or calculation of values for various data associated with that pixel (such as distance, color, speed, etc.). Frame rate refers to the number of frames per unit time and is typically a fraction of the scan rate with multiple scans per frame.

[0037] In general, the processes, methods, or algorithms disclosed herein can be performed by a processing device, controller, or computer, which can include any existing programmable electronic control unit or dedicated electronic control unit or controller. Similarly, the processes, methods, or algorithms can be stored as data and instructions executable by a controller or computer in many forms including, but not limited to, information permanently stored on non- writable storage media such as ROM devices and information alterably stored on writeable storage media including electronic, magnetic, and / or optical storage devices. Certain processes, methods, or algorithms may also be implemented in a software executable object. Alternatively, the processes, methods, or algorithms can be embodied in whole or in part using suitable dedicated or custom hardware components, such as Application Specific Integrated Circuits (ASICs), Field- Programmable Gate Arrays (FPGAs), state machines, controllers, or any other hardware components or devices, or a combination of hardware, software and firmware components. Similarly, illustration or description of a process, algorithm, or function in a particular sequence or order may not be required to perform the described operation or outcome. Some processes, functions, algorithms, or portions thereof may be repeatedly performed, performed in a different sequence, or omitted for certain applications.

[0038] As described in greater detail herein, the present disclosure provides an integrated color lidar apparatus that fuses 3D spatial data from a lidar sensor with aligned high resolution 2D color imagery from a camera. This generates a combined colored 3D point cloud output that provides enhanced environmental perception compared to either sensor modality alone. The lidar sensor actively scans the field of view in three dimensions by emitting laser light and measuring return times and intensities. This generates a 3D point cloud containing precise spatial location information about the environment. Aligned with the lidar sensor is a passive camera that captures 2D images containing color information with high resolution. The fields of view of both sensors are matched and aligned using specialized optical components so that points in 3D space correspond with color pixels from the same perspective without parallax. This enables fusing the strengths of both modalities - highly accurate 3D spatial data and high fidelity 2D color imagery.

[0039] The apparatus comprises specialized hardware for aligning the sensors without parallax while also synchronizing their captures in time. A dichroic mirror is positioned betweenthe lidar’s Transmit / Receive optical path and the camera at an angle to reflect the lidar's infrared laser wavelengths to the environment, while passing the visible light spectrum to the camera. This separates the optical paths while aligning their perspectives along the same axis, eliminating parallax errors. Furthermore, a synchronization module triggers the camera's shutter to acquire images at specific instances when the lidar's scanning mirror reaches preset angles in its sweep. The firmware determines the mirror angle through encoders and triggers camera frames in sync, enabling accurate time alignment of the 3D and 2D data captures.

[0040] The aligned lidar and camera generate independent data streams that can be accessed separately. The lidar outputs a 3D point cloud containing spatial position information for measured points across the field of view. It can optionally output a multitude of range colored images representing point properties (such as distance or height values) across the scanned scene. The camera generates 2D color images with higher pixel resolution across the entire scene. A key advantage is that the 3D and 2D data streams can be retrieved out of the lidar system at different resolutions and frame rates based on application needs. The lidar output rate may be reduced to save bandwidth while the camera can still stream high frame-per-second (FPS) color frames. The specialized alignment optics allow the best settings for each sensor without compromising capabilities due to fusion requirements. It also preserves vision integrity due to optical path alignment to the final field of view.

[0041] Advanced software algorithms fully leverage the aligned 3D and 2D data streams. Object identification modules analyze the 3D point cloud to segment and recognize objects in the environment using geometric features. The identified object locations are then correlated to the 2D camera images to extract color image subsets containing those objects with high resolution detail. Multiple classification algorithms may run in parallel, some analyzing 3D geometry, others 2D visual features, and others fusing both data types. This produces robust object labels with certainty scores. Further class-specific algorithms utilize the enhanced 2D views of objects to determine unique properties, like reading text on signs or identifying traffic light colors.

[0042] Additional point cloud processing algorithms may be employed to make full use of the aligned 2D color imagery to improve the 3D representations. Techniques like point completion and surface prediction leverage the texture and continuity evident in the corresponding 2D viewsto fill gaps and smooth surfaces in the sparser lidar point cloud. Propagating and extending surfaces that match the overlayed 2D color pixels enables reconstruction of missing geometry. This provides a complete and enhanced 3D scene representation by fusing lidar spatial data with 2D optical relationships.

[0043] Figure 1 provides a block diagram of a color lidar apparatus 10 according to the disclosure. A lidar sensor 12 composed of a line sensor 14, transmitting (Tx) laser fibers 16 and receiving (Rx) laser fibers 18, a beam combiner 20, and lidar optics 22is shown on the left, emitting multiple laser beams 24 toward a rotating galvanometric mirror 26 on the right. The lidar beams 24 go through a dichroic mirror element 28 that transmits the infrared (IR) light 24 of the Lidar (1550nm or other wavelength in this area) and reflect the visible light 30 (RGB color range) toward an RGB sensor 32 on the bottom. The RGB sensor 32 in this case is composed of a RGB CMOS line 34 that is attached to collecting fibers 36 and receive optics 38 that both are responsible to collect the visible light 30 received by the lidar apparatus 10 and to transfer the light 30 into the line sensor 34 pixels in an orderly manner. The RGB sensor 32 and the Tx / Rx Lidar line sensor 12 are perfectly aligned and go out of the system through the Galvo scanning mirror 26 at the exact same time and direction. This aligns the Lidar and camera fields of view along the same optical path, preventing parallax and skew intime of both images. It is important to note that the use of fibers 36 in front of the RGB line sensor 34 is not mandatory and one can place the RGB line sensor 34 right behind the receiving optics 38, at its focal plane, with no fibers in the middle. However, by placing fibers 36, one can place the RGB line sensor 34 far away from the rest of the components, and / or increase the amount of light that the sensor RGB line sensor collects, as well as connect several Lidar units with fiber arrays into one single line sensor that has large number of pixels that can cover much larger field of view than needed for a single Lidar.

[0044] Figure 2 is a block diagram of an embodiment of a color lidar apparatus 10 according to the disclosure. Similar to the system illustrated in Figure 1, the RGB line sensor 34 is used to scan the field of view parallel to the Lidar scan line. However, unlike the configuration of Figure 1, the RGB line sensor 34 is positioned directly in the focal plane of the receiving visible optics 38, with no fibers in between.

[0045] Figure 3 is a block diagram illustrating another embodiment of a color lidar apparatus according to the disclosure. Similar to the system illustrated in Figure 2, the RGB sensor 32 is used to scan the field of view parallel to the Lidar scan line. However, unlike the system of Figure 2, the RGB sensor 32 comprises a 2D matrix 40 (such as a CMOS sensor used for cellphone cameras) that is positioned directly in the focal plane of the receiving visible optics 38, with no fibers in between. The matrix 40 can be of a rolling shutter or global shutter type. In either type, a time and spatial conversion can be performed to match the scan pattern of the galvo mirror 26 and the exposure time of the RGB sensor 32 to create a coherent image that matches the Lidar image created by the scanning galvo.

[0046] Figure 4 is a block diagram illustrating another embodiment of a color lidar apparatus 10 according to the disclosure. Similar to the embodiment illustrated in Figure 3, the RGB sensor 32 is a 2D matrix 40 (such as a CMOS sensor used for cellphone cameras). However, unlike the embodiment of Figure 3, the RGB sensor 32 is positioned in front of the galvo scanning mirror 26 and is combined into the field of view of the Lidar through a dichroic mirror 42. In this configuration the Lidar is scanning the field of view with the galvo mirror 26, but the RGB sensor 32 takes continuous images / frames of the same field of view without the need to scan. The RGB matrix 40 can be of a rolling shutter or global shutter type with a time stamp associated with each frame so that the RGB frame can then be assigned to the same frame taken by the lidar sensor 12.

[0047] Figure 5 is a block diagram of another embodiment of a color lidar apparatus 10 according to the disclosure. Similar to the embodiment of Figure 4, the RGB sensor 32 is a 2D matrix 40 (such as a CMOS sensor used for cellphone cameras). However, unlike the embodiment of Figure 4, the RGB sensor 32 is positioned behind the galvo scanning mirror 26 where its field of view is at least partially overlapping the Lidar field of view. In this case, the galvo mirror 26 is composed of a dichroic mirror that reflects the IR wavelength of the Lidar and transmits the RGB visible wavelengths coming from the targets in the field of view. This configuration is more efficient than the configuration of Figure 4 and reduces the need for an additional dichroic filter. The Figure 5 configuration also provides a better fit between the field of view of the Lidar and the RGB sensor 32. Similar to the configuration of Figure 4, the RGB matrix 40 can be of a rollingshutter or global shutter types, where in each of the types a time stamp is given to each frame and this RGB frame can then be assigned to the same frame taken by the Lidar.

[0048] Figures 6, 7, 8, 9, 10, and 11 illustrate various synchronization strategies that may be used with the configurations illustrated and described with respect to Figures 1, 2, 3, 4, and 5. There are three (3) main types of strategies to coordinate and match accumulated Lidar and RGB data into a single frame according to the present disclosure. Use of a particular type of strategy is based on the type of RGB sensor that is being used within the Lidar scheme and the way the RGB sensor is integrated into the optical path of the Lidar. The main groups of strategies described in greater detail herein include: 1) RGB collection with a Line sensor; 2) RGB collection with an RGB matrix and Rolling Shutter mechanism; and 3) RGB collection with an RGB matrix and Global shutter mechanism.

[0049] Figure 6 is a graphical representation illustrating Lidar and RGB data collection and Figure 7 is a corresponding flowchart illustrating a first strategy for synchronization of Lidar and RGB data in a system configuration similar to the configurations illustrated and described with respect to Figures 1 and 2. A processor (not shown) of color lidar apparatus 10 performs the operations of the first strategy for synchronization of Lidar and RGB data. The processor is in communication with the lidar sensor 12, the RBG sensor 32, and the galvo scanning mirror 26 to perform the operations. Timing sequence 700 includes receiving and storing Lidar scan line data at 710 and receiving and storing associated RGB line data as represented at 720. The angle of the scanning mirror 26 is determined / read and stored based on a corresponding sensor position signal as represented at 730. The Lidar and RGB pixels are received at the same time point due to the use of properly aligned fiber arrays to collect the light with the Lidar and RGB lines then combined into one frame line as represented at 740. The scanning mirror is then moved to the next angle position as represented at 750 with the process repeated until a full frame has been captured as represented at block 760. The combined Lidar and RGB frame data may then be processed for object identification, etc. as represented at 770 and described in greater detail herein.

[0050] As those of ordinary skill in the art will appreciate with reference to the systems represented in Figures 1 and 2 and scanning / synchronization strategies of Figures 6 and 7, the Lidar line scan is perfectly matched with the RGB scan line and combined through a dichroicmirror 28 to the scanning mirror 26. The alignment is achieved through the use of a line of fibers that collect the light and direct it to the lidar detection line 14 and to the RGB detection line 34. While the scanning mirror 26 is moving, both RGB and Lidar lines are read and sent to the processor that builds a point cloud frame of both XYZ position and RGB color information for each pixel. Optional subsequent processing of the combined frame data can be used to identify objects or provide various other data based on processing one or more frames contained within the point cloud.

[0051] Figure 8 is a graphical representation illustrating Lidar and RGB data collection / synchronization and Figure 9 is a corresponding flowchart illustrating a second strategy for synchronization of Lidar and RGB data in a system configuration similar to the configurations illustrated and described with respect to Figure 3. The processor performs the operations of the second strategy for synchronization of Lidar and RGB data. Timing sequence 900 includes receiving and storing Lidar scan line data matched with the scanning mirror angle / movement at 910. Receiving and storing associated RGB line data from the rolling shutter of a CMOS sensor 40 is represented at 920. The angle of the scanning mirror 26 is determined / read and stored based on a corresponding sensor position signal along with the rolling shutter speed and phase relative to the scanning mirror as represented at 930. The Lidar and RGB lines are combined into one frame based on matching time stamps as represented at 940 with the process repeated until a full frame has been captured as represented at block 950. The combined Lidar and RGB frame data may then be processed for object identification, etc. as represented at 960 and described in greater detail herein.

[0052] As those of ordinary skill in the art will appreciate with reference to the systems represented in Figure 3 and the scanning / synchronization strategy represented in Figures 8 and 9, the Lidar line scan is perfectly matched with an RGB rolling shutter CMOS matrix 40 and combined through a dichroic mirror 28 to the scanning mirror 26. While the scanning mirror 26 is moving in a pre-defined speed and scan pattern, and the Lidar line is perfectly matched with this movement. While the RGB rolling shutter 40 may accumulate pixel lines in a different speed and pattern relative to the mirror, this may be accommodated by accumulating the RGB lines in a separate memory. After accumulating the Lidar pixels and the RGB pixels having different timestamps, the programmed processor may perform matching between the RGB line and Lidar line based on the knowledge of the respective line speeds so that corresponding RGB and Lidar pixels may be associated with one another. The RGB and Lidar matched pixels are then sent to the processor that builds and stores a point cloud frame of both XYZ position and RGB color of each pixel in an associated non-transitory volatile or non-volatile computer readable storage medium or memory.

[0053] Figure 10 is a graphical representation illustrating Lidar and RGB data collection / synchronization and Figure 11 is a corresponding flowchart illustrating a third strategy for synchronization of Lidar and RGB data in a system configuration similar to the configurations illustrated and described with respect to Figures 3, 4, and 5. The processor performs the operations of the third strategy for synchronization of Lidar and RGB data. Timing sequence 1100 includes receiving and storing Lidar scan line data matched with the scanning mirror angle / movement at 1110. Receiving and storing associated RGB line data from a global shutter CMOS matrix 40 that is exposed at a certain time when the mirror is at the center of its scan sector is represented at 1120. The Lidar frame (created from lines across the scanning mirror scan sector) and the RGB frame taken from the RGB matrix at the center of the scan are combined as represented at 1130. As represented at 1140, RGB values are then assigned to each Lidar pixel based on the frame matching performed at 1130. The combined Lidar and RGB frame data may then be processed for object identification, etc. as represented at 1150 and described in greater detail herein.

[0054] As those of ordinary skill in the art will appreciate with reference to the systems represented in Figures 3, 4, and 5 and the scanning / synchronization strategy represented in Figures 10 and 11, the Lidar line scan is perfectly matched with an RGB global shutter CMOS matrix 40 and combined through a dichroic mirror 42 to the scanning mirror 26. While the scanning mirror 26 is moving in a pre-defined or otherwise determined scan pattern at a pre-defined or otherwise determined speed, and the Lidar line is perfectly matched with this movement, the RGB global shutter 40 can collect a full frame at any given time relative to the mirror 26 position. Providing a short exposure of the RGB CMOS sensor 40 at the center point of the scan allows later matching to the Lidar frame with the RGB frame with assigning of RGB data to each Lidar pixel at the scan field of view.

[0055] Figure 12A illustrates representative LiDAR 3D data represented by color gradients and Figure 12B illustrates fused LiDAR and RGB color image data. Figures 12A and 12B illustrate one example of fused or combined Lidar and RGB point cloud output combining lidar 3D points with aligned / synchronized RGB color data from an associated camera. The resulting color-3D point are visualized through side by side images of a color gradient and a true color image of the same scene.

[0056] Figure 13A illustrates color LiDAR point cloud data and Figure 13B illustrates operation of object identification and color snippet segmentation using color LiDAR data according to one or more embodiments of the disclosure. Object identification and segmentation modules executed by a programmed processor on the combined Lidar and RGB point cloud data may be used to perform object identification and color snippet segmentation of 3D objects (in this case a person 50 standing about 10m away from the system). In the example of Figures 13A and 13B, a person is detected / identified based on processing of the point cloud data with the detection being marked by the blue bounding box around the object 48. A 2D color segment extracted from the camera images that are aligned with the Lidar is presented in Figure 13B. The value of the RGB data added to the database of the object 48 is clear in that various details of the person 50, such as clothes, posture, etc., may be analyzed with other inferences made, such as intent (i.e. whether the person looks at the Lidar or not may provide insight with respect to intention and alertness of the person to the system).

[0057] As such, those of ordinary skill in the art will appreciate that one or more embodiments described herein may provide associated significant advantages over traditional independent lidar and camera systems for capturing multidimensional color scene data, such as the combination of high resolution 2D color imagery with 3D spatial data to enable more detailed 3D environmental models with realistic colors and textures, for example. Such enhanced 3D perception improves scene understanding capabilities. Accurate, near perfect alignment of RGB camera and Lidar fields of view along the same axis without parallax using dichroic mirror optics avoids laborious error-prone software registration strategies commonly found in the prior art. Triggering the camera shutter when the lidar mirror is at set points enables time-matched captures of camera and lidar data streams to provide synchronized frames. Unlike other fusion approaches,both lidar and camera retain full independent configurations for range, resolution, frame rate, etc. reducing or eliminating resolution compromises common to prior strategies. Separate outputs provide the ability to access both 2D images and 3D point clouds independently with custom parameters to provide flexibility and efficiency. Aligned 3D and 2D data streams enable sophisticated processing with advance algorithms to enable object classification, property identification, surface reconstruction, and point cloud completion, for example. Use of modular hardware such as individual lidar, camera, optical, and electronic components provides flexibility compared to fully integrated sensors.

[0058] While representative embodiments are described above, it is not intended that these embodiments describe all possible forms of the claimed subject matter. The words used in the specification are words of description rather than limitation, and it is understood that various changes may be made without departing from the scope of the disclosure and claimed subject matter. Additionally, the features of various implementing embodiments may be combined to form further embodiments not explicitly described or illustrated, but within the scope of the disclosure and claimed subject matter and recognizable to one of ordinary skill in the art. Various embodiments may have been described as providing advantages or being preferred over other embodiments or prior art implementations with respect to one or more desired characteristics. As one of ordinary skill in the art is aware, one or more features or characteristics may be compromised to achieve desired overall system attributes, which may depend on the specific application and implementation. These attributes include, but are not limited to cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, serviceability, weight, manufacturability, ease of assembly, etc. Embodiments described as less desirable than other embodiments or prior art implementations with respect to one or more characteristics are not necessarily outside the scope of the disclosure and may be desirable for certain applications.

Claims

WHAT IS CLAIMED IS:

1. A color lidar system comprising: a lidar sensor that emits laser light and detects reflected laser light to scan an environment in three dimensions and output a three-dimensional (3D) point cloud containing spatial position data of the environment; a camera having a color sensor to output two-dimensional (2D) color images of the environment, the camera being aligned with the lidar sensor; a dichroic mirror between the lidar sensor and the camera to reflect the laser light and transmit visible light from the environment to align the fields of view of the lidar sensor and the camera without parallax; and a processor programmed to combine the 3D point cloud from the lidar sensor and the 2D color images from the camera in real time to minimize distortion, time differences, and parallax between pixels using synchronization and calibration techniques to generate a fused 3D color point cloud.

2. The color lidar system of claim 1 further comprising: a scanning galvanometric mirror in communication with the dichroic mirror.

3. The color lidar system of claim 2 wherein: the processor is further programmed to trigger a shutter of the camera at set positions during a sweep or scan of the scanning galvanometric mirror to synchronize the 2D color images output by the camera with the scanning of the environment by the lidar sensor thereby creating aligned frames of the 2D color images and the 3D point cloud for combining the 3D point cloud and the 2D color images to generate the fused 3D color point cloud.

4. The color lidar system of claim 3 wherein: the processor is further programmed to provide synchronized instances of the shutter of the camera and scanning angles of the scanning galvanometric mirror to create the aligned frames of the 2D color images and the 3D point cloud.

5. The color lidar system of claim 3 wherein: the processor is further programmed to vary a scanning angle of the scanning galvanometric mirror at a scanning speed that is variable at different horizontal rates and to trigger the shutter of the camera only when the scanning galvanometric mirror reaches predetermined positions independent of the scanning speed and to integrate data of the 2D color images horizontally into the 3D point cloud based on the scanning speed to provide variable horizontal resolution in the fused 3D color point cloud reflecting the scanning speed.

6. The color lidar system of claim 1 wherein: the lidar sensor further outputs a 2D image buffer containing a high resolution image of the environment in two dimensions, wherein the 2D image buffer and the 3D point cloud are accessible separately and utilizable independently, and a frame rate and a resolution of the 2D image buffer are different than a frame rate and a resolution of the 3D point cloud.

7. The color lidar system of claim 1 wherein: the processor is further programmed to analyze the 3D point cloud to identify objects within the environment.

8. The color lidar system of claim 7 wherein: the processor is further programmed to match identified objects with corresponding subsets of the 2D color images.

9. The color lidar system of claim 7 wherein: the processor is further programmed to extract color snippets segmented from the 2D color images based on location and boundaries of the identified objects.

10. The color lidar system of claim 1 wherein: the dichroic mirror is positioned at a 45 degree angle to the optical axes of the lidar sensor and the camera.

11. The color lidar system of claim 1 wherein: the lidar sensor is configured to timestamp each 3D point cloud output; the camera is configured to timestamp each 2D color image output; and the processor is further programmed to execute a synchronization algorithm to align the timestamps of the lidar and the camera data to identify the 3D point cloud output and the 2D color image output to be combined to generate the fused 3D color point cloud.

12. The color lidar system of claim 1 wherein: the color sensor of the camera is a red-green-blue (RGB) sensor.

13. The color lidar system of claim 1 wherein: the color sensor of the camera is a line sensor, a 2D matrix sensor having a rolling shutter, or a 2D matrix sensor having a global shutter.

14. The color lidar system of claim 1 wherein: the color sensor of the camera is in communication with the dichroic mirror via coupling optics, the coupling optics either including (i) a plurality of optical fibers and at least one lens or (ii) at least one lens without any optical fibers.

15. The color lidar system of claim 1 further comprising: a scanning galvanometric mirror in communication with the dichroic mirror; and wherein the camera is positioned in front of the scanning galvanometric mirror and is combined into a field of view of the lidar sensor through a second dichroic mirror.

16. A method for use with a color lidar system including a lidar sensor configured to emit laser light and detect reflected laser light to scan an environment in three dimensions and to output a three-dimensional (3D) point cloud containing spatial position data of the environment, a camera having a color sensor aligned with the lidar sensor and configured to output two-dimensional (2D) color images of the environment, and a dichroic mirror between thelidar sensor and the camera to reflect the laser light and transmit visible light from the environment to align the fields of view of the lidar sensor and the camera without parallax, the method comprising: combining the 3D point cloud from the lidar sensor and the 2D color images from the camera in real time to minimize distortion, time differences, and parallax between pixels using synchronization and calibration techniques to generate a fused 3D color point cloud.

17. A method for use with a color lidar system including a lidar sensor, a camera aligned with the lidar sensor, a dichroic mirror between the lidar sensor and the camera, and a scanning galvanometric mirror in communication with the dichroic mirror, the method comprising: positioning the scanning galvanometric mirror at multiple positions one at a time; colleting lidar scan line data of an environment from the lidar sensor at each position of the scanning galvanometric mirror; collecting color sensor line data of the environment from the camera at each position of the scanning galvanometric mirror; at each position of the scanning galvanometric mirror, combining the lidar scan line data at that position and the color sensor line data at that position into one frame line to thereby generate a full frame including multiple frame lines respectively corresponding to the multiple positions of the scanning galvanometric mirror; and processing the full frame for object identification in the environment.

18. The method of claim 17 wherein the camera has a rolling shutter, the method further comprising: at each position of the scanning galvanometric mirror, reading an angle of the scanning galvanometric mirror, a speed of the rolling shutter of the camera, and a phase of the rolling shutter of the camera relative to the scanning galvanometric mirror; and wherein combining the lidar scan line data and the color sensor line data to thereby generate the full frame includes combining the lidar scan line data and the color sensor line data into the full frame based on time stamp matching of the lidar scan line data and the color sensor line data.

19. The method of claim 17 wherein the camera has a global shutter, the method further comprising: collecting the color sensor line data of the environment from the camera at each position of the scanning galvanometric mirror by capturing the color sensor line data at a center of a scan sector of the scanning galvanometric mirror at each position of the scanning galvanometric mirror; wherein combining the lidar scan line data and the color sensor line data to thereby generate the full frame includes combining the lidar scan line data across the scan sector of the scanning galvanometric mirror and the color sensor line data at the center of the scan sector of the scanning galvanometric mirror.

20. The method of claim 19 further comprising: assigning color values to each pixel of the lidar scan line data based on frame matching.

Images