Vizualizing objects being tracked using multiple sensors

The method addresses inaccuracies in merging nonsynchronized sensor data by estimating object characteristics based on displacement and time differences, enhancing object visualization accuracy in dynamic scenarios.

WO2026235580A1PCT designated stage Publication Date: 2026-11-12TRATON AB
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/SE2026/010160
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-09
Filing Date
2026-04-13
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Conventional object tracking systems in autonomous vehicles face challenges in merging nonsynchronized sensor data from multiple perception sensors, particularly in dynamic scenarios, leading to inaccuracies in visualizing tracked objects due to time differences in data capture.

Method used

A method that estimates object characteristics using displacement and time differences between sensor data points to render a merged image, transforming 3D positions and poses into 2D images, ensuring accurate visualization of tracked objects.

Benefits of technology

Enhances the accuracy of object visualization by aligning 3D sensor data with 2D images, improving the performance of perception algorithms and simplifying verification processes in dynamic environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure SE2026010160_12112026_PF_FP_ABST
    Figure SE2026010160_12112026_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to A method performed by a control arrangement configured to visualize a tracked object in a scene (110) observed by a plurality of perception sensors comprised in a vehicle, the tracked object moving along a first three-dimensional, 3D, trajectory (130) within the scene (110), the method comprising: obtaining first data, the first data being indicative of first tracked object characteristics, the tracked object characteristics at least being indicative of: a first 3D position (P1) along the first 3D trajectory (130) at a first point in time (T1), a first 3D object pose at the first point in time (T1), and a kinematic model of the first 3D trajectory (130), obtaining second data indicative of a two-dimensional, 2D, image of the scene (110) at a second point in time (T2), the second point in time being different than the first point in time, obtaining third data at least indicative of a second 3D trajectory (140) of the vehicle between the first point in time and the second point in time, rendering a merged image using the first data, the second data and the third data by estimating a second object characteristics, the second object characteristics comprising at least a second position and a second object pose of the tracked object at the second point in time, and rendering the merged image using the second position and a second object pose, visualizing the rendered image.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] VIZUALIZING OBJECTS BEING TRACKED USING MULTIPLE SENSORS TECHNICAL FIELD

[0002] The present invention relates to visualization of moving objects.

[0003] BACKGROUND

[0004] With the increasing trend of fully autonomous and semi-autonomous vehicles, the ability to detect and track objects in the environment of the vehicle is crucial for safe driving of the vehicle.

[0005] Some conventional solutions rely on object racking using point cloud-based perception sensors, such as Radio, Detection and Ranging, RADAR, or Light Detection and Ranging, LIDAR. Most modern LiDAR and RADAR sensors deliver 3D point clouds.

[0006] The object is typically tracked in three dimensions, i.e., in 3D.

[0007] The tracked objects are often visualized in a bounding box and being overlaid on a two -dimensional, 2D, image, e.g., on a map or an image captured by a front facing camera.

[0008] The perceived accuracy of the visualized object will depend on many links in a perception chain, such as sensor calibration, object detector quality or quality of a machine learning model.

[0009] A further challenge is that the perception chain typically comprises multiple sensors nonsynchronized sensors whose outputs needs to be merged to a single merged image. The nonsynchronized sensors will register the object at different time instances making the merge challenging, especially in highly dynamic scenarios, such as driving on highways.

[0010] Thus, there is a need for improving the performance of perception algorithms tracking objects.

[0011] OBJECTS OF THE INVENTION

[0012] An objective of embodiments of the present invention is to provide a solution which mitigates or solves the drawbacks described above.SUMMARY OF THE INVENTION

[0013] The above and further objectives are achieved by the subject matter described herein. Further advantageous implementation forms of the invention are described herein. The invention is set out in the appended claims.

[0014] According to a first aspect, the objectives are achieved by a method performed by a control arrangement configured to visualize a tracked object in a scene observed by a plurality of perception sensors comprised in a vehicle, the tracked object moving along a first three-dimensional, 3D, trajectory within the scene, the method comprising: obtaining first data, the first data being indicative of first tracked object characteristics, the tracked object characteristics at least being indicative of: a first 3D position along the first 3D trajectory at a first point in time, a first 3D object pose at the first point in time, and a kinematic model of the first 3D trajectory, obtaining second data indicative of a two-dimensional, 2D, image of the scene at a second point in time, the second point in time being different than the first point in time, obtaining third data at least indicative of a second 3D trajectory of the vehicle between the first point in time and the second point in time, rendering a merged image using the first data, the second data and the third data by estimating second object characteristics, the second object characteristics comprising at least a second position and a second object pose of the tracked object at the second point in time, and rendering the merged image using the second position and the second object pose, visualizing the rendered image.

[0015] In one embodiment, the second object characteristics is estimated using a displacement of the object, a displacement of the vehicle and a difference in time between the first point in time and the second point in time, the second object characteristics comprising at least a second 3D position and a second 3D object pose of the tracked object at the second point in time, wherein rendering the merged image comprises projecting the tracked object onto a 2D position of a 2D plane of the 2D image using the estimated second 3D position and the estimated second 3D object pose.

[0016] In one embodiment, the first data is provided in a coordinate system of the vehicle, wherein the third data is provided in a real-world coordinate system, wherein the method further comprises: transforming the first 3D position to the real-world coordinate system, deriving a position of the vehicle at the second point in time from a second 3D trajectory of the vehicle and estimating the second 3D position and the second 3D object pose of the tracked object at the second point in time using the derived position of the vehicle, transforming the estimated second 3D position and the second 3D object pose from the real-world coordinate system to the coordinate system of the vehicle, projecting the transformed and estimated second 3Dposition and the second 3D object pose onto the 2D position of the 2D plane of the 2D image using the second 3D trajectory.

[0017] In one embodiment, the tracked object characteristics is further being indicative of a first 3D velocity and a first 3D heading, wherein estimating the second object characteristics is performed by: projecting the tracked object characteristics onto a 2D plane of the 2D image, the projected object characteristics comprising an intermediate 2D position, 2D pose, a 2D velocity and a 2D heading, determining a final 2D position by determining a displacement of the intermediate 2D position in the 2D plane of the 2D image, wherein the displacement of the intermediate 2D position is determined using a difference between the second point in time and the first point in time, the 2D pose, a 2D velocity and a 2D heading, wherein rendering the merged image comprises rendering the tracked object at the final 2D position.

[0018] In one embodiment, the tracked object characteristics comprises a selection of any of heading of the tracked object, velocity of the tracked object acceleration of the tracked object, altitude of the tracked object, orientation of a respective bounding shape.

[0019] According to a second aspect, the objectives are achieved by a control arrangement, the control arrangement comprising: a processor, and a memory, said memory containing instructions executable by said processor, wherein said control arrangement is operable to perform the method according to the first aspect.

[0020] According to a third aspect, the objectives are achieved by a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to the first aspect.

[0021] According to a fourt aspect, the objectives are achieved by a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to the first aspec.

[0022] Reference will be made to the appended sheets of drawings that will first be described briefly.

[0023] BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Fig. 1 illustrates a three-dimensional, 3D, trajectory of a tracked object.

[0025] Fig. 2 illustrates movement of a tracked object and an ego vehicle.

[0026] Fig. 3A-B illustrates an example of estimating object characteristics according to one or more embodiments of the present disclosure.Fig. 4A illustrates a further example of estimating object characteristics according to one or more embodiments of the present disclosure.

[0027] Fig. 4B illustrates an intermediate 2D position and a final 2D position on the 2D plane Fig. 5 shows a flowchart of a method according to one or more embodiments of the present disclosure.

[0028] Fig. 6 shows a computer according to one or more embodiments of the present invention. Fig. 7 illustrates a system for visualizing a tracked object in a scene being observed by a plurality of perception sensors comprised in a vehicle.

[0029] A more complete understanding of embodiments of the invention will be afforded to those skilled in the art, as well as a realization of additional advantages thereof, by a consideration of the following detailed description of one or more embodiments. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures.

[0030] DETAILED DESCRIPTION

[0031] As described in the background section, the present disclosure relates to object detection and tracking in the environment of vehicles. The present disclosure further relates to visualization of such tracked objects.

[0032] Vehicles provided with multiple perception sensors typically first processes sensor data individually and then aggregate the sensor data captured during a defined time interval, sometimes referred to as a frame.

[0033] Vehicles, in particular autonomous vehicles, are typically equipped with different perception sensors, e.g., lidar, cameras, and radar sensors.

[0034] Active sensors such as lidar or radars send out a pulse of energy and detect the changes in the return signal. To avoid interferences, it is desirable that these sensors do not scan the same area at same time.

[0035] In one example, a lidar sensor registers an object in a real-world scene at a first point in time, and an image sensor captures a 2D image of the scene at a second point in time. When aggregating the data for visualization this will cause a shift of the objects position in the image. This is due to the difference in time between the first point in time and the second point in time, during which time both the tracked object and the ego vehicle carrying the sensors have moved.Machine learning, ML, solutions are expected to be crucial for reliable perception for autonomous driving vehicles. A very common approach is to train the ML solution using supervised learning, i.e., providing sensor data as input and 3D bounding boxes are expected as output. Other approaches, such self-supervised learning, that might not require reference 3D bounding boxes during the training phase. However, no matter which solution is used for detecting 3D objects, reference 3D objects are needed during verification phase. Reference 3D objects are typically obtained through an annotation process, which is typically performed by companies specialized for this task.

[0036] Many annotated datasets comprise of camera and lidar sensor data, where the lidars are phase coordinated, i.e., they illuminate objects at the same time. In other words, they “see” the tracked object at the same time.

[0037] The lidar data used for one annotation instance typically corresponds to a complete lidar sweep. For mechanically rotating lidars, this typically correspond to a time interval of 100 ms. Therefore, an annotation instance corresponds in fact to a time interval. It is also common that cameras are also synchronized with the lidars. Moreover, many annotated datasets correspond to low dynamic scenarios, in which both the data colleting vehicle (aka. ego vehicle) and other dynamic objects move relatively slowly. Linder these circumstances, it is possible to define a reference timestamp per annotated time interval. Typically, this is a timestamp where the lidar beams illuminate an object, later visualized using a 2D image from a reference camera.

[0038] As verification, the 3D bounding boxes are back-projected onto the reference camera image. Since all sensors would “see” an object at the same time, one would expect a perfect match between the projected 3D bounding boxes and the object in the camera image.

[0039] As mentioned in the background section, in a 360 degrees perception system with many sensors, it is likely that different sensors will record data at different time instances within the annotation interval.

[0040] For instance, if the reference annotation timestamp is chosen in the middle of a 100ms annotation interval, objects could be seen by lidar beams or camera snapshots at timestamps differing as much as + / - 50 ms difference with respect to the refence timestamp. Even if the ego vehicle would be static, in a highly dynamic environment with vehicles moving at 30 meters per second or more could have moved with 1.5 m or more within 50 ms. A 1.5 m displacement of an object could correspond to a very large angular change in the field of view of a sensor.Hence, the same object could be detected by two or more sensors at different time instances. If the detected object has high velocity, then each sensor may observe the object in different positions. Therefore, a sensor fusion algorithm is typically selecting a reference time at which is estimates the object position; this reference time might be quite different that the time instances when each individual sensor observes the object. In particular, a camera timestamp could be quite different than the reference timestamp at which the fusion algorithm is estimated position. Therefore, if the 3D bounding box position provided by the fusion algorithm at a reference timestamp is projected onto a camera image, then there could be a mismatch between the projected 3D bounding box and the 2D position of the object in the image. The mismatch is expected to be larger if the difference between the camera timestamp and the reference timestamp is big; it is also expected to be larger if the object velocity is larger. In the following, we use a fusion algorithm or a perception algorithm to refer to any object detection procedure, including a human-assisted annotation procedure. Also, reference time and annotation time will be used interchangeably to refer to the time for which the fusion algorithm is estimating the object position. ’’The present solution assumes that the 3D bounding box is described by a 3D pose at the annotation timestamp, and also by a 3D velocity vector. This information is then used to match the annotated bounding boxes with the timestamp of the camera image.

[0041] A first approach of the present disclosure is to consider a time difference between the annotation timestamp and the camera timestamp.

[0042] Then use the 3D pose and velocity at the reference timestamp and the time difference with regards the camera timestamp to estimate the pose of the 3D object at the camera timestamp. Then the 3D bounding box position can be visualized at the camera timestamp instead of the annotation reference timestamp. If more than one camera image is used for visualization, then the 3D bounding boxes position will be visualized at different timestamp in each image.

[0043] A further approach of the present disclosure is to consider the time difference between the (annotation) timestamp and the camera timestamp.

[0044] Then project the 3D pose of the annotation into the 2D image at the reference timestamp. Then use the projected 2D position and 3D velocity at the reference timestamp and the time difference with regards to the camera timestamp to estimate the position of the 2D object at the camera timestamp.Then the 2D bounding box position is visualized at the camera timestamp instead of the annotation reference timestamp. If more than one camera image is used for visualization, then the 2D bounding boxes position will be visualized at different timestamp in each image.

[0045] The use of the disclosed method is not limited to the annotation process and can be used for visualizing the output from any perception algorithm, replacing the annotation timestamp with the timestamp for which the detected object position is estimated.

[0046] The disclosed method simplifies verification processes in a perception chain. For example, it can be used for the visual inspection of how good the output of an object detection algorithm is or how well calibrated a perception sensor is.

[0047] The method can also be used in an vehicle to visualize detected objects in the 3D space onto the camera images to view the current situation around the vehicle to a passenger or safety driver in a precise manner.

[0048] The disclosure is further presented below.

[0049] Fig. 1 illustrates a three-dimensional, 3D, trajectory 130 of a tracked object.

[0050] The trajectory describes the movement of the object, or object characteristics of the tracked object, such as 3D position in a real-world coordinate system120. An example of a real-world coordinate system120 is e.g., the World Geodetic System 1984.

[0051] The object characteristics may further include a 3D object pose, a 3D velocity and a kinematic model of the first 3D trajectory ,or any other suitable information.

[0052] Fig. 2 illustrates movement of a tracked object and an ego vehicle. The ego vehicle is typically the vehicle on which the perception sensors are mounted on.

[0053] As mentioned above, a first 3D position at a first point in time when the tracked object is registered by a perception sensor, e.g., a lidar, needs to be transferred to 2D position a second point in time when a two-dimensional, 2D, image is captured, onto which the object should be visualized. Additionally, besides the difference in time of detection of the object, the movement of the ego vehicle relative to the tracked object will also heavily influence rendering of the 3D position onto a 2D plane of the 2D image.

[0054] The tracked object moves along a 3D trajectory 130, and the ego vehicle moves along the 3D trajectory 140.

[0055] The trajectory 140 describes the movement of the vehicle, or vehicle characteristics of the vehicle, such as 3D position in a real-world coordinate system120. An example of a real-world coordinate system120 is e.g., the World Geodetic System 1984.In one example, at a first point in time, the object is detected at position P1 and the ego vehicle is detected at a position VP1. At a third point in time, the object is detected at position P2 and the ego vehicle is detected a position VP2. The object has been displaced a distance D1 between the first point in time and the third point in time. The ego vehicle has been displaced a distance D2 between the first point in time and the third point in time.

[0056] A camera capturing the scene with a 2D image, might capture the 2D image at a second point in time, the second point in time being located between the first and third point in time. In other words, the positions of the tracked object P1, P2 are detected at different points in time to when 2D images are captured.

[0057] The present disclosure estimates object characteristics for the tracked object at the second point in time, e.g., velocity, heading, position and pose. The object characteristics may be estimated using the mentioned displacements D1, D2 and / or the time difference between the first and second point in time.

[0058] Fig. 3A illustrates an example of estimating object characteristics according to one or more embodiments of the present disclosure. Any of the time difference between the first and second point in time, a 3D position of the object P1 , a 3D pose / movement direction and 3D velocity of the object at the first point in time is used to estimate object characteristics of the tracked object at the second point in time.

[0059] The object characteristics at the second point in time may be estimated as a 3D position P2’, a 3D pose / movement direction and 3D velocity of the object at the second point in time T2, and then transformed to the 2D image plane to visualize the tracked object in the 2D image, e.g. at a 2D position P2”. In other words, the movement direction and velocity is used together with the time difference between the first and second point in time to find object characteristics , such as an estimated 3D position.

[0060] Fig. 3B illustrates the position of the traced object on a 2D plane.

[0061] The figure shows the estimated object characteristics being projected onto a 2D position P2” on the 2D plane 310 of the 2D image.

[0062] Fig. 4A illustrates a further example of estimating object characteristics according to one or more embodiments of the present disclosure.

[0063] Alternatively, the 3D position P1 and the 3D pose / movement direction at the first point in time is directly transformed to the 2D image plane, e.g., as an intermediate 2D position P1”. The 3D velocity of the object at the first point in time is further transformed to a 2D velocity of the object at the first point in time, and then used together with the time difference between thefirst and second point in time to determine a displacement D3 and estimate a 2D position P2” at the second point in time T2 . The estimated 2D position P2” can then be used to visualize the tracked object in the 2D image.

[0064] Fig.4B illustrates an intermediate 2D position P1 ’ and a final 2D position P2” on the 2D plane. The figure shows the estimated object characteristics being projected onto a 2D position P2” on the 2D plane 310 of the 2D image.

[0065] The final 2D position P2”is determined by first determining a displacement D3 of the intermediate 2D position P1” in the 2D plane of the 2D image. The displacement D3 of the intermediate 2D position P1” is determined using a difference between the second point in time T2 and the first point in time T1, a 2D pose, a 2D velocity and a 2D heading obtained by transforming characteristics of the tracked object.

[0066] Fig. 5 shows a flowchart of a method 500 according to one or more embodiments of the present disclosure. The method is performed by a control arrangement configured to visualize a tracked object in a scene 110 observed by a plurality of perception sensors comprised in a vehicle. The tracked object moving along a first three-dimensional, 3D, trajectory 130 within the scene 110, the method comprising:

[0067] Step 510: obtaining first data, the first data being indicative of first tracked object characteristics, the tracked object characteristics at least being indicative of:

[0068] a first 3D position P1 along the first 3D trajectory 130 at a first point in time T 1 ,

[0069] a first 3D object pose at the first point in time T 1 , and

[0070] a kinematic model of the first 3D trajectory 130,

[0071] Step 520: obtaining second data indicative of a two-dimensional, 2D, image of the scene 110 at a second point in time T2, the second point in time being different than the first point in time, Step 530:obtaining third data at least indicative of a second 3D trajectory 140 of the vehicle between the first point in time and the second point in time,

[0072] Step 540: rendering a merged image using the first data, the second data and the third data by estimating second object characteristics, the second object characteristics comprising at least a second position and a second object pose of the tracked object at the second point in time T2, and rendering the merged image using the second position and the second object pose,

[0073] Optional Step 550: visualizing the rendered image.In embodiments, the second object characteristics is estimated using a displacement of the object, a displacement of the vehicle and a difference in time between the first point in time T1 and the second point in time T2, the second object characteristics comprising at least a second 3D position P2’ and a second 3D object pose of the tracked object at the second point in time T2, wherein rendering the merged image comprises projecting the tracked object onto a 2D position P2” of a 2D plane of the 2D image using the estimated second 3D position P2’ and the estimated second 3D object pose.

[0074] In embodiments, the first data is provided in a coordinate system of the vehicle, wherein the third data is provided in a real-world coordinate system 120, wherein the method further comprises: transforming the first 3D position P1 to the real-world coordinate system 120, deriving a position of the vehicle at the second point in time T2 from a second 3D trajectory 140 of the vehicle and estimating the second 3D position P2’ and the second 3D object pose of the tracked object at the second point in time using the derived position of the vehicle, transforming the estimated second 3D position P2’ and the second 3D object pose from the real-world coordinate system 120 to the coordinate system of the vehicle, projecting the transformed and estimated second 3D position and the second 3D object pose onto the 2D position P2” of the 2D plane of the 2D image using the second 3D trajectory 140.

[0075] In embodiments, the tracked object characteristics is further being indicative of a first 3D velocity and a first 3D heading, wherein estimating the second object characteristics is performed by: projecting the tracked object characteristics onto a 2D plane of the 2D image, the projected object characteristics comprising an intermediate 2D position (P1 ”), 2D pose, a 2D velocity and a 2D heading, determining a final 2D position (P2”) by determining a displacement of the intermediate 2D position (P1 ”) in the 2D plane of the 2D image, wherein the displacement (D3) of the intermediate 2D position (P1”) is determined using a difference between the second point in time (T2) and the first point in time (T 1 ), the 2D pose, a 2D velocity and a 2D heading, wherein rendering the merged image comprises rendering the tracked object at the final 2D position (P2”).

[0076] In embodiments, the tracked object characteristics comprises a selection of any of heading of the tracked object, velocity of the tracked object acceleration of the tracked object, altitude of the tracked object, orientation of a respective bounding shape.

[0077] Fig. 6 shows a computer 600 according to one or more embodiments of the present invention. The computer may e.g., be in the form of an Electronic Control Unit, a server, an on-board computer, a control arrangement, a vehicle mounted computer system or a navigation device. The computer may comprise a processor or processing means 612 communicatively coupledto a transceiver 604 configured for wired or wireless communication. Further, the computer may further comprise at least one optional antenna (not shown in figure). The antenna may be coupled to the transceiver 604 and is configured to transmit and / or emit and / or receive wireless signals in a wireless communication system, e.g., wireless signals comprising diagnostic actions or diagnostic data. In one example, the processor 612 may be any of a selection of processing circuitry and / or a central processing unit and / or processor modules and / or multiple processors configured to cooperate with each other. Further, the computer may further comprise a memory 615. The memory 615 may contain instructions executable by the processor to perform any of the methods described herein. The memory and / or computer-readable storage medium referred to herein may comprise of essentially any memory, such as a ROM (Read-Only Memory), a PROM (Programmable Read-Only Memory), an EPROM (Erasable PROM), a Flash memory, an EEPROM (Electrically Erasable PROM), ora hard disk drive.

[0078] In a further embodiment, the computer may further comprise and / or be coupled to one or more sensors (not shown) configured to e.g., receive and / or obtain and / or measure physical properties pertaining to the vehicle 10Oor object and send one or more control signals or sensor signals indicative of the physical properties to the processing means 612.

[0079] In one or more embodiments the computer may further comprise an input device 617, configured to receive input or indications from a user and send a user-input signal indicative of the user input or indications to the processor or processing means 612.

[0080] In one or more embodiments the computer may further comprise a display 6188 configured to receive a display signal indicative of rendered objects, such as text or graphical user input objects, from the processor or processing means 612 and to display the received signal as objects, such as text or graphical user input objects.

[0081] In one embodiment the display 618 is integrated with the user input device 617 and is configured to receive a display signal indicative of rendered objects, such as text or graphical user input objects, from the processing means 612 and to display the received signal as objects, such as text or graphical user input objects, and / or configured to receive input or indications from a user and send a user-input signal indicative of the user input or indications to the processing means 612.

[0082] In embodiments, the processing means 612 is communicatively coupled to a selection of any of the memory 615 and / or the communications interface and / or transceiver and / or the input device 617 and / or the display 6188 and / or the one or more sensors. In embodiments, the transceiver 404 communicates using wired and / or wireless communication techniques. Thewired or wireless communication techniques may comprise any of a CAN bus, Bluetooth, WiFi, GSM, UMTS, LTE orLTE advanced communications network or any other wired or wireless communication network known in the art.

[0083] The control arrangement, 710, described herein may comprise all, ora selection of the features described in relation to Fig. 4. In other words, the computer 600 may be comprised in the control arrangement 710.

[0084] In one embodiment, a control arrangement is provided, the control arrangement comprising: a processor, and a memory, said memory containing instructions executable by said processor, whereby said control arrangement is operative to perform any of the methods described herein.

[0085] In one embodiment, a computer program (product) is provided and comprises instructions which, when the program is executed by a computer, cause the computer to carry out any of the methods described herein.

[0086] In one embodiment, a computer-readable medium is provided and comprises instructions which, when executed by a computer, cause the computer to carry out any of the methods described herein.

[0087] The term “computer” may be defined as any hardware or hardware / firmware device implemented using processing circuity such as, but not limited to, a processor, Central Processing Unit (CPU), a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a System-on-Chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit, or any other device capable of electronically performing operations in a defined manner.

[0088] In some embodiments, the computer-readable medium may be a non-transitory computer-readable medium, such as a tangible electronic, magnetic, optical, infrared, electromagnetic, and / or semiconductor system, apparatus, and / or device.

[0089] In one embodiment, the computer program product may be a computer readable medium, and the computer program may be stored in the computer readable medium described herein. In embodiments, the communications network herein communicate using wired or wireless communication techniques that may include at least one of a Local Area Network (LAN), Metropolitan Area Network (MAN), Global System for Mobile Network (GSM), Enhanced Data GSM Environment (EDGE), Universal Mobile Telecommunications System, Long term evolution, High Speed Downlink Packet Access (HSDPA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access(TDMA), Bluetooth®, Zigbee®, Wi-Fi, Voice over Internet Protocol (VoIP), LTE Advanced, IEEE802.16m, Wireless MAN-Advanced, Evolved High-Speed Packet Access (HSPA+), 3GPP Long Term Evolution (LTE), Mobile WiMAX (IEEE 802.16e), Ultra Mobile Broadband (UMB) (formerly Evolution-Data Optimized (EV-DO) Rev. C), Fast Low-latency Access with Seamless Handoff Orthogonal Frequency Division Multiplexing (Flash-OFDM), High Capacity Spatial Division Multiple Access (iBurst®) and Mobile Broadband Wireless Access (MBWA) (IEEE 802.20) systems, High Performance Radio Metropolitan Area Network (HIPERMAN), Beam-Division Multiple Access (BDMA), World Interoperability for Microwave Access (WiMAX) and ultrasonic communication, etc., but is not limited thereto.

[0090] Moreover, it is realized by the skilled person that the system and / or devices may comprise the necessary communication capabilities in the form of e.g., functions, means, units, elements, etc., for performing the present solution. Examples of other such means, units, elements and functions are: processors, memory, buffers, control logic, , multipliers, decision units, selecting units, switches, modulators, demodulators, inputs, outputs, antennas, amplifiers, receiver units, transmitter units, DSPs, MSDs, encoder, decoder, power supply units, power feeders, communication interfaces, communication protocols, etc. which are suitably arranged together for performing the present solution.

[0091] Especially, the processor and / or processing means of the present invention may comprise one or more instances of processing circuitry, processor modules and multiple processors configured to cooperate with each-other, Central Processing Unit (CPU), a processing unit, a processing circuit, a processor, an Application Specific Integrated Circuit (ASIC), a microprocessor, a Field-Programmable Gate Array (FPGA) or other processing logic that may interpret and execute instructions. The expression “processor” and / or “processing means” may thus represent a processing circuitry comprising a plurality of processing circuits, such as, e.g., any, some, or all of the ones mentioned above. The processing means may further perform data processing functions for inputting, outputting, and processing of data comprising data Fig. 7 illustrates a system 700 for visualizing a tracked object 720 in a scene being observed by a plurality of perception sensors S1, S2 comprised in a vehicle.

[0092] The system comprises a control arrangement 700.

[0093] The system further comprises at least a first perception sensor S1 and a second perception sensor S2 communicatively coupled to the control arrangement 700.

[0094] The first perception sensor S1 is at least configured to capture tracked object characteristics, such as 3D position of the tracked object.The second perception sensor S2 is at least configured to capture a 2D image of a scene comprising the tracked object.

[0095] The control arrangement 700 being configured or operable to perform any of the methods described herein.

[0096] In one embodiment the control arrangement 700 comprises an object detection unit 730 communicatively coupled to perception sensors S1 and S2, and being configured to track / detect objects. An interface 731 which is configured to provide an description of the tracked / detected detected object 720. The control arrangement 700 further comprises a localization unit 740 which uses an interface 741 to obtain information and provide a description the pose of the ego vehicle at first and second point in time. Alternatively, The localization unit 740 provides a change of pose between first and second point in time. The control arrangement 700 further comprises an interface 751 by which the sensor information from S2 and the timestamp are transmitted. The control arrangement 700 further comprises a visualization unit 770, designed to visualize information, e.g., 2D image information. The control arrangement 700 further comprises a processing unit 760, which receives information about tracked object 720 over interface 731, ego vehicle information over interface 741, and information from a 2D perception sensor S2 over interface 751 and provides visualization information via an interface 761 which provides information on how to visualize information about the tracked object superposed onto the 2D image, so that the displayed bounding box information about object 720 matches the sensor information from S2. Finally, it should be understood that the invention is not limited to the embodiments described above, but also relates to and incorporates all embodiments within the scope of the appended independent claims.

Claims

CLAIMS1. A method performed by a control arrangement configured to visualize a tracked object in a scene (110) observed by a plurality of perception sensors comprised in a vehicle, the tracked object moving along a first three-dimensional, 3D, trajectory (130) within the scene (110), the method comprising:obtaining, from at least one of the plurality of perception sensors, first data, the first data being indicative of first tracked object characteristics, the tracked object characteristics at least being indicative of:a first 3D position (P1) along the first 3D trajectory (130) at a first point in time (T1),a first 3D object pose at the first point in time (T 1 ), anda kinematic model of the first 3D trajectory (130),obtaining, from at least one of the plurality of perception sensors, second data indicative of a two-dimensional, 2D, image of the scene (110) at a second point in time (T2), the second point in time being different than the first point in time, obtaining, from at least one of the plurality of perception sensors, third data at least indicative of a second 3D trajectory (140) of the vehicle between the first point in time and the second point in time,rendering a merged image using the first data, the second data and the third data by estimating second object characteristics based on said data, the second object characteristics comprising at least a second position and a second object pose of the tracked object at the second point in time (T2), and rendering the merged image using the second position and the second object pose, visualizing the rendered image2. The method according to claim 1, wherein the second object characteristics is estimated using a displacement of the object, a displacement of the vehicle and a difference in time between the first point in time (T1) and the second point in time (T2), the second object characteristics comprising at least a second 3Dposition (P2’) and a second 3D object pose of the tracked object at the second point in time (T2),wherein rendering the merged image comprises projecting the tracked object onto a 2D position (P2”) of a 2D plane of the 2D image using the estimated second 3D position(P2’) and the estimated second 3D object pose.

3. The method according to claim 2, wherein the first data is provided in a coordinate system of the vehicle, wherein the third data is provided in a real-world coordinate system (120), wherein the method further comprises:transforming the first 3D position (P1) to the real-world coordinate system (120),deriving a position of the vehicle at the second point in time (T2) from a second 3D trajectory (140) of the vehicle and estimating the second 3D position (P2’) and the second 3D object pose of the tracked object at the second point in time using the derived position of the vehicle, transforming the estimated second 3D position (P2’) and the second 3D object pose from the real-world coordinate system (120) to the coordinate system of the vehicle,projecting the transformed and estimated second 3D position and the second 3D object pose onto the 2D position (P2”) of the 2D plane of the 2D image using the second 3D trajectory (140).

4. The method according to claim 1 , wherein the tracked object characteristics is further being indicative of a first 3D velocity and a first 3D heading, wherein estimating the second object characteristics is performed by:projecting the tracked object characteristics onto a 2D plane of the 2D image, the projected object characteristics comprising an intermediate 2D position (P1”), 2D pose, a 2D velocity and a 2D heading,determining a final 2D position (P2”) by determining a displacement of the intermediate 2D position (P1”) in the 2D plane of the 2D image, wherein the displacement (D3) of the intermediate 2D position (P1 ”) is determined using adifference between the second point in time (T2) and the first point in time (T 1 ), the 2D pose, a 2D velocity and a 2D heading,wherein rendering the merged image comprises rendering the tracked object at the final 2D position (P2”).

5. The method according to any of the preceding claims, wherein the tracked object characteristics comprises a selection of any of heading of the tracked object, velocity of the tracked object acceleration of the tracked object, altitude of the tracked object, orientation of a respective bounding shape.

6. A control arrangement, the control arrangement comprising:a processor, and a memory, said memory containing instructions executable by said processor, wherein said control arrangement is operable to perform the method according to any of claims 1-5.

7. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to claims 1-5.

8. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to claims 1-5.