Method of calibrating camera of trailer attached to vehicle, and ECU thereof

Abstract

The present disclosure relates to method for calibrating camera of trailer (104) attached to vehicle (102). The method comprises receiving rear imaging data from rear camera (110) associated with vehicle (102). Further, the method comprises determining a keyframe from plurality of rear frames, based on plurality of corner features of each of the plurality of rear frames. Furthermore, the method comprises comparing the keyframe with each of a plurality of trailer frames associated with trailer imaging data. The trailer imaging data is received from a trailer camera (108) associated with the trailer (104). Moreover, the method comprises determining matched keypoints corresponding to a trailer frame of the plurality of trailer frames and the keyframe, based on the comparison. Thereafter, the method comprises calibrating the trailer camera (108) based on the matched keypoints, and camera parameters of the rear camera (110) and the trailer camera (108).

Description

[0001] 202403225

[0002] 1

[0003] TITLE: METHOD OF CALIBRATING CAMERA OF TRAILER ATTACHED TO VEHICLE, AND ECU THEREOF

[0004] TECHNICAL FIELD

[0005] The present disclosure generally relates to automotive industries. More particularly, the present disclosure relates to a method and an Electronic Control Unit (ECU) for calibrating a camera of a trailer attached to a vehicle.

[0006] BACKGROUND

[0007] Trailers are unpowered vehicles towed by powered vehicles. Trailers are hinged to vehicles to carry goods or materials of different natures. Generally, such trailers increase the blind spot in rearview of the vehicle by large amounts. To achieve a seamless view behind the trailer attached to the vehicle, a camera is mounted on the rear end of the trailer. Video data from the camera of the trailer is displayed to a driver of the vehicle. Hence, the camera on the trailer (also referred to as a trailer camera) plays an important role in providing visualization behind the trailer to the driver. In addition to the visualization, many trailering functions such as trailer hitch assist and trailer reverse assist also rely on the video data from the trailer camera. Therefore, accurate calibration of the trailer camera is essential to maintain the integrity of the vehicle.

[0008] However, there are many challenges faced in calibration of the trailer camera.

[0009] Trailers are non-standard objects and are bought in aftermarket. Also, mounting location of the trailer camera on the rear end of the trailer is often decided by the driver, and not by Original Equipment Manufacturers (OEMs). These aspects make the calibration of the trailer camera challenging. Particularly, extrinsic calibration of the trailer camera is challenging in such cases.

[0010] Existing techniques for calibrating the trailer camera depends on multiple cameras attached to the trailer. The existing techniques use an approach to calibrate a three-camera setup (two cameras to the side and one to the rear of the trailer body). This multi-camera setup exploits the overlapping field of view and performs 3D-reconstruction. However, this approach is dependent on the multiple trailer cameras202403225

[0011] 2

[0012] and also requires overlapping Field of View (FoV) between cameras. The overlapping FOV constraint requires placement of cameras in close proximity. Some other existing techniques rely on initial or factory calibration or factory setup. These techniques have challenges from inaccuracies in setup and are time-consuming. It requires dedicated equipment and setup in the factory. Also, some existing techniques rely on positioning of a set of markers on a plane to calibrate the trailer cameras. Such methods require accurate planar targets and markers. These techniques are time consuming and have an operational domain limited to factory or service center.

[0013] The information disclosed in this background of the disclosure section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.

[0014] SUMMARY

[0015] In an embodiment, the present disclosure discloses a method for calibrating a camera of a trailer attached to a vehicle. The method comprises receiving rear imaging data from a rear camera associated with the vehicle. The rear imaging data comprises a plurality of rear frames. Further, the method comprises determining a keyframe from the plurality of rear frames, based on a plurality of corner features of each of the plurality of rear frames. Furthermore, the method comprises comparing the keyframe with each of a plurality of trailer frames associated with trailer imaging data. The trailer imaging data is received from a trailer camera associated with the trailer. Moreover, the method comprises determining matched keypoints corresponding to a trailer frame of the plurality of trailer frames and the keyframe, based on the comparison. Thereafter, the method comprises calibrating the trailer camera based on the matched keypoints, and camera parameters of the rear camera and the trailer camera.

[0016] In an embodiment, the present disclosure discloses an Electronic Control Unit (ECU) for calibrating a camera of a trailer attached to a vehicle. The ECU comprises a processor and a memory. The ECU is configured to receive rear imaging data from a202403225

[0017] 3

[0018] rear camera associated with the vehicle. The rear imaging data comprises a plurality of rear frames. Further, the ECU is configured to determine a keyframe from the plurality of rear frames, based on a plurality of corner features of each of the plurality of rear frames. Furthermore, the ECU is configured to compare the keyframe with each of a plurality of trailer frames associated with trailer imaging data. The trailer imaging data is received from a trailer camera associated with the trailer. Moreover, the ECU is configured to determine matched keypoints corresponding to a trailer frame of the plurality of trailer frames and the keyframe, based on the comparison. Thereafter, the ECU is configured to calibrate the trailer camera based on the matched keypoints, and camera parameters of the rear camera and the trailer camera.

[0019] In an embodiment, the present disclosure discloses a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the above-stated method steps.

[0020] In an embodiment, the present disclosure discloses a computer-readable medium having stored the above-stated computer program.

[0021] As used in this summary, in the description below and in the claims below, the term “vehicle” refers be a car, a truck, a van, a cargo vehicle, a vehicle embedded with driver assistance functions, or any other vehicle with a trailer attached to it.

[0022] As used in this summary, in the description below and in the claims below, the term “trailer” is defined as an unpowered vehicle towed by a powered vehicle. It is commonly used for the transport of goods, people or materials. The trailer may be a utility trailer, a cargo trailer, a dump trailer, and the like.

[0023] As used in this summary, in the description below and in the claims below, the term “rear camera” is defined as a camera to capture an image or a video of a scene behind the vehicle. The rear camera may be a thermal camera, an Infrared (IR) camera, and the like.202403225

[0024] 4

[0025] As used in this summary, in the description below and in the claims below, the term “trailer camera” is defined as a camera to capture an image or a video of the scene in a rearview of the vehicle and the trailer. The trailer camera may be fitted on an outer surface of the trailer. The trailer camera may be a thermal camera, an Infrared (IR) camera, and the like.

[0026] As used in this summary, in the description below and in the claims below, the term “Electronic Control Unit (ECU)” is an embedded system that controls one or more electrical systems or subsystems in the vehicle.

[0027] As used in this summary, in the description below and in the claims below, the term “at least” followed by a number is used in to denote the start of a range beginning with that number (which may be a range having an upper limit or no upper limit, depending on the variable being defined). For example, “at least one” means one or more than one.

[0028] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

[0029] BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS

[0030] The novel features and characteristics of the disclosure are set forth in the appended claims. The disclosure itself, however, as well as a preferred mode of use, further objectives, and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying figures. One or more embodiments are now described, by way of example only, with reference to the accompanying figures wherein like reference numerals represent like elements and in which:

[0031] Figure 1 illustrates an exemplary environment for calibrating a camera of a trailer attached to a vehicle, in accordance with some embodiments of the present disclosure;202403225

[0032] 5

[0033] Figure 2 illustrates a detailed diagram of an Electronic Control Unit (ECU), in accordance with some embodiments of the present disclosure;

[0034] Figure 3 shows an exemplary illustration of matching frames from a trailer camera and a rear camera, in accordance with some embodiments of the present disclosure;

[0035] Figure 4 shows an exemplary flow chart illustrating method steps for calibrating the trailer camera, in accordance with some embodiments of the present disclosure; and

[0036] Figure 5 shows a block diagram of a general-purpose computing system for calibrating the trailer camera, in accordance with embodiments of the present disclosure.

[0037] It should be appreciated by those skilled in the art that any block diagram herein represents conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0038] DESCRIPTION

[0039] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0040] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.202403225

[0041] 6

[0042] The terms “comprise”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus proceeded by “comprises... a” does not, without more constraints, preclude the existence of other elements or additional elements in the system or apparatus.

[0043] Trailers are hinged to vehicles to carry goods or materials of different nature. Video data from trailer camera provides visualization behind the trailer to drivers of the vehicles. However, there are many challenges faced in calibration of the trailer camera. Particularly, extrinsic calibration of the trailer camera is challenging.

[0044] Generally, a camera of the vehicle is calibrated using camera calibration techniques such as offline calibration and online calibration. The offline calibration requires a scene containing one or more objects whose geometry is known. The online calibration or auto-calibration has no constraints on contents of the scene.

[0045] Embodiments of the present disclosure relate to a method and an Electronic Control Unit (ECU) for online or on-the-fly extrinsic calibration of a trailer camera. In the present disclosure, imaging data from the rear camera of the vehicle as well as the trailer camera are received. A keyframe is determined among multiple rear frames from the imaging data of the rear camera. The keyframe is compared with each trailer frame from multiple trailer frames associated with the imaging data of the trailer camera. A trailer frame comprising matched keypoints with the keyframe is determined. Finally, the trailer camera is calibrated based on camera parameters of the rear camera and the matched keypoints. Accordingly, the method of the present disclosure relies on image features present in the ground plane observed by both the trailer camera and the rear camera at different time instances.

[0046] The present disclosure provides a methodology of auto-calibrating the trailer camera based on a well- calibrated rear camera of the vehicle. The rear camera which is mounted on the vehicle is usually factory calibrated and accurate within reasonable202403225

[0047] 7

[0048] limits. The present disclosure exploits the rear camera placed on the vehicle to calibrate a rear-facing trailer camera dynamically when the trailer is being towed. Hence, the present disclosure enables accurate auto-calibration of the trailer camera.

[0049] The present disclosure also finds its applications in estimating a length of the trailer and trailer reverse assist functions. Further, the present disclosure is extendable to multiple trailer cameras and finds its applications in parking functions.

[0050] Figure 1 illustrates an exemplary environment 100 for calibrating a camera of a trailer attached to a vehicle, in accordance with some embodiments of the present disclosure. The exemplary environment 100 includes a vehicle 102 and a trailer 104 attached to the vehicle 102. In an example, the vehicle 102 may include, without limiting to, a car, a truck, a van, a cargo vehicle and the like. In another example, the vehicle 102 may be any vehicle embedded with driver assistance functions. In yet another example, the vehicle 102 may be any other vehicle with a trailer attached to it. The trailer 104 is an unpowered vehicle towed by a powered vehicle. It is commonly used for the transport of goods and materials. In an example, the trailer 104 may include, without limiting to, a utility trailer, a cargo trailer, a dump trailer, and the like.

[0051] The vehicle 102 is associated with a rear camera 110. The rear camera 110 may be configured to capture an image or a video of a scene behind the vehicle 102. The scene may comprise one or more obstacles and / or a free space region. In an example, the rear camera 110 may include, without limiting to, a thermal camera, an Infrared (IR) camera, and the like. An exemplary location of the rear camera 110 is represented in Figure 1. A person skilled in the art will appreciate installation of the rear camera 110 at other locations of the vehicle 102.

[0052] The trailer 104 is associated with a trailer camera 108. The trailer camera 108 may be configured to capture an image or a video of the scene in a rearview of the vehicle 102. In an example, the trailer camera 108 may be a thermal camera, an Infrared (IR) camera, and the like. An exemplary location of the trailer camera 108 is202403225

[0053] 8

[0054] represented in Figure 1. A person skilled in the art will appreciate installation of the trailer camera 108 at other locations of the vehicle 102.

[0055] The vehicle 102 is associated with an Electronic Control Unit (ECU) 106. The ECU 106 is an embedded system that controls one or more electrical systems or subsystems in the vehicle 102. A person skilled in the art will appreciate that the vehicle 102 may comprise multiple ECUs. However, Figure 1 illustrates one ECU 106 for explanation purposes.

[0056] In the present disclosure, the ECU 106 is configured to perform online calibration of the trailer camera 108 of the vehicle 102. Herein, the ECU 106 receives rear imaging data from the rear camera 110 and trailer imaging data from the trailer camera 108. The rear imaging data comprises a plurality of rear frames. The ECU 106 determines a keyframe from the plurality of rear frames, based on a plurality of corner features of the plurality of rear frames. Further, the ECU 106 compares the keyframe with each of a plurality of trailer frames associated with the trailer imaging data. Then, the ECU 106 determines matched keypoints corresponding to a trailer frame of the plurality of trailer frames and the keyframe, based on the comparison. The ECU 106 calibrates the trailer camera 108 based on the matched keypoints, and camera parameters of the rear camera 110 and the trailer camera 108.

[0057] Figure 2 illustrates a detailed diagram 200 of the ECU 106 to calibrate the trailer camera 108 of the trailer 104 attached to the vehicle 102, in accordance with some embodiments of the present disclosure. The ECU 106 may include a Central Processing Units 202 (also referred as “CPUs” or “one or more processors 202”), Input / Output (I / O) interface 206, and a memory 204. In some embodiments, the memory 204 may be communicatively coupled to the processor 202. The memory 204 stores instructions executable by the processor 202. The processor 202 may comprise at least one data processor for executing program components for executing user or system-generated requests. The memory 204 may be communicatively coupled to the processor 202. The memory 204 stores instructions, executable by the processor 202, which, on execution, may cause the processor 202 to calibrate the trailer camera 108 of the trailer 104 attached to the vehicle 102. In an202403225

[0058] 9

[0059] embodiment, the memory 204 may include one or more modules 210 and data 208. The one or more modules 210 may be configured to perform the steps of the present disclosure using the data 208, to calibrate the trailer camera 108 of the trailer 104 attached to the vehicle 102. In an embodiment, each of the one or more modules 210 may be a stand-alone hardware unit, which may be coupled with the ECU 106. As used herein, the term modules 210 refers to an Application Specific Integrated Circuit (ASIC), an electronic circuit, a Field-Programmable Gate Arrays (FPGA), Programmable System-on-Chip (PSoC), a combinational logic circuit, and / or other suitable components that provide described functionality. The one or more modules 210 when configured with the described functionality defined in the present disclosure will result in a novel hardware. Further, the I / O interface 206 is coupled with the processor 202 through which an input signal or / and an output signal is communicated.

[0060] In one implementation, the modules 210 may include, for example, an input module 224, a keyframe determination module 226, a comparison module 228, a keypoint determination module 230, a calibration module 232, and other modules 234. It will be appreciated that such aforementioned modules 210 may be represented as a single module or a combination of different modules. In one implementation, the data 208 may include, for example, input data 212, keyframe determination data 214, comparison data 216, keypoint data 218, calibration data 220, and other data 222.

[0061] In an embodiment, the input module 224 may be configured to receive rear imaging data from the rear camera 110 associated with the vehicle 102. The rear imaging data comprises a plurality of frames of a rearview of the vehicle 102. Further, the input module 224 may be configured to receive trailer imaging data from the trailer camera 108 associated with the trailer 104. The trailer imaging data comprises a plurality of trailer frames. In an embodiment, the rear imaging data and the trailer imaging data may be received in real-time from the rear camera 110 and the trailer camera 108, respectively. The rear imaging data and the trailer imaging data is received in real-time for performing real-time calibration of the trailer camera 108. In an example, the rear imaging data and the trailer imaging data may be a video stream of a scene in the rearview of the vehicle 102. In an embodiment, the input202403225

[0062] 10

[0063] module 224 may be further configured to receive odometry information or position data of the vehicle 102 at each time instance.

[0064] In an embodiment, the input module 224 may obtain the position data, rear imaging data and the trailer imaging data, via a communication network. In an example, the communication network may be a vehicular communication network such as Controller Area Network (CAN). In an embodiment, the rear imaging data and the trailer imaging data may be received in real-time as and when the rear imaging data and the trailer imaging data is captured. In another embodiment, the rear imaging data and the trailer imaging data may be stored in a memory (for example, cache) associated with the vehicle 102 for later processing.

[0065] In an embodiment, the input module 224 may be configured to perform preprocessing of the rear imaging data and the trailer imaging data. For example, the pre-processing may be performed to remove noise from the rear imaging data and the trailer imaging data. The position data, rear imaging data, and the trailer imaging data may be stored as the input data 212 in the memory 204.

[0066] In an embodiment, the keyframe determination module 226 may be configured to receive the input data 212 from the input module 224. Further, the keyframe determination module 226 may be configured to determine a keyframe from the plurality of rear frames in the rear imaging data. The keyframe is determined as a reference frame for performing calibration of the trailer camera 108. In an embodiment, the keyframe determination module 226 determines the keyframe based on a plurality of corner features of each of the plurality of rear frames. Herein, the keyframe determination module 226 extracts the plurality of corner features in each of the plurality of rear frames. The corner features refer to interest points, which are invariant to translation, rotation, illumination, and the like. In an embodiment, the keyframe determination module 226 may extract the plurality of corner features using techniques such as Harris Corner Detection, Forstner Corner Detection, and the like. A person skilled in the art will appreciate that any technique other than the above-mentioned techniques may be used to extract the plurality of corner features.202403225

[0067] 11

[0068] In an embodiment, the keyframe determination module 226 may determine the keyframe from the plurality of rear frames with a number of the plurality of corner features exceeding a pre-defined threshold value. In an embodiment, the keyframe determination module 226 may determine the keyframe based on non-colinear point features in each of the plurality of frames. This ensures that the keyframe with features along different planes are considered for improved accuracy. The keyframe determination module 226 may determine a position of the vehicle 102 at a timestamp associated with the keyframe. The keyframe and corresponding position of the vehicle 102 may be stored as the keyframe determination data 214 in the memory 204.

[0069] In an embodiment, the comparison module 228 may receive the input data 212 and the keyframe determination data 214 from the input module 224 and the keyframe determination module 226, respectively. Further, the comparison module 228 may be configured to compare the keyframe with each of the plurality of trailer frames associated with trailer imaging data. Herein, the comparison module 228 compares keypoints from each frame of plurality of trailer frames with keypoints in the keyframe. In an embodiment, the comparison module 228 may assign a match score for each trailer frame of the plurality of frames, based on the comparison. The match score for each of the plurality of frames may be stored as the comparison data 216 in the memory 204.

[0070] In an embodiment, the keypoint determination module 230 may be configured to receive the comparison data 216 from the comparison module 228. Further, the keypoint determination module 230 may be configured to determine matched keypoints corresponding to a trailer frame of the plurality of trailer frames and the keyframe, based on the comparison. Herein, the keypoint determination module 230 may determine a trailer frame with a match score exceeding a threshold value.

[0071] Figure 3 illustrates an exemplary keyframe 300 associated with the rear camera 110. As the keyframe 300 is captured from the rear camera 110, the trailer 104 is visible in the keyframe 300. Further, a matched trailer frame 302 is illustrated.

[0072] Referring back to Figure 2, the keypoint determination module 230 may determine202403225

[0073] 12

[0074] the matched keypoints corresponding to the trailer frame. The matched keypoints may be stored as the keypoint data 218 in the memory 204.

[0075] In an embodiment, the calibration module 232 may be configured to receive the matched keypoints from the keypoint determination module 230. The calibration module 232 may be configured to calibrate the trailer camera 108 based on the matched keypoints, and camera parameters of the rear camera 110 and the trailer camera 108. The camera parameters of the rear camera 110 and the trailer camera 108 comprise at least one of extrinsic parameters and intrinsic parameters. The extrinsic parameters of the rear camera 110 may include, without limitation, rotation parameters such as pitch, yaw and roll angles, and camera position parameters along X-, Y- and Z-axis parameters. The intrinsic parameters of the rear camera 110 and the trailer camera 108 may include, without limitation, optical center, focal length, and the like. In an embodiment, the camera parameters of the rear camera 110 and the trailer camera 108 may be pre-stored in the memory 204. The extrinsic parameters of the trailer camera 108 are determined by the way of extrinsic calibration, in accordance with embodiments of the present disclosure.

[0076] In an embodiment, the vehicle 102 may be assumed to be moving in linear motion in positive or forward direction. Further, in an embodiment, a ground plane may be assumed to be flat with no undulation. In an embodiment, the calibration module 232 may project the matched keypoints corresponding to a plurality of features in the keyframe to a ground plane, based on the camera parameters of the rear camera 110. Herein, the calibration module 232 obtains the camera parameters of the rear camera 110 from the memory 204. Also, the calibration module 232 receives the matched keypoints corresponding to the plurality of features in the keyframe from the rear camera 110 which are two-dimensional (2D). The calibration module 232 projects the matched keypoints to the ground plane, based on the camera parameters or the calibration of the rear camera 110. The projection of the matched keypoints to the ground plane is a three-dimensional (3D) projection.

[0077] In an embodiment, the calibration module 232 identifies the matched keypoints corresponding to similar features in the trailer frame. Further, the calibration module202403225

[0078] 13

[0079] 232 calibrates the trailer camera 108 based on the projected matched keypoints of the keyframe and the trailer frame, and intrinsic parameters of the trailer camera 108. In an embodiment, the calibration module 232 performs the calibration based on Perspective and Point technique which uses minimum four points. A person skilled in the art will appreciate that any other techniques may be used to perform the calibration. Herein, the calibration of the trailer camera 108 is an extrinsic calibration in a past vehicle coordinate system, as the calibration is with respect to a position of the vehicle 102 at a timestamp associated with the keyframe.

[0080] In an embodiment, the calibration module 232 is configured to calibrate the trailer camera 108 by translating the camera parameters of the rear camera 110 based on position data of the vehicle 102. Herein, the translation of the camera parameters is equal to a translation of a position of the vehicle 102 between the keyframe and the trailer frame, based on the position data. The translation is performed for calibration of the trailer camera 108 in a current vehicle coordinate system., based on the position data of the vehicle 102. In an embodiment, a length of the trailer may be estimated as a difference of a translation vector between the keyframe and the trailer frame. The extrinsic parameters or calibrated parameters of the trailer camera 108 may be stored as the calibration data 220 in the memory 204.

[0081] The other data 222 may store data, including temporary data and temporary files, generated by the one or more modules 210 for performing the various functions of the ECU 106. The one or more modules 210 may also include the other modules 234 to perform various miscellaneous functionalities of the ECU 106. The other data 222 may be stored in the memory 204. It will be appreciated that the one or more modules 210 may be represented as a single module or a combination of different modules.

[0082] Figure 4 shows an exemplary flow chart illustrating method steps to calibrate the trailer camera 108, in accordance with some embodiments of the present disclosure. As illustrated in Figure 4, the method 400 may comprise one or more steps. The method 400 may be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines,202403225

[0083] 14

[0084] programs, objects, components, data structures, procedures, modules, and functions, which perform particular functions or implement particular abstract data types.

[0085] The order in which the method 400 is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method. Additionally, individual blocks may be deleted from the methods without departing from the scope of the subject matter described herein. Furthermore, the method can be implemented in any suitable hardware, software, firmware, or combination thereof.

[0086] At step 402, the ECU 106 receives rear imaging data from the rear camera 110 associated with the vehicle 102. The rear imaging data comprises a plurality of rear frames. Also, the ECU 106 receives trailer imaging data from the trailer camera 108 associated with the trailer 104. The trailer imaging data comprises a plurality of trailer frames. In an embodiment, the rear imaging data and the trailer imaging data may be received in real-time from the rear camera 110 and the trailer camera 108, respectively. The rear imaging data and the trailer imaging data is received in realtime for performing real-time calibration of the trailer camera 108.

[0087] At step 404, the ECU 106 determines a keyframe from the plurality of rear frames in the rear imaging data. The keyframe is determined as a reference frame for performing calibration of the trailer camera 108. In an embodiment, the ECU 106 determines the keyframe based on a plurality of corner features of each of the plurality of rear frames. Herein, the ECU 106 extracts the plurality of corner features in each of the plurality of rear frames. The corner features refer to interest points which are invariant to translation, rotation, illumination, and the like. In an embodiment, the ECU 106 may determine the keyframe from the plurality of rear frames with a number of the plurality of corner features exceeding a pre-defined threshold value. In an embodiment, the ECU 106 may determine the keyframe based on non-colinear point features in each of the plurality of frames. The ECU 106 may determine a position of the vehicle 102 at a timestamp associated with the keyframe.202403225

[0088] 15

[0089] At step 406, the ECU 106 compares the keyframe with each of the plurality of trailer frames associated with trailer imaging data. Herein, the ECU 106 compares keypoints from each frame of plurality of trailer frames with keypoints in the keyframe. In an embodiment, the ECU 106 may assign a match score for each trailer frame of the plurality of frames, based on the comparison.

[0090] At step 408, the ECU 106 determines matched keypoints corresponding to a trailer frame of the plurality of trailer frames and the keyframe, based on the comparison. Herein, the ECU 106 may determine a trailer frame with a match score exceeding a threshold value.

[0091] At step 410, the ECU 106 calibrates the trailer camera 108 based on the matched keypoints, and camera parameters of the rear camera 110 and the trailer camera 108. The camera parameters of the rear camera 110 and the trailer camera 108 at least one of extrinsic parameters and intrinsic parameters. The extrinsic parameters of the rear camera 110 may include, without limitation, rotation parameters such as pitch, yaw and roll angles, and camera position parameters along X-, Y- and Z-axis parameters. The intrinsic parameters of the rear camera 110 and the trailer camera 108 may include, without limitation, optical center, focal length, and the like.

[0092] In an embodiment, the ECU 106 may project the matched keypoints corresponding to a plurality of features in the keyframe to a ground plane, based on the camera parameters of the rear camera 110. Herein, the ECU 106 obtains the camera parameters of the rear camera 110 from the memory 204. Also, the ECU 106 receives the matched keypoints corresponding to the plurality of features in the keyframe from the rear camera 110 which are two-dimensional (2D). The ECU 106 projects the matched keypoints to the ground plane, based on the camera parameters or the calibration of the rear camera 110. The projection of the matched keypoints to the ground plane is a three-dimensional (3D) projection.

[0093] In an embodiment, the ECU 106 identifies the matched keypoints corresponding to similar features in the trailer frame. Further, the ECU 106 calibrates the trailer202403225

[0094] 16

[0095] camera 108 based on the projected matched keypoints of the keyframe and the trailer frame, and intrinsic parameters of the trailer camera 108.

[0096] In an embodiment, the ECU 106 is configured to calibrate the trailer camera 108 by translating the camera parameters of the rear camera 110 based on position data of the vehicle 102. Herein, the translation of the camera parameters is equal to a translation of a position of the vehicle 102 between the keyframe and the trailer frame, based on the position data. The translation is performed for calibration of the trailer camera 108 in a current vehicle coordinate system, based on the position data of the vehicle 102.

[0097] COMPUTER SYSTEM

[0098] Figure 5 illustrates a block diagram of an exemplary computer system 500 for implementing embodiments consistent with the present disclosure. In an embodiment, the computer system 500 may be used to implement the ECU 106. Thus, the computer system 500 may be used for calibrating the trailer camera 108. The computer system 500 may comprise a Central Processing Unit 502 (also referred as “CPU” or “processor”). The processor 502 may comprise at least one data processor. The processor 502 may include specialized processing units such as integrated system (bus) controllers, memory management control units, floating point units, graphics processing units, digital signal processing units, etc.

[0099] The processor 502 may be disposed in communication with one or more Input / Output (I / O) devices (not shown) via I / O interface 501. The I / O interface 501 may employ communication protocols / methods such as, without limitation, audio, analog, digital, monoaural, RCA, stereo, IEEE (Institute of Electrical and Electronics Engineers) -1394, serial bus, universal serial bus (USB), infrared, PS / 2, BNC, coaxial, component, composite, digital visual interface (DVI), high-definition multimedia interface (HDMI), Radio Frequency (RF) antennas, S-Video, VGA, IEEE 802. n / b / g / n / x, Bluetooth, cellular (e.g., code-division multiple access (CDMA), highspeed packet access (HSPA+), global system for mobile communications (GSM), long-term evolution (LTE), WiMax, or the like), etc.202403225

[0100] 17

[0101] Using the I / O interface 501 , the computer system 500 may communicate with one or more I / O devices. For example, the input device 510 may be an antenna, keyboard, mouse, joystick, (infrared) remote control, camera, card reader, fax machine, dongle, biometric reader, microphone, touch screen, touchpad, trackball, stylus, scanner, storage device, transceiver, video device / source, etc. The output device 511 may be a printer, fax machine, video display (e.g., cathode ray tube (CRT), liquid crystal display (LCD), light-emitting diode (LED), plasma, Plasma display panel (PDP), Organic light-emitting diode display (OLED) or the like), audio speaker, etc.

[0102] The processor 502 may be disposed in communication with the communication network 509 via a network interface 503. The network interface 503 may communicate with the communication network 509. The network interface 503 may employ connection protocols including, without limitation, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), transmission control protocol / internet protocol (TCP / IP), token ring, IEEE 802.11a / b / g / n / x, etc. The communication network 509 may include, without limitation, a direct interconnection, local area network (LAN), wide area network (WAN), wireless network (e.g., using Wireless Application Protocol), the Internet, etc. The network interface 503 may employ connection protocols include, but not limited to, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), transmission control protocol / internet protocol (TCP / IP), token ring, IEEE 802.11a / b / g / n / x, etc.

[0103] The communication network 509 includes, but is not limited to, a direct interconnection, an e-commerce network, a peer to peer (P2P) network, local area network (LAN), wide area network (WAN), wireless network (e.g., using Wireless Application Protocol), the Internet, Wi-Fi, and such. The first network and the second network may either be a dedicated network or a shared network, which represents an association of the different types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol / internet Protocol (TCP / IP), Wireless Application Protocol (WAP), etc., to communicate with each other. Further, the first network and the second network may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices, etc.202403225

[0104] 18

[0105] In some embodiments, the processor 502 may be disposed in communication with a memory 505 (e.g., RAM, ROM, etc. not shown in Figure 5) via a storage interface 504. The storage interface 504 may connect to memory 505 including, without limitation, memory drives, removable disc drives, etc., employing connection protocols such as serial advanced technology attachment (SATA), Integrated Drive Electronics (IDE), IEEE-1394, Universal Serial Bus (USB), fiber channel, Small Computer Systems Interface (SCSI), etc. The memory drives may further include a drum, magnetic disc drive, magneto-optical drive, optical drive, Redundant Array of Independent Discs (RAID), solid-state memory devices, solid-state drives, etc.

[0106] The memory 505 may store a collection of program or database components, including, without limitation, user interface 506, an operating system 507, web browser 508 etc. In some embodiments, computer system 500 may store user / application data, such as, the data, variables, records, etc., as described in this disclosure. Such databases may be implemented as fault-tolerant, relational, scalable, secure databases such as Oracle® or Sybase®.

[0107] The operating system 507 may facilitate resource management and operation of the computer system 500. Examples of operating systems include, without limitation, APPLE MACINTOSH® OS X, UNIX®, UNIX-like system distributions

[0108] (e.g., BERKELEY SOFTWARE DISTRIBUTION™ (BSD), FREEBSD™, NETBSD™, OPENBSD™, etc.), LINUX DISTRIBUTIONS™ (E.G., RED HAT™, UBUNTU™, KUBUNTU™, etc.), IBM™ OS / 2, MICROSOFT™ WINDOWS™ (XP™, VISTA™ / 7 / 8, 10 etc.), APPLE® IOS™, GOOGLE® ANDROID™, BLACKBERRY® OS, or the like.

[0109] In some embodiments, the computer system 500 may implement the web browser 508 stored program component. The web browser 508 may be a hypertext viewing application, for example MICROSOFT® INTERNET EXPLORER ™, GOOGLE® CHROME ™, MOZILLA® FIREFOX ™, APPLE® SAFARI™, etc. Secure web browsing may be provided using Secure Hypertext Transport Protocol (HTTPS), Secure Sockets Layer (SSL), Transport Layer Security (TLS), etc. Web browsers 508 may utilize facilities such as AJAX ™, DHTML ™, ADOBE® FLASH ™,202403225

[0110] 19

[0111] JAVASCRIPTTM, JAVATM, Application Programming Interfaces (APIs), etc. In some embodiments, the computer system 500 may implement a mail server (not shown in Figure) stored program component. The mail server may be an Internet mail server such as Microsoft Exchange, or the like. The mail server may utilize facilities such as ASP™, ACTIVEX™, ANSI™ C++ / C#, MICROSOFT®, .NET ™, CGI SCRIPTS ™, JAVA™, JAVASCRIPT™, PERL™, PHP™, PYTHON™, WEBOBJECTS™, etc. The mail server may utilize communication protocols such as Internet Message Access Protocol (IMAP), Messaging Application Programming Interface (MAPI), MICROSOFT® exchange, Post Office Protocol (POP), Simple Mail Transfer Protocol (SMTP), or the like. In some embodiments, the computer system 500 may implement a mail client stored program component. The mail client (not shown in Figure) may be a mail viewing application, such as APPLE® MAIL™, MICROSOFT® ENTOURAGE™, MICROSOFT® OUTLOOK™, MOZILLA® THUNDERBIRD™, etc.

[0112] Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to perform steps or stages consistent with the embodiments described herein. The term “computer-readable medium” should be understood to include tangible items and exclude carrier waves and transient signals, i.e., be non-transitory. Examples include Random Access Memory (RAM), Read-Only Memory (ROM), volatile memory, nonvolatile memory, hard drives, Compact Disc Read-Only Memory (CD ROMs), Digital Video Disc (DVDs), flash drives, disks, and any other known physical storage media.

[0113] The present disclosure provides a methodology of auto-calibrating the trailer camera based on a well- calibrated rear camera of the vehicle. The rear camera which is mounted on the vehicle is usually factory calibrated and accurate within reasonable limits. The present disclosure exploits the rear camera placed on the vehicle to calibrate a rear-facing trailer camera dynamically when the trailer is being towed.202403225

[0114] 20

[0115] Hence, the present disclosure enables accurate auto-calibration of the trailer camera.

[0116] The present disclosure also finds its applications in estimating a length of the trailer and trailer reverse assist functions. Further, the present disclosure is extendable to multiple trailer cameras and finds its applications in parking functions.

[0117] The terms "an embodiment", "embodiment", "embodiments", "the embodiment", "the embodiments", "one or more embodiments", "some embodiments", and "one embodiment" mean "one or more (but not all) embodiments of the invention(s)" unless expressly specified otherwise.

[0118] The terms "including", "comprising", “having” and variations thereof mean "including but not limited to", unless expressly specified otherwise.

[0119] The enumerated listing of items does not imply that any or all of the items are mutually exclusive, unless expressly specified otherwise. The terms "a", "an" and "the" mean "one or more", unless expressly specified otherwise.

[0120] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary a variety of optional components are described to illustrate the wide variety of possible embodiments of the invention.

[0121] When a single device or article is described herein, it will be readily apparent that more than one device / article (whether or not they cooperate) may be used in place of a single device / article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device / article may be used in place of the more than one device or article or a different number of devices / articles may be used instead of the shown number of devices or programs. The functionality and / or the features of a device may be alternatively embodied by one or more other devices which are not explicitly202403225

[0122] 21

[0123] described as having such functionality / features. Thus, other embodiments of the invention need not include the device itself.

[0124] The illustrated operations of Figure 4 show certain events occurring in a certain order. In alternative embodiments, certain operations may be performed in a different order, modified, or removed. Moreover, steps may be added to the above described logic and still conform to the described embodiments. Further, operations described herein may occur sequentially or certain operations may be processed in parallel. Yet further, operations may be performed by a single processing unit or by distributed processing units.

[0125] Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the invention be limited not by this detailed description, but rather by any claims that issue on an application based here on. Accordingly, the disclosure of the embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.

[0126] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.202403225

[0127] 22

[0128] Reference Signs:

[0129] Referral number Description

[0130] 100 Exemplary environment

[0131] 102 Vehicle

[0132] 104 Trailer

[0133] 106 ECU

[0134] 108 Trailer camera

[0135] 110 Rear camera

[0136] 200 Block diagram

[0137] 202 Processor

[0138] 204 Memory

[0139] 206 I / O interface

[0140] 208 Data

[0141] 210 Modules

[0142] 212 Input data

[0143] 214 Keyframe determination data 216 Comparison data

[0144] 218 Keypoint data

[0145] 220 Calibration data

[0146] 222 Other data

[0147] 224 Input module

[0148] 226 Keyframe Determination module 228 Comparison module

[0149] 230 Keypoint determination module 232 Calibration module

[0150] 234 Other modules

[0151] 500 Computer system

[0152] 501 I / O interface of the computer system 502 Processor of the computer system 503 Network interface

[0153] 504 Storage interface

[0154]

[0155] 202403225

[0156] 23

[0157] 505 Memory of the computer system 506 User interface

[0158] 507 Operating system

[0159] 508 Web browser

[0160] 509 Communication network 510 Input device

[0161] 511 Output device

[0162]

Claims

20240322524Patent claims:We claim:

1. A method of calibrating a camera of a trailer attached to a vehicle (102), the method comprising:receiving, by an Electronic Control Unit (ECU) (106) associated with a vehicle (102), rear imaging data from a rear camera (110) associated with the vehicle (102), wherein the rear imaging data comprises a plurality of rear frames;determining, by the ECU (106), a keyframe from the plurality of rear frames, based on a plurality of corner features of each of the plurality of rear frames;comparing, by the ECU (106), the keyframe with each of a plurality of trailer frames associated with trailer imaging data, wherein the trailer imaging data is received from a trailer camera (108) associated with a trailer (104) attached to the vehicle (102);determining, by the ECU (106), matched keypoints corresponding to a trailer frame of the plurality of trailer frames and the keyframe, based on the comparison; andcalibrating, by the ECU (106), the trailer camera (108) based on the matched keypoints, and camera parameters of the rear camera (110) and the trailer camera (108).

2. The method as claimed in claim 1 , wherein determining the keyframe from the plurality of rear frames comprising:extracting the plurality of corner features in each of the plurality of rear frames; anddetermining the keyframe from the plurality of rear frames with a number of the plurality of corner features exceeding a pre-defined threshold value.

3. The method as claimed in any of the preceding claims 1 or 2, wherein calibrating the trailer camera (108) comprising:20240322525projecting the matched keypoints corresponding to a plurality of features in the keyframe to a ground plane, based on the camera parameters of the rear camera (110);identifying the matched keypoints corresponding to similar features in the trailer frame; andcalibrating the trailer camera (108) based on the projected matched keypoints of the keyframe and the trailer frame, and intrinsic parameters of the trailer camera (108).

4. The method as claimed in any of the preceding claims 1-3, wherein the camera parameters of the rear camera (110) and the trailer camera (108) comprise at least one of, extrinsic parameters and intrinsic parameters.

5. The method as claimed in any of the preceding claims 1-4, wherein calibrating the trailer camera (108) further comprises translating the camera parameters of the rear camera (110) based on position data of the vehicle (102).

6. The method as claimed in any of the preceding claims 1-5, wherein the translation of the camera parameters is equal to a translation of a position of the vehicle (102) between the keyframe and the trailer frame, based on the position data.

7. The method as claimed in any of the preceding claims 1-6, wherein the rear imaging data and the trailer imaging data is received in real-time from the rear camera (110) and the trailer camera (108), respectively, for performing real-time calibration of the trailer camera (108).

8. An Electronic Control Unit (ECU) (106) for calibrating a camera of a trailer (104) attached to a vehicle (102), the ECU (106) comprises:a processor (202); anda memory (204), wherein the memory (204) stores processor-executable instructions, which, on execution, causes the processor (202) to:20240322526receive rear imaging data from a rear camera (110) associated with a vehicle (102), wherein the rear imaging data comprises a plurality of rear frames;determine a keyframe from the plurality of rear frames, based on a plurality of corner features of each of the plurality of rear frames;compare the keyframe with each of a plurality of trailer frames associated with trailer imaging data, wherein the trailer imaging data is received from a trailer camera (108) associated with a trailer (104) attached to the vehicle (102);determine matched keypoints corresponding to a trailer frame of the plurality of trailer frames and the keyframe, based on the comparison; and calibrate the trailer camera (108) based on the matched keypoints, and camera parameters of the rear camera (110) and the trailer camera (108).

9. The ECU (106) as claimed in claim 8, wherein the processor (202) is configured to determine the keyframe from the plurality of rear frames by:extracting the plurality of corner features in each of the plurality of rear frames; anddetermining the keyframe from the plurality of rear frames with a number of the plurality of corner features exceeding a pre-defined threshold value.

10. The ECU (106) as claimed in any of the preceding claims 8-9, wherein the processor (202) is configured to calibrate the trailer camera (108) by:projecting the matched keypoints corresponding to a plurality of features in the keyframe to a ground plane, based on the camera parameters of the rear camera (110);identifying the matched keypoints corresponding to similar features in the trailer frame; andcalibrating the trailer camera (108) based on the projected matched keypoints of the keyframe and the trailer frame, and intrinsic parameters of the trailer camera (108).2024032252711. The ECU (106) as claimed in any of the preceding claims 8-10, wherein the camera parameters of the rear camera (110) and the trailer camera (108) comprise at least one of, extrinsic parameters and intrinsic parameters.

12. The ECU (106) as claimed in any of the preceding claims 8-11, wherein the processor (202) is further configured to calibrate the trailer camera (108) by translating the camera parameters of the rear camera (110) based on position data of the vehicle (102), wherein the translation of the camera parameters is equal to a translation of a position of the vehicle (102) between the keyframe and the trailer frame, based on the position data.

13. The ECU (106) as claimed in any of the preceding claims 8-12, wherein the processor (202) receives the rear imaging data and the trailer imaging data in realtime from the rear camera (110) and the trailer camera (108), respectively, for performing real-time calibration of the trailer camera (108).

14. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method of any one of the claims 1 to 7.

15. A computer-readable medium having stored thereon, the computer program of claim 14.

Images