Method and system for precisely synchronizing, through signal processing, time between inertial measurement unit and camera incapable of time synchronization

Abstract

Disclosed are a method and system for precisely synchronizing, through signal processing, time between an inertial measurement unit (IMU) and a camera incapable of time synchronization. The method for synchronizing time according to an embodiment may comprise the steps of: extracting a first image of a target camera and a second image of a reference camera, which correspond to each other, by analyzing images respectively captured by the target camera and the reference camera that is time-synchronized with an inertial measurement unit; and setting an initial time stamp of the target camera for time synchronization between the inertial measurement unit and the target camera by using a time stamp included in the first image and an index of the second image.

Description

Method and system for precisely synchronizing time between a camera and an inertial measurement unit, where time synchronization is impossible, through signal processing

[0001] The following description relates to a method and system for precisely synchronizing time between a camera and an Inertial Measurement Unit (IMU) through signal processing, where time synchronization is impossible.

[0002] Visual-Inertial Odometry (VIO) is a technology that fuses visual and inertial data. It is primarily used in robotics, autonomous driving, and drone navigation, and combines cameras with Inertial Measurement Units (IMUs) to enable more accurate position tracking and state estimation.

[0003] In this context, the time at which measurements from different sensors, such as cameras and IMUs, are recorded is a critical factor for the system's robustness and accuracy; therefore, time synchronization between the camera and IMU in a VIO can significantly impact sensor fusion performance. For instance, in situations where there is no relative time information between different sensors, the measurements from those sensors cannot be used directly for fusion. Particularly in the case of commercial cameras, there are limitations to performing time synchronization with existing equipment (e.g., IMUs) by applying separate signal processing methods.

[0004] A method and system for synchronizing time between a camera and an Inertial Measurement Unit (IMU), which cannot be synchronized via signal processing, are provided.

[0005] A time synchronization method for a time synchronization system implemented by at least one computer device, wherein the at least one computer device includes at least one processor, and the time synchronization method comprises: a step of analyzing images captured by each of a target camera and a reference camera time-synchronized with an Inertial Measurement Unit (IMU) by the at least one processor to extract a first image of the target camera and a second image of the reference camera that correspond to each other; and a step of setting an initial time stamp of the target camera for time synchronization between the Inertial Measurement Unit and the target camera by the at least one processor using a time stamp included in the first image and an index of the second image.

[0006] According to one aspect, the time synchronization method may further include: a step of correcting online a time offset representing a time difference between the inertial measurement unit and the target camera using an initial time stamp for the target camera and visual-inertial odometry (VIO) by the at least one processor; and a step of setting a final time stamp of an image of the target camera using the time offset corrected online by the at least one processor.

[0007] According to another aspect, the step of correcting the time offset online may be characterized by correcting the time offset by continuously updating the time offset during the process of simultaneously optimizing the reprojection error and IMU propagation error in the time-inertial odometry.

[0008] According to another aspect, the reprojection error includes an error that occurs during the process of converting a feature point into a camera frame and projecting it onto an image plane when a 3D feature point is given in a world coordinate system, and the step of correcting the time offset online may be characterized by including: a step of correcting the coordinates of the feature point through a time offset calculated through the initial time stamp; and a step of calculating the reprojection error using the corrected feature point coordinates.

[0009] According to another aspect, the IMU propagation error includes an error occurring in position and attitude estimation, wherein the state transition model of the inertial measurement unit is calculated based on a time axis having the time offset by the time offset between the target camera and the inertial measurement unit, and the step of correcting the time offset online may be characterized by including the step of calculating the IMU transition error by reflecting the time offset in the time interval of the state transition model based on the data of the inertial measurement unit.

[0010] According to another aspect, the step of setting the final time stamp of the image of the target camera may be characterized by setting the sum of the initial time stamp of the image of the target camera and the time offset corrected online as the final time stamp of the image of the target camera.

[0011] According to another aspect, the step of setting an initial time stamp of the target camera may be characterized by setting an initial time stamp for an image of the current index of the target camera based on the ratio of the difference between the current index of the target camera and the index of the second image to the frame rate of the target camera and the sum of the time stamps included in the first image.

[0012] According to another aspect, the step of extracting the first image of the target camera and the second image of the reference camera corresponding to each other may be characterized by analyzing images obtained by each of the target camera and the reference camera continuously capturing the same time display device to extract the first image of the reference camera and the second image of the target camera in which the same time of the time display device is captured.

[0013] A computer program stored on a computer-readable recording medium is provided to be combined with a computer device to execute the above method on the computer device.

[0014] A computer-readable recording medium is provided on which a computer program for executing the above method is recorded on a computer device.

[0015] A time synchronization system implemented by at least one computer device, wherein the at least one computer device comprises at least one processor, and wherein the at least one processor analyzes images captured by a target camera and a reference camera time-synchronized with an Inertial Measurement Unit (IMU), respectively, to extract a first image of the target camera and a second image of the reference camera that correspond to each other, and sets an initial time stamp of the target camera for time synchronization between the Inertial Measurement Unit and the target camera using a time stamp included in the first image and an index of the second image.

[0016] A method and system for precisely synchronizing time between a camera and an Inertial Measurement Unit (IMU), which cannot be synchronized through signal processing, can be provided.

[0017] FIG. 1 is a diagram illustrating an example of a general view of a time synchronization system in one embodiment of the present invention.

[0018] FIG. 2 is a drawing illustrating an example of the internal configuration of a time synchronization system according to an embodiment of the present invention.

[0019] FIG. 3 is a diagram illustrating an example of a time synchronization method according to an embodiment of the present invention.

[0020] FIG. 4 is a diagram illustrating an example of a process for acquiring mutually corresponding images of a target camera and a reference camera in an embodiment of the present invention.

[0021] FIG. 5 is a drawing illustrating an example of images taken at the same time in an embodiment of the present invention.

[0022] FIG. 6 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention.

[0023] Hereinafter, embodiments will be described in detail with reference to the attached drawings.

[0024] FIG. 1 is a diagram illustrating an example of a general view of a time synchronization system in an embodiment of the present invention. The time synchronization system (100) can communicate with a target camera (110), an inertial measurement unit (IMU, 120), and a reference camera (130). The target camera (110) may be a camera that cannot or is very difficult to time synchronize with the inertial measurement unit (120) through separate signal processing, such as a commercial camera. In other words, in this embodiment, it is assumed that the target camera (110) and the IMU (120) are not time-synchronized with each other, so the measurement values ​​of the target camera (110) and the IMU (120) cannot be directly used for sensor fusion. Meanwhile, the reference camera (130) may be a camera that is easy to time-synchronize with the IMU (120) using separate signal processing, and it is assumed that time synchronization has been achieved between the reference camera (130) and the IMU (120).

[0025] In this case, the time synchronization system (100) can set an initial value of relative time for the target camera (110) using measurements through coarse temporal calibration between the target camera (110) and the reference camera (120). Additionally, the time synchronization system (100) can set a final value of relative time for more precise time synchronization between the target camera (110) and the IMU (120) by performing fine temporal calibration using the initial value. Coarse temporal calibration and fine temporal calibration will be explained in more detail later.

[0026] FIG. 2 is a diagram illustrating an example of the internal configuration of a time synchronization system according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating an example of a time synchronization method according to an embodiment of the present invention. As shown in FIG. 2, the time synchronization system (100) may include an image analysis unit (210), an initial value setting unit (220), a precision time correction unit (230), and a final value setting unit (240). Substantially, the time synchronization system (100) may be implemented by at least one computer device, and the image analysis unit (210), the initial value setting unit (220), the precision time correction unit (230), and the final value setting unit (240) may be functional expressions of a processor for controlling the time synchronization system (100) so that the time synchronization system (100) performs steps (310 to 340) included in the time synchronization method of FIG. 3. For example, the time synchronization method according to the present embodiment may be performed by a time synchronization system (100) implemented by at least one computer device. In this case, at least one processor included in the at least one computer device may be implemented to execute a control instruction according to the code of an operating system included in memory or the code of at least one computer program. Here, the at least one processor may operate according to a control instruction provided by the code stored in the at least one computer device to control the time synchronization system (100) implemented by the at least one computer device so that the time synchronization system (100) performs steps (310 to 340) included in the method of FIG. 3.

[0027] In step (310), the image analysis unit (210) can analyze images captured by each of the target camera and the reference camera time-synchronized with the inertial measurement unit to extract a first image of the target camera and a second image of the reference camera that correspond to each other. For example, the image analysis unit (210) can analyze images obtained by each of the target camera and the reference camera continuously capturing the same time display device to extract a first image of the reference camera and a second image of the target camera that captured the same time of the time display device.

[0028] In step (320), the initial value setting unit (220) can set an initial time stamp of the target camera for time synchronization between the inertial measurement unit and the target camera using the time stamp included in the first image and the index of the second image. For example, the initial value setting unit (220) can set an initial time stamp for the image of the current index of the target camera based on the ratio of the difference between the current index of the target camera and the index of the second image to the frame rate of the target camera and the sum of the time stamp included in the first image. As a more specific example, assuming that the time stamp included in the first image of the reference camera is t1, the current index of the target camera is i, the index of the second image (number of images from the initial image acquisition time to the second image) is k, and the frame rate of the target camera is r, the initial time stamp t2 (i) can be expressed as in the following mathematical formula 1.

[0029]

[0030] Meanwhile, the initial time stamp is an initial value of relative time for the target camera obtained through coarse time correction, and the time synchronization system (100) can utilize the initial time stamp in subsequent steps (330) and (340) to process precise time synchronization between the target camera and the inertial measurement unit.

[0031] In step (330), the precision time correction unit (230) can correct the time offset representing the time difference between the inertial measurement unit and the target camera online using the initial time stamp for the target camera and Visual-Inertial Odometry (VIO). For example, the precision time correction unit (230) can correct the time offset by continuously updating the visual offset during the process of simultaneously optimizing the reprojection error and IMU propagation error in the Visual-Inertial Odometry (the process of minimizing the reprojection error and IMU propagation error or finding the optimal error level). Here, the reprojection error may include the error that occurs during the process of converting a feature point into a camera frame and projecting it onto an image plane when a 3D feature point is given in the world coordinate system. The precision time correction unit (230) can correct the coordinates of the feature point through the time offset calculated via the initial time stamp and calculate the reprojection error using the corrected feature point coordinates. Additionally, the IMU propagation error may include errors in position and attitude estimation caused by the time offset between the target camera and the inertial measurement unit, which causes the state transition model of the inertial measurement unit to be calculated based on a time axis having a time offset. The precision time correction unit (230) can calculate the IMU transition error by reflecting the time offset in the time interval of the state transition model based on the data of the inertial measurement unit. Subsequently, as previously described, the precision time correction unit (230) can correct the time offset by continuously updating the time offset during the process of simultaneously optimizing the reprojection error and the IMU transition error.

[0032] In step (340), the final value setting unit (240) can set the final time stamp of the target camera image using a time offset corrected online. For example, the final value setting unit (240) can set the sum of the initial time stamp of the target camera image and the time offset corrected online as the final time stamp of the target camera image. As a more specific example, at the time of the image of index i, the time offset t d (i), initial timest init (i) Let the final time stamp t final (i) can be calculated as shown in mathematical formula 2 below.

[0033]

[0034] FIG. 4 is a diagram illustrating an example of a process for acquiring mutually corresponding images of a target camera and a reference camera in an embodiment of the present invention. In the embodiment of FIG. 4, a time display device (410) that displays time is continuously captured by the target camera (110) and the reference camera (130), thereby generating a first image (420) captured at the first frame rate of the target camera (110) and a second image (430) captured at the second frame rate of the reference camera (130). Both the target camera (110) and the reference camera (130) continuously capture images, but the start times of the capture may differ from each other. Additionally, each frame (image) of the second image (430) captured by the reference camera (130) may include a time stamp of time synchronized with the IMU (120), but the time stamp for each frame of the first image (420) captured by the target camera (110) is unknown.

[0035] FIG. 5 is a diagram illustrating an example of images taken at the same time in an embodiment of the present invention. A time synchronization system (100) may receive a first image (420) from a target camera (110) and a second image (430) from a reference camera (130), and may analyze the first image (420) and the second image (430) to extract a first image (440) of the first image (420) and a second image (450) of the second image (430) that were taken at the same time. In the embodiment of FIG. 5, examples of images taken at the same time and the second image (450) of the reference camera (130) have a time stamp, whereas the time stamp of the first image (440) of the target camera (110) is unknown. Nevertheless, since the difference between the time stamp of the second image (450), the current index of the target camera (110), and the index of the first image (440) can be known, and the frame rate of the target camera (110) can also be known, the time synchronization system (100) can set an initial time stamp for the first image (440) of the target camera (110) as previously explained through Equation 1.

[0036] When an initial time stamp is set, the time synchronization system (100) can perform precise time synchronization between the target camera (110) and the IMU (120) using the initial time stamp and the VIO.

[0037] VIO is a technology that performs 6-DOF (6 Degrees of Freedom) state estimation by combining a camera (Visual) and an IMU, and can be utilized in various fields such as autonomous driving, robot navigation, augmented reality (AR), and virtual reality (VR). The time offset t between the target camera (110) and the IMU (120) d For example, it can be defined as shown in mathematical formula 3 below.

[0038]

[0039] Here, t IMUis the timestamp of the IMU (120) measurement value, t cam Each may represent a time stamp of the measurement value of the target camera (110). In other words, the time stamp of the measurement value of the IMU (120) is a time offset from the time stamp of the measurement value of the target camera (110). d It can be represented as a value with added.

[0040] Meanwhile, VIO can estimate camera movement by tracking 3D feature points. However, if the target camera (110) and the IMU (120) are not time-synchronized, the timing at which the feature points are captured will not be accurate, and the camera movement may be estimated based on incorrect feature point locations. In other words, if there is a time offset, the feature points may be projected at incorrect locations, which can increase the re-projection error.

[0041] As already explained, visual feature point P l When given in this world coordinate system, reprojection error may occur during the process of converting it into a camera frame and then projecting it onto an image plane. The reprojection error in the case of no time offset can be calculated as shown in Equation 4 below.

[0042]

[0043] Here, is feature point P l The reprojection error of, is the coordinates of a feature point (feature point l) observed in an image (frame k) of a target camera (110), can represent the attitude and position of the target camera (110), and π(·) can represent a function that projects 3D feature points onto the image plane of the target camera (110).

[0044] Mathematical Equation 4 assumes that the target camera (110) and the IMU (120) are synchronized, but in reality, a time offset may exist.

[0045] Time offsetd To calculate the corrected reprojection error including, the time offset can be corrected using the movement speed of the feature point.

[0046] The movement speed of the feature point can be calculated as shown in Equation 5 below.

[0047]

[0048] Using this, time-corrected feature point coordinates can be defined as shown in Equation 6 below.

[0049]

[0050] In other words, if a time offset exists, to the existing feature point coordinates d The feature point coordinates can be corrected by adding a displacement of that amount, and through this, the new reprojection error considering the time offset can be expressed as in the following mathematical equation 7.

[0051]

[0052] In this case, during the optimization process, d By adjusting to minimize the reprojection error, the problem of time synchronization between sensors can be solved.

[0053] Meanwhile, IMU data is sampled at regular time intervals, and an IMU transition model can be used in the process of predicting the state utilizing this data. In this case, to correct the time offset of the IMU data, the time offset t in the time interval d Using this added value, a new IMU transition model can be defined, and IMU state transitions can be performed correctly.

[0054] At this time, the precision time correction unit (230) can optimize the time offset by simultaneously minimizing the IMU transfer error and the reprojection error as shown in Equation 8 below.

[0055]

[0056] Here, is the IMU transition error, and β is the set of all IMU measurements, ε₀ represents the reprojection error reflecting the time offset, and C represents the set of features observed at least twice in the image frame, respectively. The error can be weighted by the inverse covariance P, and It may be prior knowledge and prior information obtained from marginalization.

[0057] Thus, according to embodiments of the present invention, a method and system for precisely synchronizing time between a camera and an inertial measurement unit, which cannot be time-synchronized through signal processing, can be provided.

[0058] FIG. 6 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. For example, each of at least one computer device implementing a time synchronization system (100) may correspond to the computer device (600) of FIG. 6. As illustrated in FIG. 6, the computer device (600) may include memory (610), a processor (620), a communication interface (630), and an input / output interface (640). The memory (610) is a computer-readable recording medium and may include a non-perishable mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Here, the non-perishable mass storage device such as ROM and the disk drive may be included in the computer device (600) as a separate permanent storage device distinct from the memory (610). Additionally, an operating system and at least one program code may be stored in the memory (610). These software components may be loaded into the memory (610) from a computer-readable recording medium separate from the memory (610). This separate computer-readable recording medium may include computer-readable recording media such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. In another embodiment, the software components may be loaded into the memory (610) via a communication interface (630) rather than a computer-readable recording medium. For example, the software components may be loaded into the memory (610) of the computer device (600) based on a computer program installed by files received through a network (Network, 660).

[0059] The processor (620) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (620) via memory (610) or a communication interface (630). For example, the processor (620) may be configured to execute instructions received according to program code stored in a recording device such as memory (610).

[0060] The communication interface (630) may provide a function for the computer device (600) to communicate with other devices through the network (660). For example, requests, commands, data, files, etc. generated by the processor (620) of the computer device (600) according to program code stored in a recording device such as memory (610) may be transmitted to other devices through the network (660) under the control of the communication interface (630). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (600) through the communication interface (630) of the computer device (600) via the network (660). Signals, commands, data, etc. received through the communication interface (630) may be transmitted to the processor (620) or memory (610), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (600) may further include.

[0061] The input / output interface (640) may be a means for interfacing with an input / output device (I / O device, 650). For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (640) may be a means for interfacing with a device in which the functions for input and output are integrated into one, such as a touchscreen. The input / output device (650) may be composed of a computer device (600) and a single device.

[0062] Additionally, in other embodiments, the computer device (600) may include fewer or more components than those of FIG. 6. However, it is not necessary to clearly illustrate most of the prior art components. For example, the computer device (600) may be implemented to include at least some of the input / output devices (650) described above, or may include other components such as a transceiver, a database, etc.

[0063] According to an embodiment, each of the target camera (110), IMU (120), and reference camera (130) can communicate with the computer device (600) through an input / output interface (640) as an input / output device (650) or communicate with the computer device (600) through a communication interface (630) and a network (660).

[0064] The system or device described above may be implemented as a hardware component, or a combination of a hardware component and a software component. For example, the device and component described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0065] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0066] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Furthermore, the medium may be various recording or storage means in the form of a single or multiple hardware components, and is not limited to a medium directly connected to a computer system, but may also exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0067] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0068] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. A time synchronization method of a time synchronization system implemented by at least one computer device, The above at least one computer device includes at least one processor, and The above time synchronization method is, A step of analyzing images captured by each of a target camera and a reference camera time-synchronized with an Inertial Measurement Unit (IMU) by the at least one processor to extract a first image of the target camera and a second image of the reference camera that correspond to each other; and A step of setting an initial time stamp of the target camera for time synchronization between the inertial measurement unit and the target camera using the time stamp included in the first image and the index of the second image by the at least one processor. A time synchronization method characterized by including 2. In Paragraph 1, The above time synchronization method is, A step of correcting online a time offset representing the time difference between the inertial measurement unit and the target camera using an initial time stamp for the target camera and visual-inertial odometry (VIO) by the at least one processor; and A step of setting the final time stamp of the image of the target camera using the time offset corrected online by the at least one processor. A time synchronization method characterized by further including 3. In Paragraph 2, The step of correcting the above time offset online is, A time synchronization method characterized by continuously updating the time offset to correct the time offset during the process of simultaneously optimizing the reprojection error and IMU propagation error in the above-mentioned time-inertial odometry.

4. In Paragraph 3, The above reprojection error includes the error that occurs during the process of converting a 3D feature point into a camera frame and projecting it onto an image plane when the feature point is given in the world coordinate system, and The step of correcting the above time offset online is, A step of correcting the coordinates of the feature point through a time offset calculated via the initial time stamp; and Step of calculating the reprojection error using the above-mentioned corrected feature point coordinates A time synchronization method characterized by including 5. In Paragraph 3, The above IMU propagation error includes an error occurring in position and attitude estimation, whereby the state transition model of the inertial measurement unit is calculated based on a time axis having the time offset due to the time offset between the target camera and the inertial measurement unit. The step of correcting the above time offset online is, A step of calculating the IMU transition error by reflecting the time offset in the time interval of the state transition model based on the data of the inertial measurement unit. A time synchronization method characterized by including 6. In Paragraph 2, The step of setting the final time stamp of the image of the target camera above is, A time synchronization method characterized by setting the sum of the initial time stamp of the image of the target camera and the time offset corrected online as the final time stamp of the image of the target camera.

7. In Paragraph 1, The step of setting the initial time stamp of the target camera above is, A time synchronization method characterized by setting an initial time stamp for an image at the current index of the target camera based on the ratio of the difference between the current index of the target camera and the index of the second image to the frame rate of the target camera and the sum of the time stamps included in the first image.

8. In Paragraph 1, The step of extracting the first image of the target camera and the second image of the reference camera that correspond to each other is: A time synchronization method characterized by analyzing images obtained by each of the target camera and the reference camera continuously capturing the same time display device, and extracting a first image of the reference camera and a second image of the target camera that capture the same time of the time display device.

9. A computer program stored on a computer-readable recording medium combined with a computer device to execute the method of any one of claims 1 to 8 on the computer device.

10. In a time synchronization system implemented by at least one computer device, The above at least one computer device includes at least one processor, and By the above at least one processor, By analyzing images captured by each of the target camera and the reference camera time-synchronized with the Inertial Measurement Unit (IMU), a first image of the target camera and a second image of the reference camera corresponding to each other are extracted, and Setting an initial time stamp of the target camera for time synchronization between the inertial measurement unit and the target camera using the time stamp included in the first image and the index of the second image. A time synchronization system characterized by 11. In Paragraph 10, By the above at least one processor, Using an initial time stamp and visual-inertial odometry (VIO) for the target camera, a time offset representing the time difference between the inertial measurement unit and the target camera is corrected online, and Setting the final time stamp of the image of the target camera using the time offset corrected online above. A time synchronization system characterized by 12. In Paragraph 11, To correct the above time offset online, by the at least one processor, Correcting the time offset by continuously updating the time offset during the process of simultaneously optimizing the reprojection error and IMU propagation error in the above-mentioned time-inertial odometry. A time synchronization system characterized by 13. In Paragraph 10, To set the initial time stamp of the target camera, by the at least one processor, Setting an initial time stamp for the image of the current index of the target camera based on the ratio of the difference between the current index of the target camera and the index of the second image to the frame rate of the target camera and the sum of the time stamps included in the first image. A time synchronization system characterized by 14. In Paragraph 10, In order to extract the first image of the target camera and the second image of the reference camera corresponding to each other, by the at least one processor, Analyzing images obtained by each of the target camera and the reference camera continuously capturing the same time display device to extract a first image of the reference camera and a second image of the target camera that capture the same time of the time display device. A time synchronization system characterized by

Images