Motion correction method and device based on lissajous scanning

Abstract

An embodiment of the present invention provides an image generation device and an operating method of an image generation device. The image generation device for providing a motion artifact correction function comprises: a light generation unit for generating light emitted at a subject; a light reception unit for receiving a light signal emitted at the subject; a driving unit for controlling a light movement path; and at least one processor, wherein the processor emits light at the subject so as to generate at least one scanning image, estimates a motion speed, corrects the at least one scanning image on the basis of the motion speed so as to generate a corrected scanning image, and generates a final correction image on the basis of the at least one corrected scanning image.

Description

Lissajous scanning-based motion correction method and device

[0001] The present invention relates to a method and apparatus for correcting motion artifacts that may occur in a Lissajous scanning-based image generation device and thereby generating high-quality images.

[0002] Image generation devices are designed to acquire images of an object by irradiating it with light, and are widely used in various fields such as lidar and optical microscopes.

[0003] Korean Patent No. 10-2374439 describes a technology in which an image generation device restores an image by adjusting the phase and frequency of a signal received from an object. This technology provides the foundation for generating high-resolution images by precisely restoring the signal from the object. However, in actual Lissajous scanning-based image generation, the signal strength at a specific point varies over time due to motion caused by the user operating the image generation device; a problem arises in that if this motion is not taken into account, errors in the phase occur during restoration, potentially leading to a degradation in image quality.

[0004] To solve these problems, the present invention proposes an improved technology that includes motion artifact correction and frame-to-frame alignment processes in addition to the existing technology.

[0005] The present disclosure is conceived in response to the aforementioned background technology and aims to provide an image generation device that improves image quality by precisely correcting motion artifacts occurring in Lissajous scanning.

[0006] In addition, the present disclosure aims to provide an image generation device that improves the user interface by providing an indicator for high-quality images when using the image generation device.

[0007] According to an embodiment of the present invention, an image generation device providing a motion artifact correction function may include a light generating unit that generates light irradiated onto an object, a light receiving unit that receives a light signal irradiated onto the object, a driving unit that controls a light movement path, and at least one processor, wherein the processor irradiates light onto the object to generate at least one scanning image, estimates the motion speed of the scanning image, corrects the at least one scanning image based on the motion speed to generate a corrected scanning image, and generates a final corrected image based on the at least one corrected scanning image.

[0008] In addition, the processor can generate at least one scanning image by inputting the intensity of a signal corresponding to a scanning pattern to each pixel.

[0009] In addition, the processor can estimate the motion speed based on the deviation in signal intensity at duplicate scanned pixels among the pixels included in the scanning image.

[0010] In addition, the processor can calculate the sum of deviations in signal intensities at each of the duplicate scanned pixels among the pixels included in the scanning image, and estimate the speed that minimizes the sum as the motion speed.

[0011] In addition, the processor can generate the corrected scanning image by assigning the signal strength of the scanning coordinates containing motion to the scanning coordinates corrected based on the motion speed.

[0012] In addition, the processor can generate the corrected scanning image by correcting at least one scanning image based on the motion speed and the time at which the motion occurred.

[0013] In addition, the processor can combine a plurality of corrected scanning images to generate a final scanning image.

[0014] Additionally, the processor may align at least one scanning image adjacent to the first scanning image, calculate the sum of deviations in signal intensities at each of the duplicated scanned pixels among the pixels included in the aligned scanning image, and estimate the speed that minimizes the sum as the motion speed.

[0015] Additionally, the processor may generate each of the corrected scanning images by assigning the signal intensity of the scanning coordinates containing motion to the corrected scanning coordinates based on the motion speed for each of the matched scanning images, and may generate a final corrected image by combining at least one of the corrected scanning images.

[0016] In addition, it may include an image quality indicator that visually guides the user on the quality status of the image currently being viewed.

[0017] According to an embodiment of the present invention, a method of operation of an image generation device that provides a motion artifact correction function may include: (a) a step of generating at least one scanning image by irradiating light onto the object; (b) a step of estimating the motion speed of the scanning image; (c) a step of generating a corrected scanning image by correcting the at least one scanning image based on the motion speed; and (d) a step of generating a final corrected image based on the at least one corrected scanning image.

[0018] Additionally, the above step (a) may include the step of generating at least one scanning image by inputting the intensity of a signal corresponding to a scanning pattern to each pixel of the processor.

[0019] Additionally, the above step (b) may include a step in which the processor estimates the motion speed based on the deviation of the signal intensity in duplicate scanned pixels among the pixels included in the scanning image.

[0020] Additionally, the above step (b) may include the step of the processor calculating the sum of deviations in signal intensities at each of the duplicate scanned pixels among the pixels included in the scanning image, and the step of estimating the speed that minimizes the sum as the motion speed.

[0021] Additionally, the above step (c) may include the step of the processor generating the corrected scanning image by assigning the signal intensity of the scanning coordinates containing motion to the corrected scanning coordinates based on the motion speed.

[0022] Additionally, the above step (c) may include the step of the processor correcting at least one scanning image based on the motion speed and the time at which the motion occurred to generate the corrected scanning image.

[0023] Additionally, the above step (d) may include a step in which the processor combines a plurality of corrected scanning images to generate a final scanning image.

[0024] Additionally, (e) the processor further includes the step of aligning at least one scanning image adjacent to the first scanning image, and the step (b) may include the step of calculating the sum of deviations of signal intensities at each of the duplicated scanned pixels among the pixels included in the aligned scanning image and the step of estimating the speed that minimizes the sum as the motion speed.

[0025] Additionally, step (c) includes the step of generating each corrected scanning image by assigning the signal intensity of the scanning coordinates containing motion to the corrected scanning coordinates based on the motion speed for each of the matched scanning images, and step (d) may include the step of generating a final corrected image by combining at least one of the corrected scanning images.

[0026] In addition, it may include an image quality indicator that visually guides the user on the quality status of the image currently being viewed.

[0027] According to an embodiment of the present invention, motion artifacts occurring during the use of an image generation device can be effectively corrected.

[0028] In addition, according to an embodiment of the present invention, the quality of multi-frame fusion can be improved through alignment between adjacent frames.

[0029] In addition, according to an embodiment of the present invention, by performing fusion of multiple scanning images through alignment between adjacent frames, it is possible to expect quality improvement while minimizing the amount of computation generated during the image restoration and correction process.

[0030] FIG. 1 shows an image generating device (100) according to one embodiment of the present invention.

[0031] FIG. 2 is an example showing a scanning pattern according to one embodiment of the present invention.

[0032] FIG. 3 shows an image generating device (100) according to one embodiment of the present invention.

[0033] FIG. 4 is an example illustrating a scanning image generation process according to an embodiment of the present invention.

[0034] FIG. 5 is a flowchart illustrating a motion artifact correction process according to an embodiment of the present invention.

[0035] Figure 6 is an example illustrating a motion artifact correction process according to an embodiment of the present invention using a scanning image.

[0036] FIG. 7 is a flowchart illustrating a motion artifact correction process according to an embodiment of the present invention.

[0037] FIG. 8 is an example illustrating a motion artifact correction process according to an embodiment of the present invention using a scanning image.

[0038] FIG. 9 shows an image quality indicator according to an embodiment of the present invention.

[0039] Specific structural or functional descriptions of the embodiments disclosed herein are provided merely for the purpose of illustrating embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described herein.

[0040] Embodiments according to the concept of the present invention may be subject to various modifications and may take various forms; therefore, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all modifications, equivalents, or substitutions that fall within the spirit and scope of the present invention.

[0041] Terms such as "first" or "second" may be used to describe various components, but said components should not be limited by said terms. For the sole purpose of distinguishing one component from another, for example, without departing from the scope of rights according to the concept of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0042] The terms used herein are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to indicate the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0043] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0044] Additionally, the term “and / or” as used herein should be understood to refer to and include all possible combinations of one or more of the enumerated related items. Furthermore, the term “or” is intended to mean an implied “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X uses A or B” may apply to cases where X uses A; X uses B; or X uses both A and B. Additionally, “at least one selected from A and B” may refer to (1) A, (2) at least one of A, (3) B, (4) at least one of B, (5) at least one of A and at least one of B, (6) at least one of A and B, (7) at least one of B and A, and (8) all of A and B.

[0045] In terms used in this specification, singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as “includes” should be understood to mean that the described features, number, steps, actions, components, parts, or combinations thereof exist, and not to exclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0046] Before providing a detailed description of the drawings, it is to clarify that the classification of components in this specification is merely based on the primary function each component is responsible for. That is, two or more components described below may be combined into a single component, or a single component may be divided into two or more components based on more subdivided functions. Furthermore, each component described below may additionally perform some or all of the functions of other components in addition to its own primary function, and it goes without saying that some of the primary functions of each component may be exclusively performed by other components.

[0047] Furthermore, in performing the method or operation method, each process constituting the method may occur differently from the specified order unless a specific order is clearly indicated in the context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

[0048] Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the attached drawings.

[0049] FIG. 1 shows an image generating device (100) according to one embodiment of the present invention.

[0050] 1. Image generating device

[0051] Hereinafter, an image generating device that can be used to acquire an image of an object is described. Here, the image generating device may be an optical device in which the acquired or provided image is at least one of a reflection image, a fluorescent image, or a transmission image of the object.

[0052] FIG. 1 is a block diagram showing the configuration of an image generation device according to one embodiment.

[0053] Referring to FIG. 1, an image generating device according to one embodiment may include a processor (110), a light generating unit (120), a driving unit (130), a light receiving unit (140), and a display (160). According to one embodiment, the image generating device may include a device that generates an image using light, such as a lidar, a laser scanner, or a confocal microscope.

[0054] The processor (110) can execute software, programs, or algorithms necessary for generating and correcting images. In other words, the processor (110) can receive electrical signals and output electrical signals. For example, the processor (110) can execute software or programs that generate images based on a data acquisition method to be described later, or execute an algorithm that generates images.

[0055] However, the use of the processor (110) is not limited to the examples described above, and the processor (110) can run software, programs, or algorithms that can be executed by a normal computing device.

[0056] The light generating unit (120) can generate light of various wavelengths, including infrared, ultraviolet, and visible light. The light generated from the light generating unit (120) can be irradiated onto an object. For example, the light generated by the light generating unit (120) may be light having a wavelength of 405 nm, 488 nm, or 785 nm band to emit fluorescent dyes, but is not limited thereto, and may be light of a wavelength band to emit fluorescent substances including autofluorescent biomaterials present in the object, such as light having a wavelength band to generate autofluorescence of cells.

[0057] Additionally, the light generated by the light generating unit (120) may be unamplified light or light amplified by stimulated emission (Light Amplification by the Stimulated Emission of Radiation; hereinafter referred to as laser).

[0058] The driving unit (130) can drive the configuration on the light path so that the path of the light generated by the light generating unit (120) changes when it is irradiated onto an object. In other words, the driving unit (130) can receive electrical energy or an electrical signal from the processor (110) and drive the configuration on the light path. Here, the configuration on the light path may be a configuration including a fiber (310) that serves as a path for light movement or a MEMS mirror that reflects the light generated from the light generating unit (120).

[0059] For example, the driving unit (130) may be a driving element including an electric motor, a magnetic motor, a piezoelectric element, or a thermoelectric element. However, not limited to the examples described above, the driving unit (130) may include a element capable of generating kinetic energy when an electric force or a magnetic force is applied.

[0060] According to one embodiment, the driving unit (130) can drive the configuration on the optical path in at least one direction. That is, the driving unit (130) can receive an electrical signal and apply force to the configuration on the optical path in at least one axial direction. For example, when one axis is determined in the space where light is irradiated onto an object, the driving unit (130) can apply force in one axial direction and in an axial direction that is different from the preceding axis. In other words, the driving unit (130) can drive the configuration on the optical path in one axial direction and in an axial direction that is different from the preceding axis.

[0061] For example, the first axis may be perpendicular to the x-axis direction and the second axis may be perpendicular to the y-axis direction, but are not limited thereto.

[0062] The light receiving unit (140) can convert the light energy of the light returned from the object into electrical energy and transmit it to the processor (110). In other words, the light receiving unit (140) can acquire the light information returned from the object in the form of an electrical signal. For convenience of explanation, the fact that the light receiving unit (140) acquires the light information returned from the object in the form of an electrical signal and transmits it to the processor (110) is described below as "the processor (110) acquiring light information." However, this does not mean that the processor (110) directly acquires the light information, but rather that the light information acquired by the light receiving unit (140) is transmitted to the processor (110) as previously mentioned. Similarly, "the light receiving unit (140) acquiring light information" can also mean that the light receiving unit (140) converts the light energy into electrical energy.

[0063] In addition, the light information described below may include the intensity of a signal in a unit expressing the color of light, such as black and white RGB or CMYK, the position information of the light, and time information related to the time at which the light was acquired. However, for the convenience of explanation, the light information expressed below may refer to the intensity of the signal.

[0064] Here, the light receiving unit (140) may include an imaging element, a light receiving element, a shooting device, a light receiver, a light detector, or a light receiving device for acquiring light information. For example, the light receiving unit (140) may include a CCD, CMOS, a PMT (photoelectron amplifier tube), or a photodiode. However, the light receiving unit (140) is not limited to the above examples, and any element capable of converting light energy into electrical energy may be included in the light receiving unit (140).

[0065] The display (160) can display an image generated by the processor (110) in an identifiable form. In other words, the display (160) can receive an image generated by the processor (110) and output it to the screen so that the user can identify it.

[0066] For example, the display (160) may include an image display element including a CRT, LCD, LED, or LCoS (Liquid Crystal on Silicon). However, the display (160) is not limited to the above examples, and a device capable of receiving an electrical signal and displaying an image may be included in the display (160).

[0067] According to another embodiment, the image generating device may be provided without including a display (160). That is, with reference to FIG. 1, the image generating device is shown as including a display (160), but is not limited thereto, and an image generating device may be provided that includes only a processor (110), a light generating unit (120), a driving unit (130), and a light receiving unit (140).

[0068] FIG. 2 shows an example of a scanning pattern according to an embodiment of the present invention.

[0069] Referring to FIG. 2, FIG. 2(a) shows a spiral pattern, FIG. 2(b) shows a raster pattern, and FIG. 2(c) shows a Lissajous pattern.

[0070] According to one embodiment, when an image generating device irradiates light onto an object, it can irradiate light onto the object such that the path of the irradiated light follows a specific pattern.

[0071] Through this, the path of light can exhibit a specific pattern.

[0072] Light irradiated onto an object is irradiated along a specific path for a specific period of time, and after the specific period of time, overlap with said specific path may occur.

[0073] In other words, a specific time can refer to the time when the pattern is completed.

[0074] Referring to FIG. 2, light irradiated onto an object can be irradiated onto the object in different patterns depending on the electrical signal input to the driving unit (130). For convenience of explanation, the irradiation of light onto an object may be described as scanning the object or scanning light onto the object.

[0075] For example, referring to FIG. 2(a), when the amplitude of the electrical signal input to the driving unit (130) is changed, the path of light irradiated onto the object may exhibit a spiral pattern.

[0076] Additionally, for example, referring to FIG. 2(b), when the electrical signal input to the driving unit (130) includes a first driving signal that drives the driving unit (130) or the configuration on the light path in one axial direction and a second driving signal that drives the driving unit (130) or the configuration on the light path in a direction perpendicular to the preceding axial direction, if the frequency of the first driving signal and the frequency of the second driving signal differ by an integer multiple, the path of light irradiated onto the object may exhibit a raster pattern.

[0077] Additionally, for example, referring to FIG. 2(c), when an electrical signal input to a driving unit (130) includes a first driving signal that drives the driving unit (130) or the configuration on the light path in one axial direction and a second driving signal that drives the driving unit (130) or the configuration on the light path in a direction different from the preceding axial direction, if the frequency of the first driving signal and the frequency of the second driving signal are different from each other, the path of light irradiated onto the object may exhibit a Lissajous pattern.

[0078] In the following description, the pattern used by the image generation device is described as a Lissajous pattern; however, as described in the embodiments above, various patterns may be used as the pattern used by the image generation device.

[0079]

[0080] 1.1 General image generation device using fiber (310)

[0081] According to one embodiment, an image generating device may be provided in which the path along which light generated from a light generating unit of an image generating device travels is an optical fiber (310) (hereinafter, fiber (310)). In other words, an image generating device may be provided in which the configuration on the aforementioned light travel path is a fiber (310).

[0082] FIG. 3 is a schematic diagram showing a part of an image generating device including a driving unit (130) and a fiber (310) according to one embodiment.

[0083] Referring to FIG. 3, at least a portion of the fiber (310) can be accommodated in at least a portion of the driving unit (130). In other words, at least a portion of the fiber (310) can be coupled with the driving unit (130).

[0084] Accordingly, when the driving unit (130) receives an electrical signal from the processor (110) and drives, the fiber (310) can be driven so that the path of light irradiated over a certain area of ​​the object exhibits a specific pattern.

[0085] For a specific example, when light generated from the light generating unit (120) is irradiated onto an object, the driving unit (130) receives an electrical signal capable of generating a Lissajous pattern and can drive the fiber (310) so that the path pointing to the object becomes a Lissajous pattern.

[0086]

[0087] 1.2 Image generation using scanning patterns

[0088] The following describes a method for generating a scanning image using a scanning pattern. The scanning image obtained here may be an image of a single frame at a specific point in time. A processor may generate a final scanning image by combining at least one scanning image.

[0089] When multiple final scanning images are generated in succession, the processor (110) can acquire an image of the object.

[0090] According to an embodiment of the present invention, the processor (110) acquiring one frame may include acquiring time information, coordinate information, and light information at predetermined time intervals. Alternatively, the processor (110) acquiring one frame may include acquiring an image based on the acquired time information, coordinate information, and light information.

[0091] According to another embodiment, additionally, the processor (110) can acquire an image using coordinate information acquired for each pixel information or light information corresponding to the coordinate information after one frame has been acquired.

[0092] In other words, since the image generating device according to an embodiment of the present invention acquires a scanning image based on the scanning pattern of the driving unit, at least some of the total pixels are not scanned, and accordingly, the signal intensity value may not be assigned.

[0093] Figure 4 shows an example of acquiring a scanning image according to an embodiment of the present invention.

[0094] Referring to FIG. 4, the processor of the image generating device (100) can generate a final scanning image (400) using at least one scanning image (41, 42, 43, 44, 45).

[0095] The first scanning image (41) represents the coordinates scanned in one frame and the signal intensity mapped to the coordinates as a result of the driving unit performing scanning based on the driving signal.

[0096] The second scanning image (42) represents an image in which the coordinates scanned in the first frame and the adjacent frame (e.g., the second frame) and the signal intensity mapped to the coordinates are shown as a result of the driving unit performing scanning based on the driving signal.

[0097] Likewise, the third scanning image (43), the fourth scanning image (44), and the fifth scanning image (45) each represent images containing coordinates scanned in frames 3, 4, and 5 and signal strengths mapped thereto.

[0098] According to an embodiment of the present invention, a processor can combine at least one of the plurality of scanning images (41, 42, 43, 44, 45) to generate a final scanning image (400).

[0099] Meanwhile, in FIG. 4, five scanning images are fused to generate a final scanning image, but this is merely an example to explain the embodiment and is not limited to the number of scanning images fused.

[0100] Meanwhile, when scanning a specific area of ​​an object using the above-mentioned image generation device, the image generation device moves according to the user's operation, and when simply combining the final scanned image using at least one scanned image, motion artifacts may occur and a low-quality image may be generated.

[0101] The above motion artifact can occur within one frame as the fiber (310) is driven so that the path of light irradiated over a certain area of ​​the object exhibits a specific pattern (e.g., the pattern of FIG. 2), and can also occur in other frames adjacent to the specific frame.

[0102] The following describes a method for correcting the above motion artifacts.

[0103]

[0104] 2. Method for Correcting Motion Speed ​​Estimation Artifacts

[0105] FIG. 5 is a flowchart illustrating a motion artifact correction process according to an embodiment of the present invention.

[0106] First, according to an embodiment of the present invention, the processor of the image generation device can acquire at least one scanning image (see FIG. 4), and the scanning image may include a Lissajous scan image.

[0107] A Lissajous scan image is a pattern formed through signals having specific frequency and phase relationships, and can be generated based on optical signals repeatedly irradiated on a specific area of ​​an object.

[0108] Referring to FIG. 5, the processor of an image generation device according to an embodiment of the present invention can obtain a scanning pattern by irradiating light onto an object (S510). Through this, the intensity of a signal assigned to each pixel included in the scanning image can be obtained. At this time, the intensity of the signal may include light information or a brightness value corresponding to the coordinate information of the scanning image.

[0109] The processor can generate a scanning image by inputting the intensity of a signal corresponding to the above scanning pattern to each pixel (S520).

[0110] As previously explained, the scanning image is scanned for a predetermined time corresponding to one frame. At this time, due to the characteristics of the Lissajous pattern scanning, multiple scans are performed on the same pixel. However, the coordinates of the scanning pattern may be distorted by motion, so different signal intensities may be obtained for the same location (pixel).

[0111] According to an embodiment of the present invention, the processor can obtain the signal strength of each of the duplicate scanned pixels (S530).

[0112] According to an embodiment of the present invention, the processor can correct a scanning image based on an estimated speed value. This will be explained in detail below.

[0113] According to an embodiment of the present invention, the processor of an image generation device can perform Lissajous scanning of a specific area of ​​an object through a light-emitting unit.

[0114] Specifically, Lissajous scanning can be defined as shown in Equation 1 below.

[0115]

[0116]

[0117]

[0118] Here, x(t) and y(t) represent the coordinates of the Lissajous pattern at a specific time t.

[0119] A and B represent the amplitude (signal strength) in the x-axis and y-axis directions, respectively.

[0120] fx and fy represent frequencies on the x-axis and y-axis, and the ratio between the two frequencies determines the shape of the Lissajous pattern.

[0121] represents the phase on each axis.

[0122] When the processor of the image generation device according to an embodiment of the present invention scans a specific area of ​​an object in an ideal environment, the scanning pattern is constant as it is generated based on a combination of specific frequencies and phases; however, when movement occurs in the image generation device by a user, the scanning path is distorted due to motion artifacts as the signal strength at the intersection point fluctuates due to the user's motion, and the signal strength at the scanning intersection point may appear inconsistent.

[0123] Specifically, at least one Lissajous pattern coordinate distorted by motion at a specific time t can be defined as Equation 2 below.

[0124]

[0125]

[0126] In this case, x'(t) and y'(t) are scanning coordinates distorted by motion

[0127] , Each is the motion speed of the x-axis and y-axis.

[0128] is the phase on each axis

[0129]

[0130] Meanwhile, in the case of a scanning image where no motion occurs, the motion speeds of the x-axis and y-axis are 0, so the signal intensity assigned to any pixel of the scanning pattern is constant. Therefore, the signal intensity values ​​of duplicate scanned pixels will be similar or identical.

[0131] According to an embodiment of the present invention, the motion speed for correcting motion artifacts can be estimated using the above principle.

[0132] Generally, in the absence of motion, redundant scanning occurs in a Lissajous scan, passing over multiple pixels included in the scanned image multiple times. The optical signals of the aforementioned redundantly scanned pixels will be identical.

[0133] However, when motion occurs, deviations in the intensities of the light signals measured at each of the duplicated scanned pixels of the scanning image occur.

[0134] According to an embodiment of the present invention, the processor can calculate the motion speed based on the intensity of the light signal in the duplicated scanned pixels among the pixels included in the scanning image that scanned the object.

[0135] Specifically, the processor can calculate the motion speed by acquiring the signal strength at each of the redundantly scanned pixels and performing a calculation to minimize the sum of the deviations of the acquired signal strengths.

[0136] The following is explained in more detail based on mathematical formulas.

[0137] According to an embodiment of the present invention, the processor inputs a plurality of random motion speed values ​​and can derive a speed value in which the sum of the errors of each pixel value is minimized (S540). Through this, the motion speed occurring in the scanning image can be estimated.

[0138] Specifically, the processor can estimate the speed of motion based on the error in signal strength at the intersection point based on the coordinates of the scanning pattern.

[0139] For example, the error in the intensity of the signal may include the intensity variance or standard deviation of the signal intensity of the duplicated scanned pixels.

[0140] The above method can be expressed by the following mathematical formula 3.

[0141]

[0142]

[0143]

[0144]

[0145]

[0146]

[0147] I represents the signal strength at the corresponding coordinate.

[0148]

[0149] To explain Equation 3, if k duplicate scans occur for the same pixel, the duplicate scanning point (x k , y kSignal strength I in ) k It can be calculated in a way that minimizes the error (e.g., variance) resulting from the difference between the and the predetermined signal strength.

[0150] As explained above, in the above case, the processor can input an arbitrary motion speed randomly into the above mathematical formula 3 to estimate the motion speed.

[0151] The processor can estimate the motion speed as the speed at which the sum of the differences in signal intensity between the observed signal and the corrected signal is minimized.

[0152] Through this, the image generation device can estimate the motion speed to correct the amount of movement and ensure coordinate consistency of overlapping scanning points.

[0153] Meanwhile, according to an embodiment of the present invention, the processor can also derive the signal strength of a pixel included in each scanning image for any value to be substituted into the motion speed by comparing it with a previously predetermined signal strength.

[0154] At this time, the aforementioned previously predetermined signal strength may be determined during the calibration of the image generation device in an environment where no motion occurs.

[0155] In addition, if the processor derives a variation in motion speed during the scan time as a result of estimating the motion speed and multiple motion speeds are calculated, it can perform the above correction by deriving the average motion speed during the scan time.

[0156] Finally, the processor can correct the scanning image based on the estimated speed value (S550).

[0157] Specifically, according to an embodiment of the present invention, a processor can acquire a corrected scanning image based on an estimated motion velocity value and a scanning time.

[0158] In addition, the scanning image can be corrected by assigning the signal strength of the scanning coordinates containing motion to the corrected scanning coordinates based on the estimated motion speed.

[0159] This will allow for maintaining constant signal strength at scanning intersections and effectively eliminating distortion caused by motion artifacts.

[0160] According to the above process, the processor can effectively correct motion artifacts for each of the multiple scanned images.

[0161]

[0162] Meanwhile, according to the embodiment described in mathematical formula 3, it is also possible to correct the image generating device to have an optimal phase.

[0163] Specifically, the image generation device has the characteristic of securing consistent signal intensity values ​​by scanning overlapping points multiple times during the scanning process. However, the phase of the Lissajous pattern ( , If distortion occurs in ), the variance of signal strength at redundant scanning points increases, which can degrade image quality.

[0164] A processor according to an embodiment of the present invention can minimize the intensity variance of the signal at redundant scanning points by searching for an optimal phase value to solve the above problem.

[0165] Meanwhile, motion velocity and phase distortion can occur simultaneously in Lissajous patterns, and it would be desirable to perform optimization by combining velocity and phase rather than processing them independently.

[0166] As above, through mathematical equation 3, the phase ( , By simultaneously deriving the ) and motion speeds (vx,vy), the intensity variance of the signal at overlapping scanning points can be minimized.

[0167] Meanwhile, the above embodiment assumes that the movement occurring in the scanning image within a single frame is relatively linear or simple. Since it is easy to estimate the motion velocity and shift resulting from this assumption, it would be desirable to correct for motion artifacts occurring within a single frame.

[0168] FIG. 6 is an example of a scanning image illustrating a method for correcting motion artifacts in an actual scanning image according to an embodiment of the present invention.

[0169] Referring to FIG. 6, an example is shown in which a horizontal motion artifact occurs for each of a plurality of adjacent scanning images (61, 62, 63, 64, 65) as the image generating device moves in a horizontal motion.

[0170] In the case of the first scanning image (61), in one frame, the motion speed (v) value is 0, which is the case where no motion occurs.

[0171] In the case of the second scanning image (62), motion artifacts occurred for a certain time (t) with respect to the motion speed (v) in the second frame, and in the case of the third scanning image (63), motion artifacts additionally occurred for a certain time (t) with respect to the motion speed (v) in the third frame when compared to the second scanning image. Likewise, the fourth scanning image (64) and the fifth scanning image (65) are images corresponding to the fourth frame and the fifth frame, respectively.

[0172]

[0173] 2.1 In-frame motion correction (S550-1)

[0174] According to an embodiment of the present invention, the processor can estimate the motion speed of each of the first scanning image (61) to the fifth scanning image (65) based on Equation 3.

[0175] For example, if duplicate scanning occurs for the same pixel in the first scanning image (61), the processor can estimate a motion speed that minimizes the error resulting from the difference between the signal strength of the first scanning image at the duplicate scanning coordinates and the signal strength of a predetermined signal.

[0176] The processor can input any motion speed randomly into the above mathematical formula 3 to perform the above operation.

[0177] The processor can estimate the motion speed as the speed at which the sum of the differences in signal intensity between the observed signal and the corrected signal is minimized.

[0178] Through this, the image generation device can estimate the motion speed to correct the amount of coordinate displacement present in each scanning image and ensure coordinate consistency of overlapping scanning points.

[0179] The above process can be performed for each of the second scanning image (62) to the fifth scanning image (65).

[0180]

[0181] 2.2 Scanning Image Movement (S550-2)

[0182] Subsequently, the processor according to an embodiment of the present invention can correct the scanning image based on the estimated motion speed and the time at which the motion occurred.

[0183] Specifically, the motion can be corrected by shifting the estimated motion speed in each of the first to fifth scanning images in the opposite direction of the motion occurrence for a duration of one frame.

[0184] For example, the processor can move the second scanning image (62) horizontally by one frame by an estimated second motion speed. Similarly, the processor can move the third scanning image (63) horizontally by two frames by an estimated third motion speed. The above process can be performed in the same way for the fourth scanning image and the fifth scanning image.

[0185] Finally, the processor can move the frame scanning image based on the estimated motion speed, and accordingly, the second scanning image (64) can be corrected to generate the corrected second scanning image (67).

[0186] Likewise, a corrected third scanning image (68), a corrected fourth scanning image (69), and a corrected fifth scanning image (70) can be generated.

[0187] According to an embodiment of the present invention, a processor can combine at least one of the plurality of scanning images (66, 67, 68, 69, 70) to generate a final scanning image (600).

[0188]

[0189] Below, a method for correcting motion artifacts using image matching is explained as a simpler approach.

[0190] FIG. 7 is a flowchart illustrating a motion artifact correction method using image matching according to an embodiment of the present invention. FIG. 8 is an example of a scanning image using a motion artifact correction method using image matching according to an embodiment of the present invention.

[0191]

[0192] 2.3 Image Matching and Motion Speed ​​Estimation Artifact Correction Method

[0193] Meanwhile, according to an embodiment of the present invention, an image generation device combines a plurality of scanning images to generate a final output image in order to output a continuous image. A method for image matching and motion speed estimation motion artifact correction to simplify the process when using '2.1 Motion correction within a frame and 2.2 Scanning image movement' is described.

[0194] According to an embodiment of the present invention, the processor of an image generation device can perform adjacent frame-by-frame image matching.

[0195] Specifically, image matching algorithms include algorithms for aligning or comparing two images by finding feature points or similar regions between different images or frames.

[0196] According to an embodiment of the present invention, the processor of an image generation device can perform image matching between adjacent frames and motion artifact correction.

[0197] According to an embodiment of the present invention, the processor can acquire a plurality of scanning images and align adjacent scanning images based on a first scanning image (S710).

[0198] According to an embodiment of the present invention, the processor can detect feature points between each adjacent scanning image and calculate the relative position of the scanning image based on the matched feature points (S720).

[0199] Specifically, the relative positions of adjacent scanned images represent positional relationships derived through transformation matrices, and an algorithm such as RANSAC can be used to eliminate errors and estimate the optimal transformation matrix.

[0200] According to an embodiment of the present invention, the processor can align each adjacent scanning image with a reference scanning image by applying a calculated transformation matrix (S730). Through this, adjacent images are aligned to the same coordinate system, and discrepancies between frames can be corrected.

[0201] According to an embodiment of the present invention, the processor can correct motion artifacts by performing 'S500' of FIG. 5 for each matched scanning image (S740). Specifically, the processor analyzes the change in signal intensity at the intersection of the scanning pattern and estimates the velocity at which the variance of the signal intensity is minimized. Based on this, image distortion caused by motion can be corrected.

[0202] According to an embodiment of the present invention, the processor can fuse at least one corrected scanning image to generate a final scanning image (S750).

[0203] During the fusion process, the processor will be able to ensure signal intensity uniformity in the final image by weighting the intensity values ​​of each frame.

[0204] Referring to FIG. 8, an example is shown in which horizontal motion artifacts occur for each of the multiple scanning images (81, 82, 83, 84, 85) as the image generating device moves in a horizontal motion.

[0205] In the case of the first scanning image (81), in one frame, the motion speed (v) value is 0, which is the case where no motion occurs.

[0206] In the case of the second scanning image (82), motion artifacts occurred for a certain time (t) with respect to the motion speed (v) in the second frame, and in the case of the third scanning image (83), additional motion artifacts occurred for a certain time (t) with respect to the motion speed (v) in the third frame. Likewise, the fourth scanning image (84) and the fifth scanning image (85) are images corresponding to the fourth frame and the fifth frame, respectively.

[0207] According to an embodiment of the present invention, a processor may acquire a plurality of scanning images and align adjacent scanning images based on a first scanning image (81). For example, a second scanning image (82), a third scanning image (83), a fourth scanning image (84), and a fifth scanning image (85) may be aligned based on the first scanning image (81).

[0208] To this end, the processor can detect feature points between each scanning image and calculate the relative position of the scanning images based on the matched feature points.

[0209] Through the above process, image matching is performed on the first to fifth scanning images based on the first scanning image, so that at least some pixels can be aligned.

[0210] Afterwards, motion artifacts present in each of the first to fifth scanning images must be corrected.

[0211] According to an embodiment of the present invention, when duplicate scanning occurs for the same pixel for each scanning image, the processor can estimate a motion speed that minimizes the error resulting from the difference between the signal strength of the scanning image at the duplicate scanning coordinates and the signal strength of a predetermined signal.

[0212] The processor can input any motion speed randomly into the above mathematical formula 3 to perform the above operation.

[0213] The processor can estimate the motion speed as the speed at which the sum of the differences in signal intensity between the observed signal and the corrected signal is minimized.

[0214] Finally, the processor can correct each of the multiple scanning images based on the estimated speed value, and accordingly, each scanning image can be corrected to generate corrected scanning images (86, 87, 88, 89, 90).

[0215] According to an embodiment of the present invention, a processor can combine at least one of the plurality of scanning images (86, 87, 88, 89, 90) to generate a final scanning image (800).

[0216] Through the above embodiment, the image generating device can provide a more efficient motion artifact correction method by first performing image matching for each of the plurality of scanning images and estimating the motion speed for each scanning image to generate a final scanning image, unlike the first embodiment (see FIG. 5) in which the device estimates the motion speed within one frame for each of the plurality of scanning images and moves each scanning image based on the estimated motion speed.

[0217]

[0218] 3. Use Cases of Image Generation Devices

[0219] The Lissajous scanning-based image generation device according to the present invention can be utilized in a real-time biopsy device. For example, diagnostic accuracy can be improved by effectively removing motion artifacts during medical imaging to provide clearer images.

[0220] In addition, the present invention can be applied to industrial non-destructive inspection, high-precision 3D scanning, LiDAR sensors for autonomous driving systems, etc.

[0221]

[0222] FIG. 9 is a drawing showing a user interface according to an embodiment of the present invention.

[0223] According to an embodiment of the present invention, the image generating device may further include a user interface (90) indicating the quality of the scanned image.

[0224] Specifically, the image generation device can visually guide the user on the quality status of the image currently being viewed by providing an image quality indicator. This function analyzes and displays the image quality status in real time and can assist the user in making decisions regarding image capture.

[0225] According to an embodiment of the present invention, an image quality indicator (90) can be displayed through a small image screen at the bottom right of the final scanned image and a vertical bar.

[0226] This allows users to intuitively check the image quality status.

[0227] Referring to FIG. 9, an image quality indicator according to an embodiment of the present invention can be divided into three states.

[0228] Not displayed: If shooting is not recommended due to low image quality, the indicator is not displayed.

[0229] Displayed, Blue Bar: The image quality is at a normal level; clicking the Capture button will improve the image quality through post-processing.

[0230] Displayed, Green Bar: High image quality; clicking the Capture button allows the image to be finally saved after a brief post-processing step.

[0231] In addition, according to an embodiment of the present invention, a high-quality image indicated by a green bar can be maintained for 2 seconds unless a new high-quality image is detected.

[0232] This allows users to secure sufficient time to check and capture optimal images.

[0233] The image quality indicator function according to an embodiment of the present invention maximizes user convenience and supports securing optimal quality when capturing images.

[0234] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all of these.

[0235] The present invention described above can be implemented as computer-readable code on a medium on which a program is recorded. A machine-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of machine-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0236] In one embodiment, the recording medium may be a memory. In one embodiment, the recording medium may be implemented in a distributed form in a networked computer system, etc. Software may be stored and executed in a distributed manner in a computer system, etc. The recording medium may be a non-transitory recording medium. A non-transitory recording medium may refer to a tangible medium that exists regardless of whether data is stored semi-permanently or temporarily.

Claims

1. An image generating device that provides a motion artifact correction function, A light generating unit that generates light irradiated onto an object; A light receiving unit that receives a light signal irradiated onto the above-mentioned object; A driving unit that controls the optical movement path; and It includes at least one processor, The above processor is, Light is irradiated onto the above object to generate at least one scanning image, and Estimating the motion speed of the above scanning image, and Based on the above motion speed, at least one scanning image is corrected to generate a corrected scanning image, and Generating a final corrected image based on at least one corrected scanning image above, Image generation device.

2. In Paragraph 1, The processor inputs the intensity of a signal corresponding to a scanning pattern to each pixel to generate at least one scanning image. Image generation device.

3. In Paragraph 1, The processor estimates the motion speed based on the deviation in signal intensity at duplicate scanned pixels among the pixels included in the scanning image. Image generation device.

4. In Paragraph 1, The processor calculates the sum of deviations in signal intensities at each of the duplicate-scanned pixels among the pixels included in the scanning image, and estimates the speed that minimizes the sum as the motion speed. Image generation device.

5. In Paragraph 1, The processor generates the corrected scanning image by assigning the signal strength of the scanning coordinates containing motion to the corrected scanning coordinates based on the motion speed. Image generation device.

6. In Paragraph 1, The processor corrects at least one scanning image based on the motion speed and the time at which the motion occurred to generate the corrected scanning image. Image generation device.

7. In Paragraph 1, The above processor combines a plurality of corrected scanning images to generate a final scanning image, Image generation device.

8. In Paragraph 1, The processor aligns at least one scanning image adjacent to a first scanning image, calculates the sum of deviations in signal intensities at each of the duplicated scanned pixels among the pixels included in the aligned scanning image, and estimates the speed that minimizes the sum as the motion speed. Image generation device.

9. In Paragraph 8, The above processor is, For each of the above-mentioned aligned scanning images, the signal intensity of the scanning coordinates containing motion is assigned to the scanning coordinates corrected based on the motion speed to generate each of the above-mentioned corrected scanning images, and Generating a final corrected image by combining at least one of the above corrected scanning images, Image generation device.

10. In Paragraph 1, Including an image quality indicator that visually guides the user to the quality status of the image currently being viewed, Image generation device.

11. A method of operation of an image generating device that provides a motion artifact correction function, (a) a step of generating at least one scanning image by irradiating light onto the object above; (b) a step of estimating the motion speed of the above-mentioned scanning image; (c) a step of correcting at least one scanning image based on the motion speed to generate a corrected scanning image; and (d) a step of generating a final corrected image based on at least one corrected scanning image, Method of operation of an image generation device.

12. In Paragraph 11, The above step (a) is, A step comprising the processor inputting the intensity of a signal corresponding to a scanning pattern to each pixel to generate the at least one scanning image, Method of operation of an image generation device.

13. In Paragraph 11, The above step (b) is, A processor including the step of estimating the motion speed based on the deviation of the signal intensity at duplicate scanned pixels among the pixels included in the scanning image. Method of operation of an image generation device.

14. In Paragraph 11, The above step (b) is, A step in which the processor calculates the sum of deviations in signal intensities at each of the duplicate scanned pixels among the pixels included in the above-mentioned scanning image, and A step comprising estimating the speed that minimizes the above sum as the above motion speed, Method of operation of an image generation device.

15. In Paragraph 11, The above step (c) is, A processor comprises the step of generating a corrected scanning image by assigning the signal strength of the scanning coordinates containing motion to the corrected scanning coordinates based on the motion speed. Method of operation of an image generation device.

16. In Paragraph 11, The above step (c) is, A processor including the step of correcting at least one scanning image based on the motion speed and the time at which the motion occurred to generate the corrected scanning image. Method of operation of an image generation device.

17. In Paragraph 11, The above step (d) is, A processor comprising the step of combining a plurality of corrected scanning images to generate a final scanning image, Method of operation of an image generation device.

18. In Paragraph 11, (e) further includes the step of the processor aligning at least one scanning image adjacent to the first scanning image, and The above step (b) is a step of calculating the sum of deviations in signal intensities at each of the duplicate scanned pixels among the pixels included in the matched scanning image, and A step comprising estimating the speed that minimizes the above sum as the above motion speed, Method of operation of an image generation device.

19. In Paragraph 18, The above step (c) includes the step of generating each corrected scanning image by assigning the signal intensity of the scanning coordinates containing motion to the corrected scanning coordinates based on the motion speed for each of the matched scanning images. The above step (d) is, A method comprising the step of generating a final corrected image by combining at least one corrected scanning image. Method of operation of an image generation device.

20. In Paragraph 11, Including an image quality indicator that visually guides the user to the quality status of the image currently being viewed, Method of operation of an image generation device.

Images