2d and 3D image acquisition system and method for driving same

Abstract

The present invention relates to a 2D and 3D (2D / 3D) image acquisition system applicable to a drone, and a method for driving same, the 2D / 3D image acquisition system comprising a 2D / 3D image acquisition camera and an image control unit, wherein the 2D / 3D image acquisition camera alternately acquires a clear 2D image and a clear 3D image in a time-sharing manner by using a 2D / 3D switching optical element, a 2D / 3D switching optical element driver, and a synchronization driving unit (trigger generator), and the image control unit separates and aligns the 2D and 3D images, compresses the 2D and 3D images, and transmits same to a data analysis and storage unit.

Description

2D and 3D image acquisition system and method of operating the same

[0001] The present invention relates to a 2D and 3D (2D / 3D) image acquisition system applicable to a drone and a method of driving the same, comprising a 2D and 3D image acquisition camera and an image control unit, wherein the 2D and 3D image acquisition camera alternately acquires clear 2D images and 3D images in a time-division manner using a 2D / 3D switching optical element, a 2D / 3D switching optical element driver, and a synchronization driving unit (trigger generator), and the image control unit separates and aligns the 2D and 3D images, compresses them, and transmits them to a data analysis and storage unit.

[0002] A light field is a field used to represent the intensity and direction of light reflected from an object in three-dimensional space, and is a technology that enables the viewing of fully realistic stereoscopic images without visual distortion.

[0003] Plenoptic is a technology for shooting, image acquisition, editing, and processing that recognizes all directions and intensities of light based on a light field during shooting, accepts them as spatial information, and provides a visual experience identical to the actual physical system. When combined with artificial intelligence, it enables the realization of objects in real space as hyper-realistic images.

[0004] A plenoptic camera, also known as a light field camera, is a camera that records a light field and is a vision technology capable of simultaneously acquiring 2D images and 3D height data in a single shot. This is made possible by installing a special micro-lens array in front of an image sensor and analyzing and processing the images with special software.

[0005] Figure 1 shows the structure of a typical plenoptic camera.

[0006] Generally, the camera has a main lens (20) at the front and a photosensor, i.e., an image sensor (30), made of CMOS or CCD at the back, which images the light that enters through the main lens.

[0007] A plenoptic camera (11) has a structure in which a microlens array (MLA) (50) is attached to the front of a light sensor, that is, an image sensor (30), in addition to a standard camera, and light passing through the main lens (20) and the MLA (50) is captured by the image sensor (30). Due to the influence of imaging by each lens of the MLA (50), the image captured by the image sensor (30) has a rounded appearance, much like the eye of a fly. Generally, the image captured by this image sensor (30) is applied to software to obtain a 3D image or 3D height data, and of course, a 2D image can also be obtained. However, the resolution of the 2D image obtained in this way is significantly lower than that of the actual image.

[0008] The main lens (20) of the camera uses a lens used in a standard camera. The image sensor (30) is made of CMOS or CCD. As shown in FIG. 1, an MLA (50) is placed at the focal length of the main lens (20), and an image sensor (30) is placed at the same focal length of each lens of the MLA (50). The aperture of each lens of a typical MLA is about 100 µm, and the focal length (f) is several mm.

[0009] Figure 2 shows a USAF target, and Figure 3 shows an example of an image formed on an image sensor when the USAF target of Figure 2 is captured by the plenoptic camera of Figure 1.

[0010] FIG. 3(a) shows an image formed on an image sensor when capturing a USAF target with a plenoptic camera, and FIG. 3(b) is an enlarged view of a part of FIG. 3(a).

[0011] Generally, as shown in the image of Fig. 3, three-dimensional (3D) depth information is obtained by analyzing an image (two-dimensional image) captured by a plenoptic camera with software.

[0012] However, in order to obtain a two-dimensional (2D) image from a plenoptic camera (10), the pixel value of the two-dimensional (2D) image is determined by the average light intensity of the pixels of the image sensor (30) covering each lens of the MLA (50) among the pixels of the image sensor (30) located at the lens focal length of the MLA (50) behind each lens of the MLA (50), and the pixel of the two-dimensional (2D) image becomes the total size of the image sensor pixels covering the lens of the MLA (50). Therefore, the 2D image has the disadvantage of obtaining an image with much lower resolution compared to the actual image.

[0013] To this end, a 2D / 3D image acquisition camera is desired that can obtain clear 2D images and 3D images separately by using a 2D / 3D converter and a main lens drive unit in a single plenoptic camera.

[0014] In particular, wireless video transmission is required when using 2D / 3D image acquisition cameras for drones or surveillance.

[0015] However, when 2D and 3D images are captured alternately in time, transmitting them without compression increases the stream's packet capacity, making it highly susceptible to transmission loss and noise, which leads to image distortion. To minimize this burden on wireless transmission, image compression is necessary. However, image compression is difficult when 2D and 3D images are captured alternately in time.

[0016] To overcome these disadvantages, the present invention proposes a 2D and 3D (2D / 3D) image acquisition system applicable to a drone and a method of driving the same, comprising a 2D and 3D image acquisition camera and an image control unit, wherein the 2D and 3D image acquisition camera alternately acquires clear 2D images and 3D images in a time-division manner using a 2D / 3D switching optical element, a 2D / 3D switching optical element driver, and a synchronization driving unit (trigger generator), and the image control unit separates and aligns the 2D and 3D images, compresses them, and transmits them to a data analysis and storage unit.

[0017] As prior art, Korean published patent application No. 10-2015-0023574, "Space-Time Optical Field Camera," is equipped with a micro-lens array and is used as a 2D / 3D switchable camera.

[0018] Another prior art is Korean patent application No. 10-2023-0052486, "2D and 3D image acquisition camera."

[0019] This invention was carried out with research funding from the Civil-Military Technology Cooperation Project conducted by the Civil-Military Cooperation Promotion Agency, funded by the Government of the Republic of Korea (Ministry of Trade, Industry and Energy and Defense Acquisition Program Administration) (Agreement No. UM22203RD2)

[0020] The problem to be solved by the present invention is to provide a 2D and 3D (2D / 3D) image acquisition system applicable to a drone and a method of driving the same, comprising a 2D and 3D image acquisition camera and an image control unit, wherein the 2D and 3D image acquisition camera alternately acquires clear 2D images and 3D images in a time-division manner using a 2D / 3D switching optical element, a 2D / 3D switching optical element driver, and a synchronization driving unit (trigger generator), and the image control unit separates the acquired images into 2D images and 3D images, aligns them, compresses them, and transmits them to a data analysis and storage unit.

[0021] To solve the above technical problem, a 2D and 3D image acquisition camera system comprises: a camera unit including a main lens and an image sensor; a 2D / 3D switching optical element positioned spaced apart between the main lens and the image sensor, which enables a microlens array mode in which an MLA (microlens array) is positioned on the optical path of light incident from the main lens, or a transparent mode in which the MLA is excluded; and a 2D / 3D switching optical element drive that drives the 2D / 3D switching optical element to enable the 2D / 3D switching optical element to become a microlens array mode or a transparent mode.

[0022] The 2D / 3D switching optical element drive is driven so that the 2D / 3D switching optical element alternates between microlens array mode and transparent mode, and the image sensor acquires a 3D image when the 2D / 3D switching optical element is in microlens array mode and acquires a 2D image when the 2D / 3D switching optical element is in transparent mode.

[0023] The 2D and 3D image acquisition camera system further includes an image control unit that receives alternating streams of 3D images and 2D images from an image sensor, separates the 3D image stream and the 2D image stream respectively, compresses the separated 3D image stream and the 2D image stream respectively, and wirelessly transmits them to a data analysis and storage unit.

[0024] The data analysis and storage unit detects and stores depth information from the received 3D image.

[0025] The 2D and 3D image acquisition camera system further includes a synchronization drive unit that, upon receiving a start signal from an image control unit, generates a 2D / 3D switching optical element drive driving signal and a camera trigger signal, transmits the 2D / 3D switching optical element drive driving signal to the 2D / 3D switching optical element drive, and transmits the camera trigger signal to the camera unit.

[0026] When the image control unit receives an image from the image sensor, it transmits an image type request signal requesting the image type of the received image to the synchronization driver, and when the synchronization driver receives the image type request signal, it transmits an image type signal indicating whether the received image is a 2D image or a 3D image to the image control unit.

[0027] In addition, the driving method of the 2D and 3D image acquisition camera system of the present invention comprises: a camera trigger signal transmission step for 2D images, wherein if a start signal is transmitted from an image control unit to a synchronization driving unit, the synchronization driving unit transmits a camera trigger signal to a camera unit so that the camera unit adjusts the camera frame rate and enters a standby state for capturing the next image; a camera trigger signal transmission step for 2D images, wherein if a start signal is transmitted from an image control unit to a synchronization driving unit, the synchronization driving unit transmits a first 2D / 3D conversion optical element driving signal to a 2D / 3D conversion optical element driver, and the 2D / 3D conversion optical element driver causes the 2D / 3D conversion optical element to become transparent mode according to the first 2D / 3D conversion optical element driving signal, and the image sensor acquires a 2D image; After the 2D image acquisition step, the image control unit receives a 2D image from an image sensor, the image control unit transmits an image type request signal to the synchronization driver, the synchronization driver transmits an image type signal indicating that it is a 2D image to the image control unit, and the image control unit temporarily stores the 2D image in a 2D image storage buffer in the order received, in a 2D image reception step; after the 2D image reception step, the synchronization driver transmits a camera trigger signal to the camera unit, causing the camera unit to adjust the camera frame rate and enter a standby state for capturing the next image, in a camera trigger signal transmission step for a 3D image; After the camera trigger signal transmission step for 3D image, the synchronization driver transmits a second 2D / 3D conversion optical device driving signal to a 2D / 3D conversion optical device driver, and the 2D / 3D conversion optical device driver sets the 2D / 3D conversion optical device to a microlens mode according to the second 2D / 3D conversion optical device driving signal, so that the image sensor acquires a 3D image, in a 3D image acquisition step;and after the 3D image acquisition step, the image control unit receives a 3D image from an image sensor, the image control unit transmits an image type request signal to a synchronization driver, the synchronization driver transmits an image type signal indicating that it is a 3D image to the image control unit, and the image control unit temporarily stores the 3D image in a 3D image storage buffer in the order received; comprising a 3D image reception step.;

[0028] In addition, the driving method of the 2D and 3D image acquisition camera system of the present invention further comprises: a compression and transmission step in which, after the compression and transmission step, the image control unit determines whether the number of image frames stored in the 3D image storage buffer and the 2D image storage buffer are each equal to the size of a preset image transmission unit, and if it is determined that the number of frames stored in the 3D image storage buffer and the 2D image storage buffer are each equal to the size of a preset image transmission unit, the image control unit performs MPEG encoding on each of the 2D image temporarily stored in the 2D image storage buffer and the 3D image temporarily stored in the 3D image storage buffer to compress them, and then wirelessly transmits them to the outside through a transceiver; and after the compression and transmission step, the image control unit determines whether a stop signal has been input, and if a stop signal has not been input, returns to a camera trigger signal transmission step for the 2D image, and if a stop signal has been input, terminates.

[0029] The 2D and 3D image acquisition system and the driving method thereof according to the present invention include a 2D and 3D image acquisition camera and an image control unit, wherein the 2D and 3D image acquisition camera uses a 2D / 3D switching optical element, a 2D / 3D switching optical element driver, and a synchronization driving unit (trigger generator) to alternately acquire clear 2D images and 3D images in a time-division manner, and the image control unit separates the acquired images into 2D images and 3D images, aligns them, compresses each of the 2D images and 3D images, and transmits them to a data analysis and storage unit, thereby enabling the separate acquisition of clear 2D images and 3D images at a relatively fast speed, and thus making it applicable to various application fields.

[0030] Generally, when 2D and 3D images are captured alternately in time, a transmission of 361 Mbps is required. If transmitted without compression, the packet capacity of the stream increases, making it highly susceptible to transmission loss and noise. Therefore, compression is required, but since the images captured by the 2D / 3D image acquisition camera of the present invention are captured alternately in time, image compression is difficult. Accordingly, the present invention provides a separate image control unit, and by separating and aligning the 2D and 3D images in the image processing unit, compression becomes possible.

[0031] In particular, the 2D and 3D image acquisition system of the present invention can be applied to video cameras and surveillance cameras for drones using wireless communication.

[0032] In addition, the 2D and 3D image acquisition camera of the present invention can obtain high-resolution 3D images and acquires 2D and 3D images at predetermined time intervals, transmits them to a data analysis and storage unit (computer), continuously monitors the 2D and 3D images, and enables the detection of three-dimensional depth information using the 3D images.

[0033] Figure 1 shows the structure of a typical plenoptic camera.

[0034] Figure 2 shows the USAF target.

[0035] Figure 3 shows an example of an image formed on an image sensor when the USAF target of Figure 2 is captured by the plenoptic camera of Figure 1.

[0036] FIG. 4 is an explanatory diagram schematically illustrating the configuration of a 2D and 3D image acquisition camera system according to the present invention.

[0037] Figure 5 is a simplified diagram showing the 2D and 3D image acquisition camera system of Figure 4.

[0038] Figure 6 is an explanatory diagram illustrating the operation of the image control unit of Figure 4.

[0039] FIG. 7 is a FLIR camera specification of an example of a plenoptic camera according to the present invention.

[0040] FIG. 8 is a flowchart illustrating a schematic driving method of the 2D and 3D image acquisition camera system of the present invention.

[0041] The 2D and 3D image acquisition camera system and the driving method thereof according to the present invention will be described in detail with reference to the attached drawings.

[0042] First, the concept of the present invention is briefly explained.

[0043] As shown in FIG. 1, in a conventional plenoptic camera (11), an image sensor (30) is not placed at the point where light entering from the main lens (20) forms an image, but a microlens array (MLA) (50) is placed. Then, an image sensor (30) is placed at the focal length of each lens of the MLA (50).

[0044] If the MLA (50) is removed from this conventional plenoptic camera (11), the image sensor (30) is not positioned at the point where light from the main lens (20) forms an image, so the image formation position becomes abnormal and the image becomes blurry. In this case, to obtain a clear image, the focal length of the main lens (30) must be changed so that light from the main lens (30) forms an image on the image sensor (30).

[0045] That is, in the present invention, when a 2D image is to be obtained using a 2D / 3D switching optical element (70), the 2D image is focused on the image sensor (30) by making the MLA absent from the optical path, and when a 3D image is to be obtained, the focus of the main lens (20) is focused on the MLA (50) lens.

[0046] FIG. 4 is an explanatory diagram schematically illustrating the configuration of a 2D and 3D (2D / 3D) image acquisition camera system according to the present invention, FIG. 5 is a simplified diagram showing the 2D and 3D (2D / 3D) image acquisition camera system of FIG. 4, and FIG. 6 is an explanatory diagram illustrating the operation of the image control unit of FIG. 4.

[0047] The 2D and 3D image acquisition camera system (7) of the present invention generally includes a main lens (20), a 2D / 3D switching optical element (70), a 2D / 3D switching optical element driver (120), an image sensor (30), a synchronization driving unit (100), and an image control unit (200).

[0048] The main lens (20) is a PZT (PbZr) that can change the focal length. x Ti 1-xIt is composed of a lens equipped with O3 (solid solution of lead titanium dioxide and lead zirconate), and, for example, may be an autofocus lens. The main lens is a commercially available camera lens. When obtaining a 2D image, the position of the main lens is moved so that the 2D image is focused on the image sensor by making the MLA absent in the 2D / 3D conversion optical element (70). To obtain a 3D image, the focus of the main lens (20) is set so that it is focused on the MLA (50) lens.

[0049] The main lens drive unit (110) adjusts the focus by moving the position of the main lens. The main lens drive unit (110) may be an automatic focus adjustment device. In some cases, the main lens drive unit (110) may be configured to perform automatic focus adjustment according to a signal provided by the synchronization drive unit (100).

[0050] The 2D / 3D switching optical element (70) is an optical element capable of selectively operating a Micro Lens Array (MLA) (50) in a micro lens mode or a transparent mode at a video frame rate via an electrical method. That is, the 2D / 3D switching optical element (70) is equipped with an MLA (50) and is a means of switching the MLA (50) to be positioned on the optical path of the main lens (20) (i.e., Micro Lens Mode (3D Mode)) or not positioned on the MLA (50) (i.e., Transparent Mode (2D Mode)) by means of a 2D / 3D switching optical element driver (120). Micro-lens mode is a mode in which an MLA (50) lens is mounted on a 2D / 3D switching optical element (70) so that light entering from the main lens (20) passes through the MLA (50) lenses to obtain a 3D image, and transparent mode is a mode in which the MLA (50) is not in the optical path where the main lens (20) forms an image so that a 2D image is obtained.

[0051] The 2D / 3D switching optical element driver (120) drives the 2D / 3D switching optical element (70) according to a signal received from the synchronization driving unit (100), so that the 2D / 3D switching optical element (70) causes the MLA (50) to be placed or removed from the optical path where the main lens (20) forms an image.

[0052] The image sensor (30) is a means for forming an image, and acquires a 3D image or a 2D image depending on whether or not there is an MLA (50) in the optical path where the main lens (20) forms an image.

[0053] Here, the main lens (20), the main lens driving unit (110), and the image sensor (30) have the same configuration as the plenoptic camera (11), and for convenience of explanation, they are referred to as the camera unit (10). Additionally, the 2D / 3D switching optical element driver (120), the 2D / 3D switching optical element (70), and the MLA (50) can be referred to as the 2D / 3D switching optical element unit.

[0054] The camera unit (10) can use a commercially available camera, and in particular, can use a FLIR (forward looking infrared) camera.

[0055] In the present invention, the camera unit (10) uses a camera having a pixel size of 4.5 µm or more. For example, in the present invention, among commercially available cameras with an image sensor, a FLIR camera (model name: BFS-u3-19s4c-c) with a sensor pixel size of 4.5 µm can be used.

[0056] The synchronization driving unit (100) includes a trigger generator (105), and when it receives a start signal from the image control unit, it generates a trigger signal and transmits it to the camera unit (10), and also generates a 2D / 3D switching optical element driving signal to enable operation in a transparent mode or a microlens array mode so as to alternately acquire 2D images and 3D images, and transmits it to the 2D / 3D switching optical element driver (120).

[0057] The trigger generator (105) is a means for generating a trigger pulse signal at predetermined time intervals and transmitting it to the camera unit.

[0058] Additionally, the synchronization driving unit (100) transmits a 2D / 3D switching optical device driving signal, which is a pulse signal, to a 2D / 3D switching optical device driver (120) at predetermined time intervals, thereby causing the 2D / 3D switching optical device driver (120) to be in a state where the MLA (50) is present or where the MLA (50) is absent, so that the camera unit acquires a 3D image (or a 2D image).

[0059] For example, as a trigger pulse signal is transmitted from the synchronization driver (100) to the camera unit (i.e., plenoptic camera) (10), the camera unit (10) becomes a driving state capable of capturing an image, and at this time, according to the 2D / 3D switching optical device driving signal transmitted from the synchronization driver (100) to the 2D / 3D switching optical device driver (120), the 2D / 3D switching optical device driver (120) drives the MLA (50) into a micro-lens mode (3D mode) or a transparent mode (i.e., 2D mode).

[0060] The trigger generator (105) of the synchronization drive unit (100) generates a camera trigger signal and transmits it to the camera unit (10). When the camera unit (10) receives the camera trigger signal, the camera unit (10) is in a state where it can acquire the next image (2D or 3D image), that is, it adjusts the camera frame rate for acquiring the next image (2D or 3D image) and remains in a standby state. At this time, the main lens drive unit of the camera unit (10) can perform autofocusing, etc.

[0061] In the synchronization drive unit (100), a camera trigger signal is generated in accordance with the selected video frame. For example, in the case of a video with 60 frames per second, the trigger generator generates a trigger signal 60 times per second. Additionally, when the synchronization drive unit (100) receives a video type request signal (i.e., a request to check whether it is a 2D video or a 3D video) from the video control unit (200), it transmits a signal (value) indicating the video type of the currently transmitted signal (i.e., whether it is a 2D video or a 3D video) to the video control unit (200).

[0062] That is, when the synchronization driving unit (100) receives a digital signal, a start signal, from the image control unit (200), it generates a camera trigger signal and transmits it to the camera unit (10), and also generates a 2D / 3D switching optical element driving signal and transmits it to the 2D / 3D switching optical element driver (120).

[0063] The image sensor (30) of the 2D and 3D image acquisition camera (7) periodically acquires 2D and 3D images alternately. The 2D or 3D images acquired from the image sensor (30) are transmitted to the image control unit (200).

[0064] The image control unit (200) is a hardware module that receives image stream data in which 2D images and 3D images alternate from a 2D and 3D image acquisition camera (7), separates the data into 2D images, i.e., 2D image stream data, and 3D images, i.e., 3D image stream data, respectively, compresses each of the 2D image stream data and 3D image stream data, and transmits them to a data analysis and storage unit (300).

[0065] In particular, the image control unit (200) can be a hardware module mounted on a drone to process images, which serves as a means of acting as a control unit, such as a mini PC or Jetson nano.

[0066] The image control unit (200), that is, the mini PC, acquires 2D / 3D images obtained from the camera, collects only the 2D images to form a 2D stream, collects only the 3D images to form a 3D stream, transmits them to the transmission / reception unit (260), and transmits them externally through the transmission / reception unit (260). In addition, the image control unit (200) stores the 2D / 3D images in the memory unit (250).

[0067] The data analysis and storage unit (300) is a means for analyzing or storing image data received from a 2D and 3D image acquisition camera (7) through an image control unit (200), and may be a computer or a server.

[0068] The data analysis and storage unit can find, record, or output three-dimensional depth information from a 3D image using built-in software. The recorded 3D image can be displayed or analyzed by the data analysis and storage unit.

[0069] As shown in FIG. 5, the 2D and 3D image acquisition camera system of the present invention comprises a conventional camera unit (10), a 2D / 3D conversion optical element (70) (i.e., MLA (50)), a 2D / 3D conversion optical element driver (120), an image control unit (200), and a synchronization driving unit (100).

[0070] The video control unit (200) generates a start signal and transmits it to the synchronization drive unit (100).

[0071] When the synchronization driving unit (100) receives a digital signal, a start signal, from the image control unit (200), it generates a camera trigger signal and a 2D / 3D switching optical element driving signal, and the trigger signal is transmitted to the camera unit.

[0072] The synchronization driving unit (100) transmits a 2D / 3D switching optical element driving signal to a 2D / 3D switching optical element driver (120), and according to the 2D / 3D switching optical element driving signal, the 2D / 3D switching optical element driver (120) drives the 2D / 3D switching optical element (70) to create a state where the MLA (50) is absent or present in the optical path, and as a result, the image sensor (30) acquires a 2D image or a 3D image. That is, according to the 2D / 3D switching optical element driving signal, the 2D / 3D switching optical element driver (120) alternately creates a state where the MLA (50) is absent or present in the optical path, and as a result, the image sensor (30) alternately acquires a 2D image and a 3D image, and the image stream data of the generated 2D image and 3D image alternately is transmitted to the image control unit (200).

[0073] The synchronization drive unit (100) transmits a camera trigger signal to the camera unit (10), and the camera unit (10) remains in a standby state after adjusting the camera frame rate to acquire the next image (2D or 3D image). Then, the image control unit (200) transmits a start signal to the synchronization drive unit (100) again, and the subsequent process is repeated.

[0074] For example, when a first 2D / 3D conversion optical device driving signal (first 2D / 3D conversion optical device driving signal) is transmitted from the synchronization driving unit (100) to the 2D / 3D conversion optical device driver, the 2D / 3D conversion optical device driver (120) sets the 2D / 3D conversion optical device (70) to a micro-lens mode to capture a 3D image, and the image sensor (30) of the camera unit (10) obtains a 3D image, which is then transmitted to the image control unit (200).

[0075] After capturing a 3D image in the camera unit (10) in this way, a camera trigger signal is transmitted from the synchronization drive unit (100) to the camera unit (10), and the camera unit (10) adjusts the camera frame rate and enters a standby state for capturing the next image. When a second 2D / 3D conversion optical element driving signal (second 2D / 3D conversion optical element driving signal) is transmitted from the synchronization drive unit (100) to the 2D / 3D conversion optical element driver, the 2D / 3D conversion optical element driver (120) sets the 2D / 3D conversion optical element (70) to transparent mode to capture a 2D image, and the image sensor (30) of the camera unit (10) obtains a 2D image, which is then transmitted to the image control unit (200).

[0076] After capturing a 3D image in the camera unit (10) in this way, a camera trigger signal is transmitted from the synchronization driving unit (100) to the camera unit (10), and the camera unit (10) adjusts the camera frame rate and becomes ready for capturing the next image. Then, a first 2D / 3D conversion optical device driving signal (first 2D / 3D conversion optical device driving signal) is transmitted from the synchronization driving unit (100) to the 2D / 3D conversion optical device driver, and the 2D / 3D conversion optical device driver (120) acquires a 3D image by setting the 2D / 3D conversion optical device (70) to micro-lens mode.

[0077] This process is repeated until a stop signal is input from the key input unit (not shown) of the 2D and 3D image acquisition camera system and transmitted to the image control unit (200), that is, until a stop signal is input.

[0078] Additionally, whenever image data is received from the image sensor (30), the image control unit (200) transmits a signal requesting the image type of the received image data (i.e., a request to check whether it is a 2D image or a 3D image) to the synchronization driving unit (100), and accordingly, the synchronization driving unit (100) transmits a signal (value) indicating the image type (i.e., whether it is a 2D image or a 3D image) to the image control unit (200).

[0079] The image control unit (200) collects only the captured 2D images sequentially in time, compresses them into a stream, compresses them, and wirelessly transmits them to the outside, and collects only the captured 3D images sequentially in time, compresses them into a stream, compresses them, and wirelessly transmits them to the outside.

[0080] Meanwhile, the compressed 3D image from the image control unit (200) is transmitted to a data analysis and storage unit (e.g., a main server computer), and three-dimensional depth information can be found and recorded using software provided in the data analysis and storage unit. The recorded 3D image can be displayed or analyzed in the data analysis and storage unit.

[0081] When transmitting an image signal from the image sensor (30) of the camera unit (10) to the image control unit (200), it can be transmitted using Ethernet or USB communication, and the signal between the image control unit (200) and the synchronization driving unit (100) can be transmitted using USB communication. In addition, the camera trigger signal may be a pulse signal of TTL logic, and the 2D / 3D switching optical element driving signal may also be a TTL logic signal.

[0082] As shown in FIG. 6, the processing unit (210) of the image control unit (200) receives an image from the image sensor (30), that is, an image stream in which 2D images and 3D images alternate in chronological order, and sorts the 2D images (2D image stream) and 3D images (3D image stream) in chronological order, and then performs MPEG (Moving Picture Experts Group) encoding on the sorted 2D images (2D image stream) and 3D images (3D image stream) respectively to compress them, stores them in the memory unit (250), and at the same time transmits them to the outside, that is, to the data analysis and storage unit (300), through the transmission and reception unit (260).

[0083] Image data can be transmitted from the image sensor (30) of the camera unit (10) to the image control unit (200) via USB 3.1, and image data can be transmitted from the image control unit (200) to the data analysis and storage unit (300) via a Real Time Streaming Protocol (RTSP). Additionally, the mini PC of the image control unit (200) may have a built-in SSD (Solid-state drive) as a memory unit (250).

[0084] As shown in FIG. 6, the images captured by the 2D and 3D image acquisition camera system (7) alternately capture 2D images and 3D images. When these images are compressed, there is no similarity between neighboring images, so the compression rate decreases, resulting in the burden of transmitting 60 frames per second of images with almost no compression. As the transmission capacity increases, loss and distortion of image data are expected. To minimize this transmission burden, the present invention maximizes the compression rate by separately collecting and compressing the alternately incoming images.

[0085] FIG. 7 is a FLIR camera specification of an example of an image camera according to the present invention.

[0086] The pixel size of the video camera is 4.5 µm, and the resolution is 1616 x 1240. When capturing video at 60 frames per second, the video data captured per second is 1616 x 1240 x 2 x 60 = 360,691,200 bits. In other words, transmission at a speed of 361 Mbps is required. If transmitted without compression, the packet capacity of the stream increases, making it highly susceptible to transmission loss and noise. Therefore, compression is necessary; however, video compression is difficult because the images captured by the 2D / 3D video acquisition camera of the present invention alternate between 2D and 3D images in time. Accordingly, in the present invention, compression is made possible by separating and aligning the 2D and 3D images in the 2D / 3D image processing unit. While a transmission speed of 361 Mbps is required without compression, H.264 compression typically yields a compression ratio of 1 / 100, reducing the speed to approximately 4 Mbps and thereby reducing the transmission burden.

[0087] FIG. 8 is a flowchart illustrating a schematic driving method of the 2D and 3D image acquisition camera system of the present invention.

[0088] In the initialization step, the image control unit (200), camera unit (10), etc. are initialized according to setting information such as a pre-set frame rate and image size (S105, S106). Here, for example, the 2D / 3D switching optical element is set to microlens mode during initialization.

[0089] In the step of transmitting a camera trigger signal for receiving 2D video, when a start signal is transmitted from the computation processing unit (210) of the video control unit (200) to the synchronization driving unit (100) (S110), or after the step of determining whether to stop (S270), the synchronization driving unit (100) transmits the camera trigger signal to the camera unit (10) (S160), and the camera unit (10) adjusts the camera frame rate and becomes a standby state for shooting the next video.

[0090] In the 2D image acquisition step, after the camera trigger signal transmission step for receiving the 2D image, the synchronization driving unit (100) transmits a first 2D / 3D switching optical element driving signal to the 2D / 3D switching optical element driver (S120), the 2D / 3D switching optical element driver (120) makes the 2D / 3D switching optical element (70) transparent mode, and the image sensor (30) of the camera unit (10) acquires a 2D image.

[0091] In the 2D image reception step, after the 2D image acquisition step, the computation processing unit (210) of the image control unit (200) receives a 2D image from the image sensor (30) of the camera unit (10) (S130), and the computation processing unit (210) of the image control unit (200) transmits a signal requesting the type of the received image (i.e., a request to check whether it is a 2D image or a 3D image) to the synchronization driving unit (100) (S140), and accordingly, the synchronization driving unit (100) transmits a signal (value) indicating the type of image (i.e., 2D image) to the image control unit (200) (S150), and the computation processing unit (210) temporarily stores the 2D images in the order they were received in the 2D image storage buffer (not shown) of the image control unit (200).

[0092] In the step of transmitting a camera trigger signal for receiving 3D video, after the step of receiving 2D video, the synchronization driving unit (100) transmits the camera trigger signal to the camera unit (10) (S160), and the camera unit (10) adjusts the camera frame rate and becomes a standby state for shooting the next video.

[0093] In the 3D image acquisition step, after the camera trigger signal transmission step for receiving 3D images, when the synchronization driving unit (100) transmits a second 2D / 3D conversion optical element driving signal to the 2D / 3D conversion optical element driver (S180), the 2D / 3D conversion optical element driver (120) makes the 2D / 3D conversion optical element (70) into a micro-lens mode, and the image sensor (30) of the camera unit (10) acquires a 3D image.

[0094] In the 3D image reception step, after the 3D image acquisition step, the computation processing unit (210) of the image control unit (200) receives a 3D image from the image sensor (30) of the camera unit (10) (S190), and the image control unit (200) transmits a signal requesting the type of the received image (i.e., a request to check whether it is a 2D image or a 3D image) to the synchronization driving unit (100) (S200), and accordingly, the synchronization driving unit (100) transmits a signal (value) indicating the type of image (i.e., 3D image) to the image control unit (200) (S210), and the computation processing unit (210) of the image control unit (200) temporarily stores the 3D image in a 3D image storage buffer (not shown) in the order in which it was received.

[0095] In the step of transmitting a camera trigger signal after receiving a 3D image, after receiving a 2D image, the synchronization driving unit (100) transmits a camera trigger signal to the camera unit (10) (S220), and the camera unit (10) adjusts the camera frame rate and becomes ready for shooting the next image.

[0096] In the step of determining whether the number of video frames stored in the 3D video storage buffer (not shown) and the 2D video storage buffer (not shown) is equal to the preset number of video transmission units (S230), if they are not equal, proceed to the step of determining whether to stop (S270).

[0097] In the compression and transmission step, if it is determined in the step of determining whether the number of frames stored in the 3D image storage buffer (not shown) and the 2D image storage buffer (not shown) is equal to the preset image transmission unit size, the image control unit (200) performs MPEG encoding on each of the 2D image temporarily stored in the 2D image storage buffer (not shown) and the 3D image temporarily stored in the 3D image storage buffer (not shown) to compress them (S250), then wirelessly transmits them to the outside through the transmission and reception unit (S260), and clears the 2D image and 3D image storage buffers.

[0098] In the stop determination step, the image control unit (200) determines whether a stop signal has been input (S270), and if a stop signal has not been input, it returns to the camera trigger signal transmission step (S110) for receiving 2D images, and if a stop signal has been input, it terminates.

[0099] The present invention has been illustrated and described with respect to specific preferred embodiments. However, the present invention is not limited to the embodiments described above, and those skilled in the art may make various modifications without departing from the gist of the technical concept of the present invention as described in the following claims.

[0100] The 2D and 3D image acquisition system and the driving method thereof according to the present invention can obtain clear 2D and 3D images separately at a relatively fast speed, and can be applied to various application fields where it is necessary to detect three-dimensional depth information using 3D images while continuously monitoring with 2D and 3D images. In particular, it can be applied to video cameras for drones and surveillance cameras using wireless communication.

Claims

1. A camera unit including a main lens and an image sensor; A 2D / 3D switching optical element positioned spaced apart between a main lens and an image sensor, which enables a microlens array mode in which an MLA (microlens array) is positioned on the optical path of light incident from the main lens, or a transparent mode in which the MLA is excluded; A 2D / 3D switching optical element drive that drives the 2D / 3D switching optical element to become a microlens array mode or a transparent mode; A 2D and 3D image acquisition camera system characterized by including 2. In Paragraph 1, The 2D / 3D switching optical device drive drives the 2D / 3D switching optical device to alternate between microlens array mode and transparent mode. A 2D and 3D image acquisition camera system characterized in that the image sensor acquires a 3D image when the 2D / 3D switching optical element is in microlens array mode and acquires a 2D image when the 2D / 3D switching optical element is in transparent mode.

3. In Paragraph 2, A 2D and 3D image acquisition camera system characterized by further including an image control unit that receives alternating streams of 3D images and 2D images from an image sensor, separates the 3D image stream and the 2D image stream respectively, compresses the separated 3D image stream and the 2D image stream respectively, and wirelessly transmits them to a data analysis and storage unit.

4. In Paragraph 3, A 2D and 3D image acquisition camera system characterized by a data analysis and storage unit detecting and storing depth information from a received 3D image.

5. In Paragraph 3, A 2D and 3D image acquisition camera system characterized by further comprising a synchronization drive unit that, upon receiving a start signal from an image control unit, generates a 2D / 3D switching optical element drive drive signal and a camera trigger signal, transmits the 2D / 3D switching optical element drive drive signal to the 2D / 3D switching optical element drive, and transmits the camera trigger signal to the camera unit.

6. In Paragraph 5, A 2D and 3D image acquisition camera system characterized in that, when an image is received from an image sensor, the image control unit transmits an image type request signal requesting the image type of the received image to a synchronization driver, and when the image type request signal is received, the synchronization driver transmits an image type signal indicating whether the received image is a 2D image or a 3D image to the image control unit.

7. A camera trigger signal transmission step for a 2D image, wherein if a start signal is transmitted from the image control unit to the synchronization driving unit, the synchronization driving unit transmits a camera trigger signal to the camera unit, causing the camera unit to adjust the camera frame rate and enter a standby state for capturing the next image; A camera trigger signal transmission step for a 2D image; if a start signal is transmitted from an image control unit to a synchronization driving unit, the synchronization driving unit transmits a first 2D / 3D conversion optical element driving signal to a 2D / 3D conversion optical element driver, and the 2D / 3D conversion optical element driver sets the 2D / 3D conversion optical element to a transparent mode according to the first 2D / 3D conversion optical element driving signal, and the image sensor acquires a 2D image, a 2D image acquisition step; After the 2D image acquisition step, the image control unit receives a 2D image from an image sensor, the image control unit transmits an image type request signal to a synchronization driver, the synchronization driver transmits an image type signal indicating that it is a 2D image to the image control unit, and the image control unit temporarily stores the 2D image in a 2D image storage buffer in the order received, 2D image reception step; After the 2D image reception step, the synchronization driver transmits a camera trigger signal to the camera unit, thereby causing the camera unit to adjust the camera frame rate and enter a standby state for capturing the next image, in a camera trigger signal transmission step for 3D image; After the camera trigger signal transmission step for 3D image, the synchronization driver transmits a second 2D / 3D conversion optical device driving signal to a 2D / 3D conversion optical device driver, and the 2D / 3D conversion optical device driver makes the 2D / 3D conversion optical device a microlens mode according to the second 2D / 3D conversion optical device driving signal, so that the image sensor acquires a 3D image, a 3D image acquisition step; and After the 3D image acquisition step, the image control unit receives a 3D image from an image sensor, the image control unit transmits an image type request signal to a synchronization driver, the synchronization driver transmits an image type signal indicating that it is a 3D image to the image control unit, and the image control unit temporarily stores the 3D image in a 3D image storage buffer in the order received, 3D image reception step; A method for operating a 2D and 3D image acquisition camera system characterized by including 8. In Paragraph 7, After the 3D video reception step, the compression and transmission step determines whether the number of video frames stored in the 3D video storage buffer and the 2D video storage buffer, respectively, is equal to the preset video transmission unit size, and if it is determined that the number of frames stored in the 3D video storage buffer and the 2D video storage buffer, respectively, is equal to the preset video transmission unit size, the video control unit performs MPEG encoding on the 2D video temporarily stored in the 2D video storage buffer and the 3D video temporarily stored in the 3D video storage buffer, respectively, to compress them, and then wirelessly transmits them externally through the transceiver; A method for operating a 2D and 3D image acquisition camera system characterized by further including 9. In Paragraph 8, After the compression and transmission step, the image control unit determines whether a stop signal is input, and if a stop signal is not input, returns to the camera trigger signal transmission step for 2D image, and if a stop signal is input, terminates the stop determination step; A method for operating a 2D and 3D image acquisition camera system characterized by further including

Images