Electronic device for acquiring stereo image by processing images acquired using different types of cameras, and operating method therefor

Abstract

An electronic device for acquiring a stereo image by processing images acquired using different types of cameras, and an operating method therefor are provided. The electronic device may comprise: a first camera to which a lens shift-type vibration prevention structure is applied; a second camera to which the vibration prevention structure is not applied; an accelerometer; at least one processor including processing circuitry; and a memory for storing one or more instructions, wherein the electronic device can acquire a first image by using the first camera, acquire a second image by using the second camera, determine, on the basis of an acceleration value measured by using the accelerometer, an area to be cropped from the entire area of the second image, crop the area to be cropped from the second image, resize the cropped image to acquire a rectified image, and generate a stereo image by using the first image and the rectified image.

Description

Electronic device for acquiring a stereo image by processing images acquired by different types of cameras and method of operation thereof

[0001] The present disclosure relates to an electronic device for acquiring stereo images by processing images acquired using different types of cameras, and a method of operation thereof. Specifically, the present disclosure provides an electronic device for generating stereo image content by performing image rectification and quality equalization processing on a plurality of images captured using a plurality of cameras having different focal lengths and field of view (FOV), and a method of operation thereof.

[0002] Augmented reality (AR) or virtual reality (VR) is a technology that overlays virtual objects onto the physical environment or real-world objects to display them together, offering the advantage of fusing virtual objects and virtual information into the real world.

[0003] Recently, as video see-through devices such as head-mounted display devices have become more widespread and utilized, there is an increasing demand for users to view content they have captured themselves (e.g., photo content or video content) through head-mounted display devices. In order to display content captured by users through head-mounted display devices, it is necessary to generate 3D image content by estimating the depth information of objects within the content. At this time, the depth information estimated from the content can be obtained for a common area captured through a stereo camera.

[0004] Users can acquire multiple images by simultaneously capturing an object using multiple cameras (multi-cameras) included in a mobile device such as a smartphone, and utilize these multiple images as stereo images to view 3D image content through an external device, such as a head-mounted display device. Generally, the multiple cameras included in a mobile device consist of different types of cameras with different specifications, such as focal length and field of view (FOV). Images captured using multiple cameras of different types may differ in the field of view (FOV), the position of matching pixels in the Y-axis direction, color temperature, or noise. Therefore, when generating a stereo image using images captured by multiple cameras of different types and viewing the generated stereo image using a head-mounted display device, the quality of the 3D image content is low due to differences between the binocular images, and the user may experience dizziness when viewing the 3D image content.

[0005] The present disclosure provides an electronic device for acquiring a stereo image by processing images acquired by different types of cameras. An electronic device according to one embodiment of the present disclosure may include a first camera having a lens-shift type vibration damping structure, a second camera not having a vibration damping structure, an accelerometer, at least one processor including a processing circuitry, and a memory storing one or more instructions. By executing the one or more instructions individually or collectively by the at least one processor, the electronic device may acquire a first image using the first camera and acquire a second image using the second camera. By executing the one or more instructions individually or collectively by the at least one processor, the electronic device may determine a crop target area corresponding to the first image captured by the tilted lens of the first camera within the entire area of ​​the second image, based on acceleration values ​​measured by the accelerometer. By executing the above one or more instructions individually or collectively by the at least one processor, the electronic device can crop a crop target area from the second image and resize the cropped image to obtain a rectified image. By executing the above one or more instructions individually or collectively by the at least one processor, the electronic device can generate a stereo image using the first image and the rectified image.

[0006] The present disclosure provides a method for an electronic device to obtain a stereo image by processing images obtained by different types of cameras. An operation method of an electronic device according to one embodiment of the present disclosure may include the step of obtaining a first image using a first camera to which a lens-shift type vibration damping structure is applied, and obtaining a second image using a second camera to which a vibration damping structure is not applied. The operation method of the electronic device may include the step of determining a crop target area corresponding to the first image captured by the tilted lens of the first camera within the entire area of ​​the second image based on an acceleration value measured using an acceleration sensor. The operation method of the electronic device may include the step of cropping the crop target area from the second image and resizing the cropped image to obtain a rectification image. The operation method of the electronic device may include the step of generating a stereo image using the first image and the rectification image.

[0007] The present disclosure provides a computer program product comprising a computer-readable storage medium. The storage medium may include instructions readable by an electronic device for performing the operations of acquiring a first image using a first camera having a lens-shift type vibration damping structure and acquiring a second image using a second camera not having a vibration damping structure, determining a crop target area corresponding to the first image captured by the tilted lens of the first camera within the entire area of ​​the second image based on an acceleration value measured using an acceleration sensor, cropping the crop target area from the second image and resizing the cropped image to acquire a rectification image, and generating a stereo image using the first image and the rectification image.

[0008] The present disclosure provides an electronic device for acquiring a stereo image by processing images acquired by different types of cameras. The electronic device of the present disclosure may include a first camera with a lens-shift type vibration damping structure, a second camera without a vibration damping structure, an accelerometer, at least one processor including a processing circuit, and a memory for storing one or more instructions. By executing the one or more instructions individually or collectively by the at least one processor, the electronic device may acquire a first image using the first camera, acquire a second image using the second camera, and determine a cropping area among the entire area of ​​the first image captured by the tilted lens of the first camera based on an acceleration value measured by the accelerometer. By executing the one or more instructions individually or collectively by the at least one processor, the electronic device may crop the cropping area from the first image and resize the cropped image to acquire a first rectification image. By executing the above one or more instructions individually or collectively by the at least one processor, the electronic device can crop an area corresponding to the field of view (FOV) of the first rectification image within the entire area of ​​the second image and resize the cropped image to obtain the second rectification image. By executing the above one or more instructions individually or collectively by the at least one processor, the electronic device can generate a stereo image using the first rectification image and the second rectification image.

[0009] The present disclosure can be easily understood from the combination of the following detailed description and the accompanying drawings, where reference numerals denote structural elements.

[0010] FIG. 1a is a conceptual diagram illustrating the operation of an electronic device capturing an object to acquire a first image and a second image, generating a stereo image using the first image and the second image, and displaying the stereo image through an external device.

[0011] FIG. 1b is a conceptual diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure determining a crop target area in the entire area of ​​a first image and a second image based on the tilting angle of a lens, cropping the crop target area, and resizing to perform image rectification.

[0012] FIG. 2 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to process images acquired by different types of cameras to acquire a stereo image.

[0013] FIG. 3 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure processing images acquired by different types of cameras.

[0014] FIG. 4 is a block diagram illustrating the components of an electronic device according to one embodiment of the present disclosure.

[0015] FIG. 5 is a diagram illustrating data input and output between sensors and software modules of an electronic device according to one embodiment of the present disclosure.

[0016] FIG. 6 is a perspective view illustrating the configurations of a camera having a lens shift type vibration prevention structure according to one embodiment of the present disclosure.

[0017] FIG. 7 is a cross-sectional view illustrating the configuration of a camera having a lens shift type vibration damping structure including a ball guide according to one embodiment of the present disclosure.

[0018] FIG. 8 is a flowchart illustrating a method in which an electronic device according to one embodiment of the present disclosure determines a crop target area in a second image based on the tilting angle of a lens.

[0019] FIG. 9a is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure measuring an acceleration value using an acceleration sensor.

[0020] FIG. 9b is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure measuring an acceleration value using an acceleration sensor.

[0021] FIG. 10 is a flowchart illustrating a method in which an electronic device according to one embodiment of the present disclosure determines a crop target area in a second image based on the tilting angle of a lens.

[0022] FIG. 11 is a flowchart illustrating a method in which an electronic device according to one embodiment of the present disclosure updates intrinsic parameters of a plurality of cameras and models extrinsic parameters to align a first image and a rectified image.

[0023] FIG. 12 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to equalize the quality of a first image and a rectification image.

[0024] FIG. 13 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to obtain a stereo image by processing images obtained by different types of cameras.

[0025] FIG. 14 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure processing images acquired by different types of cameras.

[0026] FIG. 15 is a flowchart illustrating a method in which an electronic device according to one embodiment of the present disclosure determines a crop target area in a first image based on the tilting angle of a lens.

[0027] FIG. 16 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to obtain a stereo image by processing images obtained by different types of cameras.

[0028] FIG. 17 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure to determine a cropping target area among the entire area of ​​an image based on the difference value of the tilted angles of each lens of different types of cameras.

[0029] The terms used in the embodiments of this specification have been selected to be as widely used as possible, taking into account the functions of the present disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description section of the relevant embodiments. Therefore, terms used in this specification should be defined not merely by their names, but based on their meanings and the overall content of the present disclosure.

[0030] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art as described in this specification.

[0031] Throughout this disclosure, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "...module," etc., as used in this specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware or software, or as a combination of hardware and software.

[0032] As used in this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware. Instead, in some situations, the expression “system configured to” may mean that the system is “capable of” in conjunction with other devices or components. For example, the phrase “processor configured to perform A, B, and C” may mean a dedicated processor for performing the said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in memory.

[0033] In addition, when a component is described in the present disclosure as being "connected" or "connected" to another component, it should be understood that the component may be directly connected to or directly connected to the other component, but unless otherwise specifically stated, it may also be connected or connected through another component in between.

[0034] All functions or operations described in this disclosure may be processed individually by a single processor and / or collectively by a plurality of processors. A single processor or a combination of a plurality of processors may include circuitry that performs processing, such as an Application Processor (AP), Communication Processor (CP), Graphical Processing Unit (GPU), Neural Processing Unit (NPU), Microprocessor Unit (MPU), System on Chip (SoC), Integrated Chip (IC), etc.

[0035] It should be understood that the blocks and combinations of flowcharts in the flowcharts illustrated in the present disclosure may be performed by one or more computer programs comprising computer-executable instructions. The one or more computer programs may be stored all in a single memory or may be divided and stored in multiple different memories.

[0036] In the present disclosure, 'field of view (or field of view, FOV)' is an optical technology term that represents the size of an area captured by a camera and displayed within an image in angles.

[0037] In the present disclosure, "wide-angle camera" means a camera comprising a wide-angle lens having a focal length smaller than that of a standard lens (e.g., 40 mm to 60 mm). The focal length of the wide-angle lens included in the wide-angle camera may be, for example, 26 mm to 35 mm. However, it is not limited thereto. A wide-angle camera has a wider field of view (FOV) compared to a camera having a standard lens with a standard focal length, and thereby can capture a wide area of ​​a scene and display it within an image. The field of view (FOV) of a wide-angle camera may be, for example, 90°, but is not limited thereto.

[0038] In the present disclosure, "ultra-wide angle camera" refers to a camera that includes an ultra-wide angle lens having a shorter focal length than the wide-angle lens included in the wide-angle camera. The focal length of the ultra-wide angle lens included in the ultra-wide angle camera may be 26 mm or less. The ultra-wide angle camera has a wider field of view (FOV) compared to the wide-angle camera, and thereby can capture and display a wider area of ​​the scene in the image compared to the wide-angle camera. The field of view (FOV) of the ultra-wide angle camera may be, for example, 100° or more, but is not limited thereto.

[0039] In the present disclosure, 'image rectification' refers to an image transformation process that aligns images captured through two or more cameras or images captured from different viewpoints onto the same image plane. Image rectification is a technique primarily used in stereo vision, and is mainly used to align images captured through multiple cameras or to transform images captured from different viewpoints into the same coordinate system. In one embodiment of the present disclosure, image rectification may include at least one image processing among stereo image alignment, which aligns images captured through two or more cameras onto the same image plane; distortion correction, which linearly transforms the image using a distortion model representing lens distortion; and image warping, which uses a warping matrix.

[0040] In the present disclosure, 'quality equalization' refers to image processing that equalizes the image quality of images captured through a plurality of cameras of different types. In one embodiment of the present disclosure, quality equalization may include image processing that equalizes at least one of color temperature and noise between images.

[0041] In the present disclosure, functions related to 'Artificial Intelligence' are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or AI-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or AI models stored in memory. Alternatively, if the one or more processors are AI-dedicated processors, the AI-dedicated processors may be designed with a hardware structure specialized for processing a specific AI model.

[0042] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives) are created by a basic artificial intelligence model being trained using multiple learning data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0043] In the present disclosure, an 'artificial intelligence model' may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values ​​and performs neural network operations through operations between the results of operations of a previous layer and the plurality of weights. The plurality of weights possessed by the plurality of neural network layers may be optimized by the learning results of the artificial intelligence model. For example, the plurality of weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network model may include a Deep Neural Network (DNN), such as a Convolutional Neural Network, a Recurrent Neural Network, a Restricted Boltzmann Machine, a Deep Belief Network, a Bidirectional Recurrent Deep Neural Network, or Deep Q-Networks, but is not limited to the examples described above.

[0044] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0045] Embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0046] FIG. 1a shows that an electronic device (100) captures an object (10, 20) to acquire a first image (i1) and a second image (i2), and uses the first image (i1) and the second image (i2) to form a stereo image (is Generates ) and stereo image(i s This is a conceptual diagram illustrating the operation of displaying ) through an external device (200).

[0047] Referring to FIG. 1a, the electronic device (100) may be a smartphone or tablet PC including a plurality of cameras (111, 112). However, the electronic device (100) is not limited to the one shown in FIG. 1a, and the electronic device (100) may be implemented as a mobile device such as, for example, a laptop computer, a digital camera, an e-book terminal, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a navigation device, or an MP3 player.

[0048] Referring to FIG. 1a, the electronic device (100) may include a plurality of cameras (111, 112) of different types. In one embodiment of the present disclosure, the plurality of cameras (111, 112) may be implemented as cameras of different types with different focal lengths and fields of view (FOV). For example, the first camera (111) may be a wide-angle camera and the second camera (112) may be an ultra-wide-angle camera.

[0049] An electronic device (100) can obtain multiple images (i1, i2) by photographing an object (10, 20) using different types of cameras (111, 112) (Operation ①). The electronic device (100) can obtain a first image (i1) by photographing an object (10, 20) using a first camera (111), and obtain a second image (i2) by photographing an object (10, 20) using a second camera (112). The first image (i1) and the second image (i2) may have different fields of view (FOV). In the embodiment illustrated in FIG. 1a, the field of view of the second image (i2) may be larger than the field of view of the first image (i1). The Y-axis position coordinate values ​​of the matching pixels (P1, P2) in the first image (i1) and the second image (i2) may be different.

[0050] In the embodiment illustrated in FIG. 1a, among the pixels of images (i1, i2) obtained by photographing the same object, for example, a person (10), the position coordinate value in the Y-axis direction of the first pixel (P1) of the first image (i1) may be greater than the position coordinate value in the Y-axis direction of the second pixel (P2) that matches the first pixel (P1) in the second image (i2). That is, the image of the person (10) in the first image (i1) may be positioned higher in the Y-axis direction than the image of the person (10) in the second image (i2).

[0051] The electronic device (100) can perform dynamic rectification on a plurality of images (i1, i2) (operation ②). In one embodiment of the present disclosure, the electronic device (100) can perform stereo image alignment to align the first image (i1) and the second image (i2) to the same image plane, thereby correcting the position coordinate values ​​in the Y-axis direction of the matching pixels (P1, P2) in the images and correcting the field of view (FOV) of the images (i1, i2) to be the same. In the embodiment illustrated in FIG. 1a, the electronic device (100) can correct the position coordinate values ​​in the Y-axis direction of the first pixel (P1) in the first image (i1) and the second pixel (P2) in the second image (i2) to pixels (P1', P2') having the same value by changing the principal point among the intrinsic parameters of the first camera (111) and the second camera (112), respectively. Additionally, the electronic device (100) can correct the angle of view of the first image (i1) and the second image (i2) to be the same by changing the focal length among the intrinsic parameters of the first camera (111) and the second camera (112). Through this, the electronic device (100) can acquire the first rectified image (i1') and the second rectified image (i2'). The electronic device (100) can align the epipolar lines of the first rectification image (i1') and the second rectification image (i2') in a horizontal direction.

[0052] The electronic device (100) can equalize the quality of a plurality of rectified images (i1', i2') obtained by performing dynamic rectification (Operation 3). The electronic device (100) can obtain a first image (i1'') and a second image (i2'') with equalized quality by performing image processing to make the color temperature and noise of the first rectified image (i1') and the second rectified image (i2') the same. In one embodiment of the present disclosure, the electronic device (100) can convert the color temperature of the second rectified image (i2') to be the same as the color temperature of the first rectified image (i1') by using a matching curve that converts the cumulative distribution function of the color temperature. In one embodiment of the present disclosure, an electronic device (100) can equalize the noise of the first image (i1') and the second image (i2') by inputting the first image (i1') and the second image (i2'), which have equalized color temperatures, into a noise equalization model.

[0053] The electronic device (100) uses a first image (i1'') and a second image (i2'') obtained through dynamic rectification and quality equalization to obtain a stereo image (i s Can generate ) (Operation ④). Stereo image(i s ) is the left eye image(i L ) and right eye image(i R Includes ) and left eye image(i L ) is obtained from the first image (i1'') acquired through dynamic rectification and quality equalization, the right eye image (i R ) can be obtained from a second image (i2'') obtained through dynamic rectification and quality equalization, respectively. The electronic device (100) has a stereo image (i s) can be transmitted to an external device (200). In this case, the electronic device (100) and the external device (200) are paired with each other using a short-range wireless communication network such as Bluetooth or Wi-Fi Direct, and the electronic device (100) uses a short-range wireless communication network to transmit a stereo image (i s ) can be transmitted to an external device (200).

[0054] Stereo image (i s ) can be displayed through an external device (200). The external device (200) may be composed of an augmented reality (AR) device or a virtual reality (VR) device. The external device (200) may be implemented as a video see-through device, for example, a head-mounted display device. In the present disclosure, a 'head-mounted display (HMD) device' is a device worn on a user's head that provides a virtual reality experience to the user. However, it is not limited thereto, and the external device (200) may be implemented as, for example, an augmented reality helmet that provides an augmented reality experience, a face-mounted display (FMD) device worn on a user's face, or augmented reality glasses in the shape of glasses.

[0055] The external device (200) receives a stereo image (i) from the electronic device (100). s ) can be displayed. The external device (200) includes a left-eye display and a right-eye display, and through the left-eye display, a left-eye image (i LDisplays ) and through the right eye display, the right eye image (i R It can display ).

[0056] FIG. 1b shows an electronic device (100) according to one embodiment of the present disclosure cropping target area (i) in the entire area of ​​a first image (i1) and a second image (i2) based on the tilting angle of a lens. target Determine ) and crop target area (i target This is a conceptual diagram to explain the operation of performing image rectification by cropping and resizing ).

[0057] Referring to FIG. 1b, an electronic device (100, see FIG. 1a) can obtain multiple images (i1, i2) by photographing an object using different types of cameras (111, 112, see FIG. 1a). The electronic device (100) can obtain a first image (i1) by photographing an object using a first camera (111) and obtain a second image (i2) by photographing an object using a second camera (112). The first image (i1) and the second image (i2) may have different fields of view (FOV). In the embodiment illustrated in FIG. 1b, the field of view of the second image (i2) may be larger than the field of view of the first image (i1). The Y-axis position coordinate values ​​of the matching pixels (P1, P2) in the first image (i1) and the second image (i2) may be different.

[0058] In the embodiment illustrated in FIG. 1b, among the pixels of images (i1, i2) obtained by photographing the same object, e.g., a person (10), the position coordinate value in the Y-axis direction of the first pixel (P1) of the first image (i1) may be greater than the position coordinate value in the Y-axis direction of the second pixel (P2) that matches the first pixel (P1) in the second image (i2). That is, the image of the person (10) in the first image (i1) may be positioned higher in the Y-axis direction than the image of the person (10) in the second image (i2). Because an error occurs in the intrinsic parameters of the camera as the lens of one of the different types of cameras tilts, the position coordinate values ​​of corresponding pixels in the first image (i1) and the second image (i2) differ, and as a result, a rectification error may occur.

[0059] The electronic device (100) can perform dynamic rectification on a plurality of images (i1, i2). In one embodiment of the present disclosure, the electronic device (100) includes an acceleration sensor and can obtain information regarding a tilting angle representing the tilted angle of each lens of different cameras based on an acceleration value measured using the acceleration sensor. Based on the information regarding the tilting angle, the electronic device (100) can determine a crop target area in the entire area of ​​each of the plurality of images (i1, i2). For example, the electronic device (100) determines a first crop target area (i) in the entire area of ​​the first image (i1) based on an acceleration measurement value measured by the acceleration sensor. 1_target Determine ) and the second crop target area (i) among the entire area of ​​the second image (i2). 2_target ) can be determined.

[0060] In one embodiment of the present disclosure, the electronic device (100) changes the focal length among the intrinsic parameters of each of the first camera (111) and the second camera (112) based on the tilting angle, thereby creating a crop target area (i) such that the first image (i1) and the second image (i2) have the same field of view (FOV). 1_target , i 2_target ) can be determined. In this case, the electronic device (100) can determine the crop target area (i) using a distortion model that indicates not only the tilting angle of the lens but also the image scale or the degree of lens distortion. 1_target , i 2_target ) can be determined. In addition, the electronic device (100) can change the principal point among the internal parameters of the second camera (112) to determine the first crop target area (i 1_target The first pixel (P1) and the second crop target area (i) in ) 2_target The second crop target area (i) such that the position coordinate value in the Y-axis direction of the second pixel (P2) in ) has the same value 2_target i 2_target It can be changed to '

[0061] The electronic device (100) has a first crop target area (i 1_target ) and the second crop target area (i 2_target Dynamic rectification can be performed through cropping and resizing using '). In one embodiment of the present disclosure, the electronic device (100) obtains a first crop target area (i) from a first image (i1). 1_target ) is cropped, and the cropped image is resized to obtain a first rectification image (i1'), and a second crop target area (i) is obtained from the second image (i2). 2_targetA second rectification image (i2') can be obtained by cropping ') and resizing the cropped image.

[0062] The electronic device (100) can equalize the quality of the first rectification image (i1') and the second rectification image (i2'). Since the quality equalization is the same as described in FIG. 1a, a redundant description is omitted.

[0063] Although not illustrated in FIGS. 1a and 1b, the first camera (111, see FIG. 1a) may be a camera with a lens shift type vibration damping structure applied, and the second camera (112, see FIG. 1a) may be a camera without a vibration damping structure applied. Since the lens shift method is applied only to the first camera (111), the areas of the images captured by the first camera (111) and the second camera (112) may differ. A specific embodiment for dynamically rectifying the difference in image areas between the first image (i1) and the second image (i2) that occurs as a result of the lens shift method being applied only to the first camera (111) will be described in detail with reference to FIGS. 2 and 3.

[0064] FIG. 2 is a flowchart illustrating a method in which an electronic device (100) according to one embodiment of the present disclosure processes images obtained by different types of cameras to obtain a stereo image.

[0065] FIG. 3 is a diagram illustrating the operation of an electronic device (100) according to one embodiment of the present disclosure processing images (i1, i2) acquired by different types of cameras.

[0066] Hereinafter, the function and / or operation of the electronic device (100) will be described in detail with reference to FIG. 2 and FIG. 3 together.

[0067] In step S210 of FIG. 2, the electronic device (100) acquires a first image using a first camera to which a lens shift type vibration damping structure is applied, and acquires a second image using a second camera. In one embodiment of the present disclosure, the first camera is implemented as a wide-angle camera and a lens shift type vibration damping structure may be applied. The lens shift type vibration damping structure is a vibration damping structure in which optical image stabilization (OIS) is implemented by a lens barrel including a lens being connected to a coil through a ball guide and the lens barrel being shifted in the X-axis, Y-axis, and Z-axis directions by a current applied to the coil. In one embodiment of the present disclosure, the second camera is implemented as an ultra-wide-angle camera and may not have a vibration damping structure applied.

[0068] Since the first camera is equipped with a lens-shift type dustproof structure, the lens may sag along the direction of gravity due to the weight of the lens of the first camera even when autofocus or optical image stabilization (OIS) is not operating. If the user is moving while shooting, the lens of the first camera may sag as an external force is applied to the electronic device (100). Referring to the embodiment illustrated in FIG. 3, the lens (L1) included in the first camera is equipped with a lens-shift type dustproof structure, and the lens (L1) may be tilted downward due to gravity even without the application of an external force. When the user holds the electronic device (100) and raises it upward along the Y-axis, an external force is applied to the electronic device (100), and as the inertial force due to the external force and gravity are applied together to the lens (L1), a lens sagging phenomenon may occur in which the lens (L1) tilts downward. Due to the sagging phenomenon of the lens (L1), when photographing an object (300) using the first camera, the lower part of the object (300) is photographed compared to when photographing using the lens (L2) of the second camera, which does not have a dustproof structure applied. That is, due to the sagging phenomenon of the lens (L1) of the first camera, the principal point among the intrinsic parameters of the first camera may change in the downward direction.

[0069] Although not shown in the drawing, when the user moves the electronic device (100) downward along the Y-axis, the lens (L1) of the first camera tilts upward due to the influence of inertial force and gravity caused by external forces.

[0070] The dustproof structure of the lens shift method and the lens sagging phenomenon will be explained in detail with reference to FIGS. 9a and 9b.

[0071] An electronic device (100) can obtain a first image (i1) by photographing an object (300) using a first camera, and obtain a second image (i2) by photographing an object (300) using a second camera. Referring to the embodiment illustrated in FIG. 3, in the first image (i1), an object (e.g., a tree) may be positioned at the top of the image frame due to the sagging phenomenon of the lens (L1) of the first camera, and the lower part of the object (e.g., the trunk part of the tree) may be positioned in the central area of ​​the image frame. On the other hand, in the second image (i2) obtained using a second camera to which a dustproof structure is not applied, an object (e.g., the leaf part of the tree) may be positioned in the central area of ​​the image frame. In one embodiment of the present disclosure, the first camera is implemented as a wide-angle camera and the second camera as an ultra-wide-angle camera, so that the field of view (FOV) of the second image (i2) may be larger than the field of view (FOV) of the first image (i1).

[0072] In step S220 of FIG. 2, the electronic device (100) determines a crop target area corresponding to the first image captured by the tilted lens of the first camera within the entire area of ​​the second image, based on acceleration values ​​measured using an accelerometer. Referring together to the embodiment illustrated in FIG. 3, the electronic device (100) determines a crop target area (i) corresponding to the first image (i1) captured by the tilted lens of the first camera within the entire area of ​​the second image (i2). target ) can be determined. The electronic device (100) moves the image frame of the second image (i2) downward along the Y-axis direction by a pixel distance corresponding to the tilted angle of the lens, and the crop target area (i target ) can be determined.

[0073] In one embodiment of the present disclosure, the electronic device (100) includes an acceleration sensor and can obtain acceleration measurements along the X-axis and Y-axis using the acceleration sensor. The acceleration measurements measured by the acceleration sensor may be the sum of linear acceleration and gravity generated as the first camera moves due to an external force. The acceleration measurements measured by the acceleration sensor and the tilting angle, which indicates the degree of tilt of the lens of the first camera, may be in a proportional relationship. In one embodiment of the present disclosure, the electronic device (100) may calculate the tilting angle through an operation of multiplying the acceleration measurements by a proportionality constant value. The proportionality constant may be pre-calculated through testing or estimated through online calibration.

[0074] In one embodiment of the present disclosure, the electronic device (100) further includes a gyro sensor in addition to an acceleration sensor, and can obtain an angular velocity measurement value based on the rotational inertia of the lens of the first camera using the gyro sensor. The electronic device (100) can obtain a 3-axis angular velocity measurement value including the angular velocities of roll, pitch, and yaw using the gyro sensor. In one embodiment of the present disclosure, the electronic device (100) can calculate a tilting angle indicating the degree to which the lens of the first camera is tilted by performing an operation to add the acceleration measurement value obtained through the acceleration sensor and the angular velocity measurement value obtained through the gyro sensor.

[0075] In one embodiment of the present disclosure, the electronic device (100) changes the principal point among the intrinsic parameters of the second camera using information regarding the tilting angle, thereby cropping the target area (i target The position of ) can be determined. When the principal point changes, the center point of the image moves downward along the Y-axis, so the crop target area (itarget A black bound (bb) in which the image is not displayed may occur depending on the positional movement of ). In one embodiment of the present disclosure, the electronic device (100) changes the focal length among the internal parameters of the second camera to remove the black bound (bb), thereby cropping the target area (i target You can determine the size of ').

[0076] In step S230 of FIG. 2, the electronic device (100) crops a crop target area from the second image and resizes the cropped image to obtain a rectified image. Referring together to the embodiment illustrated in FIG. 3, the electronic device (100) crops a crop target area (i2) from the second image (i2). target Cropping ') to crop the image(i crop ) obtain and crop image(i crop Resize the rectified image (i rectified ) can be obtained. In one embodiment of the present disclosure, the electronic device (100) obtains a rectification image (i) based on the focal length among the intrinsic parameters of the second camera. rectified ) can be obtained. The electronic device (100) can obtain a rectification image (i) by adjusting the focal length of the second camera. rectified The field of view (FOV) of the second image (i2) can be corrected to be the same as the field of view (FOV) of the first image (i1). Since the field of view of the second image (i2) before cropping was greater than the field of view of the first image (i1), there is no loss of field of view compared to the first image (i1) even after cropping.

[0077] In step S240 of FIG. 2, the electronic device (100) generates a stereo image using a first image and a rectification image. The electronic device (100) may generate a stereo image including a left eye image and a right eye image by applying the first image as a left eye image and the rectification image as a right eye image. In one embodiment of the present disclosure, the electronic device (100) may generate three-dimensional stereoscopic video content by storing a binocular video including a left eye image and a right eye image in a side-by-side (SBS) or MV-HEVC (Multiview High Efficiency Video Coding) format. In one embodiment of the present disclosure, the electronic device (100) may generate three-dimensional stereoscopic photo content by storing a still image composed of a left eye image and a right eye image in SBS or Samsung Extended Format (SEF).

[0078] A user can obtain multiple images by simultaneously capturing an object using multiple cameras (multi-cameras) included in an electronic device (100), and can view 3D image content through an external device, such as a head-mounted display device, by utilizing the multiple images as stereo images. Generally, the multiple cameras included in the electronic device (100) are composed of different types of cameras with different specifications, such as focal length and field of view (FOV). Images captured using multiple cameras of different types may differ in the field of view (FOV), the position of matching pixels in the Y-axis direction, color temperature, or noise. When a stereo image is generated using images captured using multiple cameras of different types and the generated stereo image is viewed using a head-mounted display device, the quality of the 3D image content is low due to the difference between the binocular images, and the user may feel dizzy when viewing the 3D image content.

[0079] The present disclosure aims to provide an electronic device (100) and a method of operation thereof for generating stereo image content by image rectification and quality equalization processing of a plurality of images captured using a plurality of cameras (111, 112) (see FIG. 1a) having different focal lengths and field of view (FOV). More specifically, the present disclosure aims to provide an electronic device (100) and a method of operation thereof that performs dynamic rectification including equalization of position coordinate values ​​in the Y-axis direction and equalization of the viewing angle between a first image (i1) and a second image (i2) due to lens sagging in a first camera (111) among a plurality of cameras (111, 112) to which a lens shift type vibration prevention structure is applied, and equalization of image quality between the first image (i1) and the second image (i2), and generates a stereo image using images obtained through dynamic rectification and image quality equalization.

[0080] An electronic device (100) according to an embodiment illustrated in FIGS. 1a to 3 acquires multiple images (i1, i2) by simultaneously capturing an object using multiple cameras (111, 112) of different types, performs distortion correction, stereo image alignment, and image warping on the multiple images (i1, i2) to make the position coordinate values ​​in the Y-axis direction of matching pixels of the multiple images (i1, i2) identical, performs dynamic rectification to equalize the angle of view, performs image quality equalization to make the color temperature and noise of the multiple images (i1, i2) identical, and can generate a stereo image using the dynamic rectified and image quality equalized images. In particular, an electronic device (100) according to one embodiment of the present disclosure can dynamically rectify a first image (i1) and a second image (i2) caused by a lens sagging phenomenon of a first camera (111) to which a lens shift type dustproof structure is applied, by obtaining information regarding the tilting angle of the lens using an acceleration measurement value measured by an acceleration sensor, and by cropping and resizing only the image area corresponding to the first image (i1) among the entire area of ​​the second image (i2) based on the tilting angle, and can obtain a rectified image and generate a stereo image using the first image (i1) and the rectified image. By doing so, the electronic device (100) according to one embodiment of the present disclosure can generate a stereo image in which the horizontal alignment and quality of the binocular images constituting the stereo image are equalized without changing the sensors of the plurality of cameras (111, 112) or adding components, thereby providing a technical effect of providing high-quality three-dimensional stereoscopic image content to the user without additional cost.In addition, an electronic device (100) according to one embodiment of the present disclosure has a crop target area (i) in a second image (i2). crop In determining ), the crop target area (i) is determined through calculations on the camera's intrinsic parameters. target Determine the position and size of ), and the rectification image (i rectified By obtaining ), dynamic rectification and image quality equalization processing can be performed at high speed, and image processing time can be reduced.

[0081] FIG. 4 is a block diagram illustrating the components of an electronic device (100) according to one embodiment of the present disclosure.

[0082] Referring to FIG. 4, the electronic device (100) may include a camera (110), a sensor (120), a processor (130), a memory (140), and a communication interface (150). The camera (110), the sensor (120), the processor (130), the memory (140), and the communication interface (150) may each be electrically and / or physically connected to one another. FIG. 4 illustrates only essential components for explaining the function and / or operation of the electronic device (100), and the components included in the electronic device (100) are not limited to those illustrated in FIG. 4. In one embodiment of the present disclosure, the electronic device (100) may further include a battery that supplies power to the camera (110), the sensor (120), the processor (130), and the communication interface (150). In one embodiment of the present disclosure, the electronic device (100) may not include a communication interface (150).

[0083] The camera (110) is configured to acquire an image of a space and an object within the space by capturing an object. In one embodiment of the present disclosure, the camera (110) may be implemented as an RGB camera. However, it is not limited thereto, and in one embodiment of the present disclosure, the camera (110) may be implemented as any known type of camera, such as an RGB-depth camera including a depth estimation function, a dynamic vision sensor camera, a stereo fisheye camera, a grayscale camera, or an infrared camera.

[0084] The camera (110) may include a lens module, an image sensor, and an image processing module. The camera (110) may acquire a still image or video of an object by means of an image sensor (e.g., CMOS or CCD). The image processing module may encode the still image, which consists of a single image frame acquired through the image sensor, or the video data, which consists of multiple image frames, and transmit it to the processor (130).

[0085] The camera (110) may be composed of multiple cameras. In the embodiment illustrated in FIG. 4, the camera (110) may include two cameras, including a first camera (111) and a second camera (112). However, it is not limited thereto, and the camera (110) may include three or more cameras.

[0086] The first camera (111) and the second camera (112) may be different types of cameras. For example, the first camera (111) may be a wide-angle camera including a wide-angle lens, and the second camera (112) may be an ultra-wide-angle camera including an ultra-wide-angle lens. In the present disclosure, "wide-angle camera" means a camera including a wide-angle lens having a focal length smaller than the focal length of a standard lens (e.g., 40 mm to 60 mm). The focal length of the wide-angle lens included in the wide-angle camera may be, for example, 26 mm to 35 mm. However, it is not limited thereto. A wide-angle camera has a wider field of view (FOV) compared to a camera having a standard lens with a standard focal length, and through this, can capture a wide area of ​​a scene and display it in an image. The field of view of the wide-angle camera may be, for example, 90°, but is not limited thereto. In the present disclosure, an "ultra-wide angle camera" refers to a camera that includes an ultra-wide angle lens having a shorter focal length than the wide-angle lens included in the wide-angle camera. The focal length of the ultra-wide angle lens included in the ultra-wide angle camera may be 26 mm or less. The ultra-wide angle camera has a wider field of view (FOV) compared to the wide-angle camera, and thereby can capture a scene of a wider area compared to the wide-angle camera and display it in the image. The field of view (FOV) of the ultra-wide angle camera may be, for example, 100° or more, but is not limited thereto. The second camera (112) includes a lens having a relatively wider field of view compared to the first camera (111), so that a second image with a wider field of view can be obtained by capturing a larger area compared to the first camera (111).The first image captured by the first camera (111) may include a relatively narrow area compared to the second image. However, it is not limited thereto, and in one embodiment of the present disclosure, the first camera (111) may be an ultra-wide angle camera and the second camera (112) may be a wide angle camera.

[0087] In one embodiment of the present disclosure, a lens shift type vibration damping structure may be applied to the first camera (111), and a second camera (112) may not have a vibration damping structure applied. More specifically, the first camera (111) may have a lens shift type vibration damping structure applied such that a lens barrel including a lens is connected to a coil through a ball guide, and optical image stabilization (OIS) is implemented by shifting the lens barrel in the X-axis, Y-axis, and Z-axis directions by the current applied to the coil. The lens shift type vibration damping structure applied to the first camera (111) will be described in detail with reference to FIGS. 6 and 7.

[0088] Since the first camera (111) has a dustproof structure with a lens shift type, the lens may sag due to the weight of the lens of the first camera even when autofocus or optical image stabilization (OIS) is not operating. When the user moves while shooting, an external force applied to the electronic device (100) may cause the lens of the first camera (111) to sag due to the inertial force against the external force.

[0089] The sensor (120) is configured to acquire a measurement value by sensing the movement, position change, rotation, speed change, etc. of the electronic device (100). In one embodiment of the present disclosure, the sensor (120) may include an acceleration sensor (122) and a gyroscope sensor (124).

[0090] The accelerometer (122) is a sensor configured to measure the rate of change of speed of the electronic device (100), i.e., linear acceleration, and can obtain measurements of three-axis acceleration of the X-axis, Y-axis, and Z-axis. The accelerometer (122) can measure linear acceleration caused by external forces applied to the electronic device (100) and acceleration caused by gravity. The accelerometer (122) can measure the acceleration, gravity, and vibration of the electronic device (100), convert them into numerical values ​​(acceleration measurements), and provide the converted numerical values ​​to the processor (130).

[0091] A gyro sensor (124) is a sensor configured to measure the angular velocity (rotational speed) of an electronic device (100), and can obtain an angular velocity measurement value by measuring the rotational inertia caused by the rotation of the electronic device (100) and converting it into an electrical signal. The gyro sensor (124) can obtain a 3-axis angular velocity measurement value including the angular velocities of roll, pitch, and yaw, and can provide the obtained 3-axis angular velocity measurement value to a processor (130).

[0092] In one embodiment of the present disclosure, the sensor (120) may further include a magnetometer in addition to the acceleration sensor (122) and the gyroscope sensor (124), and may be configured as an IMU sensor (Inertial Measurement Unit) configured to measure the movement speed, direction, angle, and gravitational acceleration of the electronic device (100) through a combination of the acceleration sensor (122), the gyroscope sensor (124), and the magnetometer. Although not shown in the drawings, the sensor (120) may further include a position sensor (or GPS sensor) in addition to the acceleration sensor (122) and the gyroscope sensor (124).

[0093] The processor (130) can execute one or more instructions of a program stored in memory (140). The processor (130) may be composed of hardware components that perform arithmetic, logic, and input / output operations and image processing. Although the processor (130) is depicted as a single element in FIG. 4, it is not limited thereto. In one embodiment of the present disclosure, the processor (130) may be composed of one or more elements.

[0094] The processor (130) may include various processing circuits and / or multiple processors. For example, the term 'processor' as used in the disclosure, including in the claims, may include at least one processor and various processing circuits. In at least one processor, one or more processors may be configured to perform the various functions described herein in a distributed manner, individually and / or collectively. As used in the disclosure, 'processor', 'at least one processor', and 'one or more processors' may be configured to perform various functions. However, these terms cover, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor can perform all functions. Additionally, at least one processor may include a combination of processors performing various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0095] One or more processors included in the processor (130) may be circuitry such as a system on chip (SoC) or an integrated circuit (IC). The processor (130) may be implemented as a general-purpose processor such as a CPU (Central Processing Unit), AP (Application Processor), or DSP (Digital Signal Processor), a graphics-dedicated processor such as a GPU (Graphic Processing Unit) or VPU (Vision Processing Unit), or an artificial intelligence-dedicated processor such as an NPU (Neural Processing Unit). The processor (130) may be controlled to process input data according to predefined operation rules or an artificial intelligence model. Alternatively, if the processor (130) is an artificial intelligence-dedicated processor, the artificial intelligence-dedicated processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0096] The memory (140) may be composed of at least one type of storage medium, such as a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), or an optical disk.

[0097] The memory (140) may store instructions executed by the processor (130) to perform functions and / or operations for the electronic device (100) to process images acquired by a plurality of cameras (111, 112) to acquire a stereo image. In one embodiment of the present disclosure, the memory (140) may store at least one of instructions, an algorithm, a data structure, program code, and an application program that the processor (130) can read. The instructions, algorithm, data structure, and program code stored in the memory (140) may be implemented in a programming or scripting language such as, for example, C, C++, Java, assembler, etc.

[0098] The memory (140) may store instructions, algorithms, data structures, or program codes for a dynamic rectification module (142), an image quality equalization module (144), and a stereo image generation module (146). A 'module' included in the memory (140) refers to a unit that processes a function or operation performed by the processor (130), and this can be implemented as software such as instructions, algorithms, data structures, or program code.

[0099] The processor (130) can be implemented by executing instructions or program codes stored in memory (140). Hereinafter, with reference to FIGS. 4 and FIGS. 5 together, the functions and / or operations performed by the processor (130) by executing instructions or program codes of each of the plurality of modules stored in memory (140), and the data input / output between the plurality of modules (dynamic rectification module (142), image quality equalization module (144), and stereo image generation module (146)) and components (e.g., accelerometer (122) and gyroscope (124)) will be described in detail.

[0100] FIG. 5 is a diagram illustrating data input / output between sensors (122, 124) of an electronic device (100) according to one embodiment of the present disclosure and software modules (142, 144, 146) stored in memory (140). Although not shown in FIG. 5, a processor (130, see FIG. 4) may execute instructions or program code of a dynamic rectification module (142), an image quality equalization module (144), and a stereo image generation module (146) to perform related functions and / or operations.

[0101] Referring to FIG. 4 and FIG. 5 together, the first camera (111) can capture an object to acquire a first image and provide the first image to the processor (130). The second camera (112) can capture an object to acquire a second image and provide the second image to the processor (130). The accelerometer (122) can measure acceleration values ​​to acquire three-axis acceleration measurements and provide the acceleration measurements to the processor (130). The gyroscope sensor (124) can acquire three-axis angular velocity measurements including roll, pitch, and yaw angular velocities and provide the angular velocity measurements to the processor (130). The processor (130) can input the first image, the second image, the acceleration measurements, and the angular velocity measurements into the dynamic rectification module (142).

[0102] The dynamic rectification module (142) is composed of instructions or program code for executing a function and / or operation to perform image rectification by using intrinsic parameters and extrinsic parameters of cameras (111, 112) to equalize the position coordinate values ​​of pixels in the Y-axis direction between multiple images acquired by different types of cameras (111, 112) and to make the field of view (FOV) the same. The processor (130) can perform at least one image processing among stereo image alignment, which corrects the Y-axis position coordinate values ​​between matching pixels in the first image and the second image to be the same by aligning the first image and the second image to the same image plane by executing instructions or program code of the dynamic rectification module (142); distortion correction, which transforms the image linearly using a distortion model representing lens distortion; and image warping, which uses a warping matrix.

[0103] In one embodiment of the present disclosure, the processor (130) models lens distortion that changes according to the position of an object and can update the intrinsic parameters of the first camera (111) and the second camera (112) using the modeled lens distortion model. The processor (130) can model the lens distortion by using information regarding the object that is in focus during an auto focus operation in each of the first camera (111) and the second camera (112), or by using the depth value of the object.

[0104] In one embodiment of the present disclosure, the processor (130) can update the internal parameters of the first camera (111) and the second camera (112) by modeling the change in image scale caused by the change in focal length that occurs during autofocus operation.

[0105] In one embodiment of the present disclosure, the processor (130) may change the principal point among the internal parameters of the camera to reflect the position of the image, because the position where the image is formed changes due to the movement of the lens when the optical image stabilization (OIS) function is operated. In particular, since a lens shift type dustproof structure is applied to the first camera (111), a lens sagging phenomenon may occur in which the lens sags downward due to the weight of the lens or inertial force caused by an external force applied to the electronic device (100). The processor (130) may calculate a tilting angle indicating the degree to which the lens of the first camera (111) is tilted using an acceleration measurement value obtained through an acceleration sensor (122). The acceleration measurement value measured by the acceleration sensor (122) and the tilting angle of the lens of the first camera (111) may be in a proportional relationship. In one embodiment of the present disclosure, the processor (130) can calculate the tilting angle by multiplying the acceleration measurement value by a proportionality constant value. The proportionality constant may be calculated in advance through testing or estimated through online calibration.

[0106] However, it is not limited thereto, and in one embodiment of the present disclosure, the processor (130) may calculate a tilting angle indicating the degree to which the lens of the first camera (111) is tilted by performing an operation to sum the acceleration measurement value obtained through the acceleration sensor (122) and the angular velocity measurement value obtained through the gyroscope sensor (124). The processor (130) may calculate the tilting angle based on the following mathematical formula.

[0107]

[0108] In mathematical formula 1, θ represents the tilting angle of each axis, A and B are experimentally obtainable constants, a is the accelerometer output value, and ω is the gyro output value. The processor (130) can calculate the tilting angle based on the above mathematical formula.

[0109] Based on the calculated tilting angle, the processor (130) can determine a crop target area corresponding to the first image captured by the tilted lens of the first camera (111) within the entire area of ​​the second image acquired through the second camera (112). The processor (130) can determine the area of ​​the entire area of ​​the second image that is moved downward along the Y-axis direction by a pixel distance corresponding to the tilting angle as the crop target area. In one embodiment of the present disclosure, the processor (130) can determine the location of the crop target area by changing the principal point among the internal parameters of the second camera (112) using information regarding the tilting angle. A specific embodiment in which the processor (130) determines the crop target area within the entire area of ​​the second image by changing the principal point among the internal parameters of the second camera (112) based on the tilting angle of the first camera (111) will be described in detail with reference to FIGS. 8 to 10.

[0110] The processor (130) can obtain a cropped image by cropping a cropping target area from the second image, and obtain a rectified image by resizing the cropped image to be the same size as the entire area of ​​the original image. In one embodiment of the present disclosure, the processor (130) obtains a focal length (f) among the internal parameters of the second camera (112). x , f y By changing ), the size of the cropping area within the entire area of ​​the second image can be determined, and the field of view (FOV) of the rectified image can be corrected to be the same as the field of view (FOV) of the first image.

[0111] The processor (130) can update an extrinsic parameter representing the relative positional relationship between the first camera (111) and the second camera (112). In one embodiment of the present disclosure, the processor (130) models the extrinsic parameter based on a rotation matrix (R) generated by video stabilization and a translation vector (t) based on the positional relationship between the first camera (111) and the second camera (112), and can update the extrinsic parameter of each of the first camera (111) and the second camera (112) using the modeled extrinsic parameter.

[0112] The processor (130) can model a warping matrix and perform hardware-assisted warping using the warping matrix. In this case, the processor (130) can model a grid-shaped warping matrix for faster computation when performing remap. In one embodiment of the present disclosure, the processor (130) can model the relationship between the input image and the first image through backward processing of the rectified image as a warping matrix.

[0113] The processor (130) can obtain a first rectification image and a second rectification image by not only changing the internal parameters of the second camera (112) but also updating the external parameters of the first camera (111) and the second camera (112) and performing image warping. The processor (130) can input the first rectification image and the second rectification image into an image quality equalization module (144).

[0114] The image quality equalization module (144) is composed of instructions or program code for executing a function and / or operation to equalize the quality of images by adjusting the color temperature and noise of a plurality of input images to be the same. By executing the instructions or program code of the image quality equalization module (144), the processor (130) can obtain a quality-equalized first rectification image and a quality-equalized second rectification image by performing image processing to make the color temperature and noise of the first rectification image and the second rectification image the same. In one embodiment of the present disclosure, the processor (130) can obtain a cumulative distribution function (CDF) of the color temperature of each of the first rectification image and the second rectification image, and generate a matching curve that converts the cumulative distribution function of the color temperature of the second rectification image into the cumulative distribution function of the color temperature of the first rectification image. The processor (130) can use a matching curve to convert the color temperature of the second rectification image to be the same as the color temperature of the first rectification image.

[0115] The processor (130) can input a first rectification image and a second rectification image with equalized color temperature into a pre-trained network model to equalize the noise between the first rectification image and the second rectification image. In one embodiment of the present disclosure, the processor (130) can equalize the noise between the first rectification image and the second rectification image using an artificial intelligence model trained to equalize the noise of the input images and output images with equalized noise. The artificial intelligence model may be implemented as a deep neural network, such as a convolution neural network, a transformer, or a diffusion model. However, it is not limited to this, and the processor (130) may also use a non-deep learning model, such as a block matching 3D model or a non-local mean, to equalize the noise of the first rectification image and the second rectification image.

[0116] The processor (130) can input the first rectification image and the quality-equalized second rectification image into the stereo image generation module (146).

[0117] The stereo image generation module (146) is composed of instructions or program code for executing a function and / or operation to generate a stereo image by utilizing input images as a left eye image and a right eye image. By executing the instructions or program code of the stereo image generation module (146), the processor (130) can generate a stereo image including a left eye image and a right eye image by applying a first rectification image as a left eye image and a quality equalization second rectification image as a right eye image, respectively. In one embodiment of the present disclosure, the processor (130) can generate three-dimensional stereoscopic video content by storing a binocular video including a left eye image and a right eye image in a side-by-side (SBS) or MV-HEVC (Multiview High Efficiency Video Coding) format. In one embodiment of the present disclosure, the processor (130) may generate three-dimensional stereoscopic photo content by saving a still image composed of a left eye image and a right eye image as SBS or Samsung Extended Format (SEF).

[0118] The communication interface (150) is a hardware device configured to perform data communication with an external device and / or server. The communication interface (150) may be configured as a device that performs data communication with an external device or server using at least one of the following data communication methods, for example: wired LAN, wireless LAN, Wi-Fi, Wi-Fi Direct, Bluetooth, BLE (Bluetooth Low Energy), infrared communication (IrDA, infrared Data Association), NFC (Near Field Communication), Wibro (Wireless Broadband Internet), WiMAX (World Interoperability for Microwave Access), SWAP (Shared Wireless Access Protocol), WiGig (Wireless Gigabit Alliance), and RF communication.

[0119] In one embodiment of the present disclosure, the communication interface (150) can be connected to an external device (200, see FIG. 1a) by the control of the processor (130) and can transmit and receive data. The communication interface (150) can be paired with the external device (200) via a short-range wireless communication network, for example, Bluetooth, BLE, or Wi-Fi Direct, and can transmit stereo image data to the external device (200).

[0120] FIG. 6 is a perspective view illustrating the configurations of a camera (600) having a lens shift type vibration prevention structure according to one embodiment of the present disclosure.

[0121] The camera (600) illustrated in FIG. 6 may be identical to, for example, the first camera (111) of the electronic device (100). However, it is not limited thereto, and the camera (600) illustrated in FIG. 6 may also be identical to the second camera (112).

[0122] Referring to FIG. 6, the camera (600) may include components for implementing optical image stabilization (OIS) and autofocus (AF) using a lens shift method. This is a dustproof structure implemented. In this disclosure, the "lens shift method" refers to a method of correcting distortion or deformation caused by vibration or shaking by moving the lens in the opposite direction to the vibration or shaking caused by an external force. The lens shift dustproof structure is distinguished from the sensor shift method, which moves the sensor. In one embodiment of this disclosure, the lens shift method may be implemented as a ball guide type lens shift, in which the lens barrel is connected to a coil through a ball guide, and the lens barrel is shifted in the X-axis, Y-axis, and Z-axis directions by the current applied to the coil to perform optical image stabilization (OIS) and autofocus. The ball guide type lens shift dustproof structure will be described in detail with reference to FIG. 7.

[0123] Referring to FIG. 6, the camera (600) may include an optical image stabilization carrier (610), a middle guide (620), an autofocus carrier (630), and a housing (640).

[0124] The OIS carrier (610) includes the lens of the camera (600) and is configured to be positioned on the top of the module of the camera (600) to perform optical image stabilization (OIS) functions. The OIS carrier (610) may include a combination of magnets and coils and a suspension. The OIS carrier (610) moves along the Y-axis direction (up and down direction) and can correct vertical shaking of the camera (600). In one embodiment of the present disclosure, the OIS carrier (610) moves finely with a magnet / coil combination and a suspension structure, and can automatically correct the position of the lens when shaking is detected.

[0125] The middle guide (620) guides the movement of the OIS carrier (610) along the X-axis direction (left and right direction). Since the OIS carrier (610) moves in the Y-axis direction and the middle guide (620) moves in the X-axis direction, 2-axis optical image stabilization can be achieved.

[0126] The autofocus carrier (630) is connected to the OIS carrier (610) and the middle guide (620) and can perform an autofocusing function while moving along the Z-axis direction. In one embodiment of the present disclosure, the autofocus carrier (630) can perform an autofocusing function by using a coil motor to push and pull the lens unit in the Z-axis direction. If the camera (600) is a ball guide type lens shift structure, the autofocus carrier (630) is connected to a ball and can be moved in the Z-axis direction by the ball.

[0127] The housing (640) is composed of an outer case or outer structure that fixes and supports the entire structure of the camera (600). The housing (640) performs the function of a frame that protects the internal components (610, 620, 630) from external impact and can provide structural restrictions so that the internal components (610, 620, 630) can move only in a preset direction.

[0128] FIG. 7 is a cross-sectional view illustrating the configuration of a camera (700) having a lens shift type vibration prevention structure including a ball guide according to one embodiment of the present disclosure.

[0129] The camera (700) illustrated in FIG. 7 may be identical to, for example, the first camera (111) of the electronic device (100). However, it is not limited thereto, and the camera (700) illustrated in FIG. 7 may also be identical to the second camera (112).

[0130] In the present disclosure, a ball guide type lens shift vibration prevention structure refers to a system designed to guide and move a lens (710) in the Z-axis direction using a small ball (e.g., a ball made of steel or ceramic). Referring to FIG. 7, a camera (700) may include a lens (710), a lens barrel (712), a magnet (720), a ball (730), a coil (740), an image sensor (750), a housing (760), a circuit board (770), and a shield can (780).

[0131] The lens (710) is mounted on the lens barrel (712) and can be secured by the lens barrel (712).

[0132] The magnet (720) is a permanent magnet and may be composed of, for example, NdFeB. The magnet (720) may have an alternating N pole and S pole. The magnet (720) is placed around the coil (740), and a ball (730) may be placed between the magnet (720) and the coil (740). When current is applied to the coil (740), the magnet (720) can form a magnetic field to induce movement of the coil (740).

[0133] The coil (740) is configured to generate a physical force through interaction with the magnet (720) by the Lorentz force when current is applied, thereby moving the lens (710) by moving the lens barrel (712). The coil (740) may be composed of, for example, a copper wire insulated with enamel.

[0134] The ball (730) is configured to rotate and induce linear movement along the Z-axis direction of the lens (710) and the lens barrel (712) while the lens barrel (712) is moved by the magnet (720) and the coil (740). The ball (730) may be made of, for example, steel or ceramic.

[0135] The image sensor (750) is configured to convert light (or photons) received through the lens (710) into an electrical signal and to process the converted electrical signal into digital image data. The image sensor (750) may be mounted on a printed circuit board (PCB) (770). Since the image sensor (750) is a well-known component in the field of cameras and optics, a detailed description is omitted.

[0136] The housing (760) consists of an outer case or outer structure that fixes and supports the entire structure of the camera (700). A shielding can (780) may be formed on the outer surface of the housing (760).

[0137] A shield can (780) is configured to block or shield external electromagnetic interference (EMI) or radio frequency interference (RFI). The shield can (780) is formed to surround the outer surface of the housing (760) and the circuit board (770) and can protect internal components such as the lens (710), magnet (720), ball (730), coil (740), and image sensor (750) from external electromagnetic interference or radiation. The shield can (780) may be implemented in a metallic material capable of shielding electromagnetic interference.

[0138] A camera (700) having a ball guide type lens shift vibration prevention structure can perform an autofocus function by applying current to a coil (740) by the processor (130, see FIG. 4) when the distance of a subject is measured by the processor (130, see FIG. 4), and the lens barrel (712) moves along the Z-axis direction by the magnet (720) and the coil (740) by the Lorentz force, thereby moving the lens (710) and focusing as the lens (710) moves. In this case, the ball (730) can serve to reduce friction so that the lens barrel (712) moves slightly between the magnet (720) and the coil (740). Additionally, the camera (700) can perform optical image stabilization (OIS) by moving the lens (710) and lens barrel (712) in the opposite direction to the shake through the interaction between the magnet (720) and the coil (740) when current is applied to the coil (740) by the control of the processor (130) when the shake is detected by the processor (130).

[0139] FIG. 8 is a flowchart illustrating a method in which an electronic device (100) according to one embodiment of the present disclosure determines a crop target area in a second image based on the tilting angle of a lens.

[0140] Steps S810 to S830 illustrated in FIG. 8 represent operations that embody the operation according to step S220 illustrated in FIG. 2. Step S810 illustrated in FIG. 8 may be performed after the operation according to step S210 of FIG. 2 has been performed. Step S230 of FIG. 2 may be performed after the operation according to step S830 illustrated in FIG. 8 has been performed.

[0141] In step S810, the electronic device (100) obtains an acceleration value by using an acceleration sensor to measure the acceleration caused by external force and gravity applied to the electronic device. In one embodiment of the present disclosure, among the plurality of cameras included in the electronic device (100), a first camera (111, see FIG. 4) may be equipped with a lens shift type vibration damping structure. When an external force is applied to the electronic device (100), a lens sagging phenomenon may occur in which the lens of the first camera (111) sags downward due to the inertial force and gravity caused by the external force. The electronic device (100) measures the acceleration value using an acceleration sensor (122, see FIG. 4) and can measure the degree of lens sagging based on the measured acceleration value.

[0142] FIGS. 9a and 9b are drawings illustrating the operation of an electronic device (100) according to one embodiment of the present disclosure measuring an acceleration value using an acceleration sensor. Referring to FIG. 9a together with step S810 of FIG. 8, when an external force is applied to the electronic device (100) in an upward direction along the Z-axis direction, the lens (L1) of the first camera (111) to which a lens-shift type vibration damping structure is applied may experience a lens sagging phenomenon in which it sags downward due to the action of inertial force and gravity caused by the external force. In this case, the processor (130, see FIG. 4) of the electronic device (100) can obtain an acceleration value by measuring the sum of the linear acceleration force and gravity caused by the external force acting on the electronic device (100) using an acceleration sensor (122, see FIG. 4).

[0143] Referring to FIG. 9b together with step S810 of FIG. 8, when an external force is applied to move the electronic device (100) downward along the Z-axis direction, the lens (L1) of the first camera (111) to which the lens shift type vibration prevention structure is applied may experience lens sagging due to the sum of the inertial force and gravity caused by the external force (although the electronic device (100) is shown in a state where it is flipped downward in the drawing, the first camera (111) is shown in its original direction that is not flipped, so it is shown as being tilted upward). In this case, the processor (130, see FIG. 4) of the electronic device (100) can obtain an acceleration value by measuring the sum of the linear acceleration force and gravity caused by the external force acting on the electronic device (100) using an acceleration sensor (122, see FIG. 4).

[0144] Referring again to FIG. 8, in step S820, the electronic device (100) obtains information regarding a tilting angle, which indicates the degree to which the lens of the first camera is tilted, based on an acceleration value. The acceleration measurement value measured by the acceleration sensor and the tilting angle, which indicates the degree to which the lens (L1) of the first camera (111) is tilted, may be in a proportional relationship. In one embodiment of the present disclosure, the processor (130) of the electronic device (100) may calculate the tilting angle through an operation of multiplying the acceleration measurement value by a proportionality constant value. The proportionality constant may be pre-calculated through testing or estimated through online calibration.

[0145] However, it is not limited thereto, and in one embodiment of the present disclosure, the electronic device (100) may perform an operation to sum the acceleration measurement value obtained through the acceleration sensor (122) and the angular velocity measurement value obtained through the gyroscope sensor (124, see FIG. 4) to calculate a tilting angle indicating the degree to which the lens of the first camera (111) is tilted.

[0146] In step S830, the electronic device (100) determines a crop target area corresponding to the first image within the entire area of ​​the second image based on information regarding the tilting angle. The processor (130) of the electronic device (100) can determine a crop target area corresponding to the first image captured by the tilted lens of the first camera (111) within the entire area of ​​the second image obtained through the second camera (112, see FIG. 3) using information regarding the tilting angle obtained in step S820. The processor (130) can determine the area within the entire area of ​​the second image that is moved downward along the Y-axis direction by a pixel distance corresponding to the tilting angle as the crop target area. In one embodiment of the present disclosure, the processor (130) can determine the crop target area by changing or adjusting the internal parameters of the second camera (112). A specific embodiment in which the processor (130) changes or adjusts the internal parameters of the second camera (112) to determine the crop target area will be described in detail with reference to FIG. 10.

[0147] FIG. 10 is a flowchart illustrating a method in which an electronic device (100) according to one embodiment of the present disclosure determines a crop target area in a second image based on the tilting angle of a lens.

[0148] Steps S1010 and S1020 illustrated in FIG. 10 represent operations that embody the operation according to step S830 illustrated in FIG. 8. Step S1010 illustrated in FIG. 10 may be performed after the operation according to step S820 of FIG. 8 has been performed. After the operation according to step S1020 illustrated in FIG. 10 has been performed, step S230 of FIG. 2 may be performed.

[0149] In step S1010, the electronic device (100) changes the principal point among the intrinsic parameters of the second camera based on information regarding the tilting angle. In one embodiment of the present disclosure, the processor (130, see FIG. 4) of the electronic device (100) may change the principal point of the second camera (112, see FIG. 4) using the following mathematical formulas.

[0150]

[0151]

[0152] In mathematical formula 2, K represents a matrix regarding the intrinsic parameters of the second camera (112), and f x , f y represents the focal length with respect to the X-axis and Y-axis, respectively, of the lens of the second camera (112). C x , C y represents the principal point with respect to the X-axis and Y-axis, respectively, of the lens of the second camera (112). Referring to Equation 3, the processor (130) [represents] the principal point (C) of the lens of the second camera (112). x , C y ) focal length (f x , f y ) and tilting angle (α x , α y C the pub by summing the values ​​multiplied by the tangent value of ) x ', C y It can be changed to '

[0153] In step S1020, the electronic device (100) determines the location of the cropping area within the entire area of ​​the second image based on the changed principal point. Referring to Equation 3 above, the processor (130) determines the changed principal point C x ', C yBy using ', the location of the crop target area corresponding to the area of ​​the first image obtained by the first camera (111, see FIG. 4) among the entire area of ​​the second image obtained by the second camera (112) can be determined.

[0154] Step S1030 of FIG. 10 represents an operation that embodies the operation according to Step S230 illustrated in FIG. 2. In Step S1030, the electronic device (100) acquires a rectification image having a field of view (FOV) corresponding to the field of view (FOV) of the first image, based on the focal length among the intrinsic parameters of the second camera. Referring to Equation 2 above, the processor (130) [acquires] the focal length (f among the intrinsic parameters of the second camera (112). x , f y By changing ), the size of the crop target area within the entire area of ​​the second image can be determined. The processor (130) can perform rectification to correct the field of view (FOV) of the second image to be the same as the field of view (FOV) of the first image by cropping the crop target area in the second image.

[0155] FIG. 11 is a flowchart illustrating a method in which an electronic device (100) according to one embodiment of the present disclosure updates intrinsic parameters of a plurality of cameras and models extrinsic parameters to align a first image and a rectified image.

[0156] Steps S1110 to S1140 illustrated in FIG. 11 may be performed after the operation according to step S230 illustrated in FIG. 2 is performed. After the operation according to step S1140 illustrated in FIG. 11 is performed, the operation according to step S240 of FIG. 2 may be performed.

[0157] In step S1110, the electronic device (100) models lens distortion that changes according to the position of an object. In one embodiment of the present disclosure, the electronic device (100) may model lens distortion by using information regarding an object that is in focus during an auto-focus operation in each of the first camera (111, see FIG. 4) and the second camera (112, see FIG. 4), or by using the depth value of the object. In one embodiment of the present disclosure, the electronic device (100) may also model a change in image scale caused by a change in focal length that occurs during an auto-focus operation.

[0158] In step S1120, the electronic device (100) updates the intrinsic parameters of the first camera and the second camera using a modeled lens distortion model. In one embodiment of the present disclosure, the electronic device (100) may update the intrinsic parameters of the first camera and the second camera based on a modeled image scale as well as a lens distortion model.

[0159] In step S1130, the electronic device (100) models an extrinsic parameter based on a rotation matrix generated by video stabilization and a positional relationship between the first camera and the second camera.

[0160] In one embodiment of the present disclosure, the electronic device (100) models a warping matrix and can perform hardware-assisted warping using the warping matrix. In this case, the electronic device (100) can model a grid-shaped warping matrix for fast computation when performing remap. In one embodiment of the present disclosure, the electronic device (100) can model the relationship between the input image and the first image through backward processing of the rectification image as a warping matrix.

[0161] In step S1140, the electronic device (100) aligns the first image and the rectification image in the Y-axis direction. In one embodiment of the present disclosure, the electronic device (100) updates both the internal parameters and the external parameters of the first camera (111) and the second camera (112), respectively, and can align the Y-axis position coordinate values ​​of corresponding pixels in the first image and the rectification image using the updated internal parameters and external parameters.

[0162] FIG. 12 is a flowchart illustrating a method in which an electronic device (100) according to one embodiment of the present disclosure equalizes the quality of a first image and a rectification image.

[0163] Steps S1210 and S1220 shown in FIG. 12 can be performed after the operation by step S240 shown in FIG. 2 has been performed.

[0164] In step S1210, the electronic device (100) converts the color temperature of the rectification image to be equal to the color temperature of the first image using a matching curve that converts the cumulative distribution function of the color temperature. In one embodiment of the present disclosure, the electronic device (100) may obtain the cumulative distribution function of the color temperature of the first image and the rectification image, respectively, and generate a matching curve that converts the cumulative distribution function of the color temperature of the rectification image to the cumulative distribution function of the color temperature of the first image. The electronic device (100) may convert the color temperature of the rectification image to be equal to the color temperature of the first image using the matching curve.

[0165] In step S1220, the electronic device (100) inputs a first image with equalized color temperature and a rectified image into a noise equalization model to equalize the noise of the first image and the rectified image. In one embodiment of the present disclosure, the electronic device (100) may equalize the noise of the first image and the rectified image using an artificial intelligence model trained to equalize the noise of the input images and output images with equalized noise. The 'artificial intelligence model' may be implemented as a deep neural network, such as a convolution neural network, a transformer, or a diffusion model. However, it is not limited to this, and the electronic device (100) may also use a non-deep learning model, such as a block matching 3D model or a non-local mean, to equalize the noise of the first image and the rectification image.

[0166] FIG. 13 is a flowchart illustrating a method for an electronic device (100) according to one embodiment of the present disclosure to obtain a stereo image by processing images obtained by different types of cameras.

[0167] FIG. 14 is a diagram illustrating the operation of an electronic device (100) according to one embodiment of the present disclosure processing images acquired by different types of cameras.

[0168] Hereinafter, the function and / or operation of the electronic device (100) will be described in detail with reference to FIG. 13 and FIG. 14 together.

[0169] In step S1310 of FIG. 13, the electronic device (100) acquires a first image using a first camera to which a lens shift type vibration damping structure is applied, and acquires a second image using a second camera. In one embodiment of the present disclosure, the first camera is implemented as a wide-angle camera and a lens shift type vibration damping structure may be applied. In one embodiment of the present disclosure, the second camera is implemented as an ultra-wide-angle camera and a dust damping structure may not be applied.

[0170] Since the first camera is equipped with a lens-shift type dustproof structure, the lens may sag along the direction of gravity due to the weight of the lens of the first camera even when autofocus or optical image stabilization (OIS) is not operating. If the user is moving while shooting, the lens of the first camera may sag as an external force is applied to the electronic device (100). Referring together to the embodiment illustrated in FIG. 14, the lens (L1) included in the first camera is equipped with a lens-shift type dustproof structure, and the lens (L1) may be tilted downward due to gravity even without the application of an external force. When the user raises the electronic device (100) upward along the Y-axis while holding it, an external force is applied to the electronic device (100), and as the inertial force due to the external force and gravity are applied together to the lens (L1), a lens sagging phenomenon may occur in which the lens (L1) tilts downward. Due to the sagging phenomenon of the lens (L1), when photographing an object (1400) using the first camera, the lower part of the object (1400) is photographed compared to when photographing using the lens (L2) of the second camera, which does not have a dustproof structure applied. That is, due to the sagging phenomenon of the lens (L1) of the first camera, the principal point among the intrinsic parameters of the first camera may change downward.

[0171] The electronic device (100) can obtain a first image (i1) by taking a picture of an object (1400) using a first camera, and obtain a second image (i2) by taking a picture of an object (1400) using a second camera. In one embodiment of the present disclosure, the first camera is implemented as a wide-angle camera and the second camera is implemented as an ultra-wide-angle camera, so that the field of view (FOV) of the second image (i2) may be larger than the field of view (FOV) of the first image (i1).

[0172] In step S1320 of FIG. 13, the electronic device (100) determines a crop target area among the entire area of ​​the first image based on acceleration values ​​measured using an accelerometer. Referring together to the embodiment illustrated in FIG. 14, the processor (130, see FIG. 4) of the electronic device (100) obtains information regarding the degree of tilt of the lens (L1) of the first camera based on the acceleration values, and a first crop target area (i) based on the degree of tilt of the lens (L1) of the first camera. 1_target The position of ) can be determined. The processor (130) calculates a pixel distance corresponding to the degree of tilt of the lens (L1) of the first camera acquired based on the acceleration value, and moves the image frame of the first image (i1) upward along the Y-axis direction by the calculated pixel distance to the first crop target area (i 1_target ) can be determined.

[0173] In one embodiment of the present disclosure, the electronic device (100) can obtain acceleration measurements along the X-axis and Y-axis using an acceleration sensor.

[0174] FIG. 15 is a flowchart illustrating a method in which an electronic device (100) according to one embodiment of the present disclosure determines a crop target area in a first image based on the tilting angle of a lens.

[0175] Steps S1510 to S1530 illustrated in FIG. 15 represent operations that embody the operation according to step S1320 illustrated in FIG. 13. Step S1510 illustrated in FIG. 15 may be performed after the operation according to step S1310 of FIG. 13 has been performed. Hereinafter, the operation method of the electronic device (100) will be described in detail with reference to step S1320 of FIG. 13 and the flowchart illustrated in FIG. 15.

[0176] In step S1510, the electronic device (100) obtains an acceleration value by using an acceleration sensor to measure the acceleration caused by the external force and gravity applied to the electronic device (100). In one embodiment of the present disclosure, among the plurality of cameras included in the electronic device (100), a lens shift type vibration prevention structure may be applied to the first camera. When an external force is applied to the electronic device (100), a lens sagging phenomenon may occur in which the lens (L1) of the first camera sags downward due to the inertial force and gravity caused by the external force. The electronic device (100) measures the acceleration value using an acceleration sensor (122, see FIG. 4) and can measure the degree of lens sagging based on the measured acceleration value. The acceleration measurement value measured by the acceleration sensor (122) may be the sum of the linear acceleration and gravity generated as the first camera moves due to the external force.

[0177] In step S1520, the electronic device (100) obtains information regarding a tilting angle, which indicates the degree to which the lens of the first camera is tilted, based on an acceleration value. The acceleration measurement value measured by the acceleration sensor (122) and the tilting angle, which indicates the degree to which the lens (L1) of the first camera is tilted, may be in a proportional relationship. In one embodiment of the present disclosure, the processor (130, see FIG. 4) of the electronic device (100) may calculate the tilting angle through an operation of multiplying the acceleration measurement value by a proportionality constant value. The proportionality constant may be pre-calculated through testing or estimated through online calibration.

[0178] In one embodiment of the present disclosure, the electronic device (100) further includes a gyroscope sensor (124, see FIG. 4) in addition to the acceleration sensor (122), and can obtain an angular velocity measurement value based on the rotational inertia of the lens (L1) of the first camera using the gyroscope sensor (124). In one embodiment of the present disclosure, the processor (130) can calculate a tilting angle indicating the degree to which the lens (L1) of the first camera is tilted by performing an operation to sum the acceleration measurement value obtained through the acceleration sensor (122) and the angular velocity measurement value obtained through the gyroscope sensor (124). Since the specific method by which the processor (130) calculates the tilting angle of the lens (L1) of the first camera using the acceleration measurement value and the angular velocity measurement value is the same as that described with reference to FIG. 4, FIG. 5, and Equation 1, a redundant description is omitted.

[0179] In step S1530, the electronic device (100) determines a crop target area among the entire area of ​​the first image based on information regarding the tilting angle. In one embodiment of the present disclosure, the processor (130) of the electronic device (100) may determine the location of the first crop target area by changing the principal point among the intrinsic parameters of the first camera based on the tilting angle calculated in step S1520. When the principal point is changed, the center point of the image may be moved downward along the Y-axis. Referring together to the embodiment illustrated in FIG. 14, the first crop target area (i 1_target A black bound (bb) in which an image is not displayed may occur depending on the positional movement of ). In one embodiment of the present disclosure, the processor (130) changes the focal length among the internal parameters of the first camera to remove the black bound (bb), thereby creating a first crop target area (i 1_target By changing the size of ) the first crop target area (i 1_target ) to i 1_targetIt can be updated to '

[0180] After the operation by step S1530 shown in FIG. 15 is performed, step S1330 of FIG. 13 may be performed.

[0181] Referring again to FIG. 13, in step S1330, the electronic device (100) crops a crop target area from the first image and resizes the cropped image to obtain a first rectified image. Referring together to the embodiment illustrated in FIG. 14, the processor (130) obtains an updated first crop target area (i1) from the first image (i1). 1_target Cropping ') to create the first cropped image (i 1_crop ) obtain and the first cropped image (i 1_crop A first rectified image (i1_rectified) can be obtained by resizing the image. In one embodiment of the present disclosure, the processor (130) can obtain a first rectified image (i1_rectified) having a field of view (FOV) equal to the ratio of the original image ('first image (i1)' in FIG. 14) without black bounds (bb) being displayed by changing the focal length among the internal parameters of the first camera. In one embodiment of the present disclosure, the focal length may be changed to a preset value, such as a factory setting, but is not limited thereto.

[0182] In step S1340 of FIG. 13, the electronic device (100) crops an area of ​​the entire area of ​​the second image that corresponds to the field of view (FOV) of the first rectified image, and resizes the cropped image to obtain the second rectified image. Referring together to the embodiment illustrated in FIG. 14, the processor (130) obtains a second crop target area (i) of the entire area of ​​the second image (i2) that has the same field of view as the field of view of the first rectified image (i1_rectified). 2_target ) can be determined. The processor (130) determines the second crop target area (i) from the second image (i2). 2_target A second rectified image (i2_rectified) can be obtained by cropping the image and resizing the cropped image. In one embodiment of the present disclosure, the processor (130) can obtain a second rectified image (i2_rectified) having the same angle of view as the first rectified image (i1_rectified) by changing the principal point and focal length among the intrinsic parameters of the second camera. Since the angle of view of the second image (i2) was greater than the angle of view of the first image (i1), there is no loss of angle of view in the second rectified image (i2_rectified) obtained as a result of performing dynamic rectification.

[0183] In step S1350 of FIG. 13, the electronic device (100) generates a stereo image using a first rectification image and a second rectification image. Referring together to the embodiment illustrated in FIG. 14, the processor (130) may generate a stereo image including a left eye image and a right eye image by applying the first rectification image (i1_rectified) as a left eye image and the second rectification image (i2_rectified) as a right eye image. In one embodiment of the present disclosure, the processor (130) may generate three-dimensional stereoscopic video content by storing a binocular video including a left eye image and a right eye image in a side-by-side (SBS) or MV-HEVC (Multiview High Efficiency Video Coding) format. In one embodiment of the present disclosure, the processor (130) may generate three-dimensional stereoscopic photo content by saving a still image composed of a left eye image and a right eye image as SBS or Samsung Extended Format (SEF).

[0184] FIG. 16 is a flowchart illustrating a method in which an electronic device (100) according to one embodiment of the present disclosure processes images acquired by different types of cameras to acquire a stereo image.

[0185] In step 1610, the electronic device (100) acquires a first image and a second image, respectively, using a first camera and a second camera to which a lens shift type vibration damping structure is applied. In one embodiment of the present disclosure, the first camera may be implemented as a wide-angle camera and the second camera may be implemented as an ultra-wide-angle camera. In the embodiment illustrated in FIG. 16, unlike step S210 of FIG. 2 and step S1310 of FIG. 13, a lens shift type vibration damping structure may be applied to both the first camera and the second camera. Since a lens shift type vibration damping structure is applied to both the first camera and the second camera, the lenses may sag along the direction of gravity due to the weight of the lenses of the first camera and the second camera, respectively, even when autofocus or optical image stabilization (OIS) is not operating. If the user moves while taking photos, an external force applied to the electronic device (100) may cause sagging in the lenses included in the first camera and the second camera, respectively.

[0186] FIG. 17 is a diagram illustrating the operation of determining a crop target area among the entire area of ​​an image based on the difference value (△α) of the tilting angles (α1, α2) of each lens (L1, L2) of different types of cameras, in an electronic device (100) according to one embodiment of the present disclosure.

[0187] Referring to step S1610 of FIG. 16 in conjunction with the embodiment illustrated in FIG. 17, since a dustproof structure of the lens shift type is applied to both the first camera and the second camera, the first lens (L1) included in the first camera and the second lens (L2) included in the second camera may both be tilted downwards by gravity even without the application of an external force. If a user holds the electronic device (100) and raises it upwards along the Y-axis, an external force is applied to the electronic device (100), and as the inertial force and gravity caused by the external force are applied together to the first lens (L1) and the second lens (L2), a lens sagging phenomenon may occur in which the lenses (L1, L2) tilt downwards. For example, the first lens (L1) included in the first camera may be tilted more than the second lens (L2), and in this case, the first tilting angle (α1), which is the tilted angle of the first lens (L1), may be greater than the second tilting angle (α2), which is the tilted angle of the second lens (L2). However, this is merely an example, and the embodiments of the present disclosure are not limited to the first lens (L1) of the first camera being tilted more than the second lens (L2) of the second camera.

[0188] In the embodiment illustrated in FIG. 17, the electronic device (100) can obtain a first image (i1) by photographing an object (1700) using a first camera and obtain a second image (i2) by photographing the object (1700) using a second camera. In the embodiment illustrated in FIG. 17, the first lens (L1) is tilted more than the second lens (L2), so when photographing the object (1700) using the first camera, the lower part of the object (1700) is photographed compared to when photographing using the second camera. That is, due to the sagging phenomenon of the first lens (L1) of the first camera, the principal point among the intrinsic parameters of the first camera may change to a lower direction than the principal point of the second camera.

[0189] In one embodiment of the present disclosure, the first camera is a wide-angle camera and the second camera is an ultra-wide-angle camera, so that the field of view (FOV) of the second image (i2) may be larger than the field of view (FOV) of the first image (i1).

[0190] Referring again to FIG. 16, in step S1620, the electronic device (100) obtains a first tilting angle indicating the degree to which the first lens of the first camera is tilted and a second tilting angle indicating the degree to which the second lens of the second camera is tilted, based on acceleration values ​​measured using an accelerometer. The electronic device (100) can obtain acceleration values ​​by measuring the acceleration caused by external force and gravity applied to the electronic device (100) using an accelerometer. The processor (130, see FIG. 4) of the electronic device (100) can obtain information regarding the tilting angles of the first lens (L1) of the first camera and the second lens (L2) of the second camera, respectively, based on the acceleration values. Referring together with the embodiment illustrated in FIG. 17, the processor (130) can calculate a first tilting angle (α1) representing the degree of tilt of the first lens (L1) and a second tilting angle (α2) representing the degree of tilt of the second lens (L2) through an operation of multiplying the acceleration measurement value by a proportionality constant value. The proportionality constant value may be predetermined as a different value for each of the first camera and the second camera, or may be estimated through online calibration. For example, the first proportionality constant value for the first camera may be greater than the second proportionality constant value for the second camera. In the embodiment illustrated in FIG. 17, the first tilting angle (α1) of the first lens (L1) may be greater than the second tilting angle (α2) of the second lens (L2). However, it is not limited thereto.

[0191] Referring again to FIG. 16, in step S1630, the electronic device (100) determines a crop target area among the entire area of ​​the first image acquired by the camera having a larger tilt angle value between the first image and the second image, based on the difference value between the first tilt angle and the second tilt angle. In one embodiment of the present disclosure, the processor (130) may identify the image acquired by the camera having a smaller tilt angle value among the first image acquired through the first camera and the second image acquired through the second camera, and determine the identified image as a reference image. Referring together to the embodiment illustrated in FIG. 17, the first lens (L1) of the first camera is tilted more than the second lens (L2) of the second camera; in this case, the processor (130) may determine the second image acquired by the second camera as a reference image. The processor (130) may determine the image not determined as a reference image as an image to be rectified and determine a crop target area in the determined image. For example, the processor (130) may determine the first image as the image to be rectified. The processor (130) may calculate a difference value (△α) between the first tilting angle (α1) and the second tilting angle (α2), and determine a cropping target area among the entire area of ​​the first image determined as the image to be rectified based on the calculated difference value (△α). The processor (130) may calculate the tilting angle difference value (△α) based, for example, on the following Equation 4.

[0192]

[0193] In one embodiment of the present disclosure, the processor (130) can determine the position of the crop target area by changing the principal point among the intrinsic parameters of the first camera based on the difference value (△α) of the tilting angle. Additionally, the processor (130) can change the size of the crop target area and adjust the angle of view by changing the focal length among the intrinsic parameters of the first camera.

[0194] Referring again to FIG. 16, in step S1640, the electronic device (100) crops the crop target area from the first image and resizes the cropped first image to obtain a first rectified image. The specific method by which the processor (130) determines the crop target area by changing the principal point and focal length among the internal parameters of the camera and obtains the first rectified image by performing cropping and resizing is the same as the method described in FIG. 13 to FIG. 15, except that it uses the difference value (△α) of the tilting angle, so a redundant description is omitted.

[0195] In step S1650, the electronic device (100) crops an area of ​​the entire area of ​​the second image that corresponds to the field of view (FOV) of the first rectification image, and resizes the cropped image to obtain the second rectification image.

[0196] In step S1660, the electronic device (100) generates a stereo image using the first rectification image and the second rectification image. Since steps S1650 and S1660 are substantially identical to steps S1340 and S1350 illustrated in FIG. 13, a redundant description is omitted.

[0197] The electronic device (100) according to the embodiment illustrated in FIGS. 16 and 17 performs dynamic rectification of an image based on the difference value (△α) of the tilting angle between the first lens (L1) of the first camera and the second lens (L2) of the second camera when a lens-shift type vibration damping structure is applied to both different cameras (first camera and second camera). This allows for the generation of a stereo image in which the horizontal alignment and quality of the binocular images constituting the stereo image are equalized without changing the sensors of the cameras or adding additional components to the electronic device (100). Accordingly, the electronic device (100) according to one embodiment of the present disclosure provides a technical effect that can provide high-quality three-dimensional stereoscopic image content to the user without additional cost.

[0198] One aspect of the present disclosure provides an electronic device (100) that obtains a stereo image by processing images obtained by different types of cameras. An electronic device (100) according to one embodiment of the present disclosure may include a first camera (111) with a lens shift type vibration damping structure, a second camera (112) without a vibration damping structure, an accelerometer (122), at least one processor (130) including a processing circuitry, and a memory (140) that stores one or more instructions. By executing the one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) may obtain a first image using the first camera (111) and obtain a second image using the second camera (112). By executing one or more of the above commands individually or collectively by the at least one processor (130), the electronic device (100) can determine a crop target area corresponding to the first image captured by the tilted lens of the first camera (111) within the entire area of ​​the second image based on acceleration values ​​measured using an acceleration sensor (122). By executing one or more of the above commands individually or collectively by the at least one processor (130), the electronic device (100) can crop the crop target area from the second image and resize the cropped image to obtain a rectified image.By executing one or more of the above instructions individually or collectively by the at least one processor (130), the electronic device (100) can generate a stereo image using the first image and the rectification image.

[0199] In one embodiment of the present disclosure, the first camera (111) is a wide camera, and the second camera (112) is an ultra-wide camera, and the field of view (FOV) of the second camera (112) may be larger than the field of view of the first camera (111).

[0200] In one embodiment of the present disclosure, by executing one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) can measure acceleration caused by external force and gravity applied to the electronic device (100) using an acceleration sensor (122), obtain an acceleration value, and obtain information regarding a tilting angle indicating the degree to which the lens of the first camera (111) is tilted based on the acceleration value. By executing one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) can determine a crop target area corresponding to the first image among the entire area of ​​the second image based on the information regarding the obtained tilting angle.

[0201] In one embodiment of the present disclosure, the electronic device (100) may further include a gyro sensor (124) that measures three-axis angular velocity. By executing one or more of the instructions individually or collectively by the at least one processor (130), the electronic device (100) can calculate a tilting angle by performing an operation that adds an acceleration value measured using an acceleration sensor (122) and an angular velocity value due to the rotational inertia of the lens of the first camera (111) measured using a gyro sensor (124).

[0202] In one embodiment of the present disclosure, by executing one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) can change the principal point among the intrinsic parameters of the second camera (112) based on information regarding the tilting angle. By executing one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) can determine the location of the cropping area within the entire area of ​​the second image based on the changed principal point.

[0203] In one embodiment of the present disclosure, by executing one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) can acquire the rectification image having a field of view (FOV) corresponding to the field of view (FOV) of the first image based on the focal length among the internal parameters of the second camera (112).

[0204] In one embodiment of the present disclosure, by executing one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) can align the first image and the rectification image horizontally such that the position coordinate values ​​along the Y-axis direction of the pixels that match each other in the first image and the rectification image are the same.

[0205] In one embodiment of the present disclosure, by executing one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) can model lens distortion that changes according to the position of an object and update intrinsic parameters of the first camera (111) and the second camera (112) using the modeled lens distortion model.

[0206] In one embodiment of the present disclosure, by executing one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) can model extrinsic parameters based on a rotation matrix generated by video stabilization and a positional relationship between the first camera (111) and the second camera (112), and update the extrinsic parameters of the first camera (111) and the second camera (112) respectively using the modeled extrinsic parameters.

[0207] In one embodiment of the present disclosure, by executing one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) can convert the color temperature of a rectified image to be equal to the color temperature of a first image using a matching curve that converts the cumulative distribution function of the color temperature, and input the first image and the rectified image with equalized color temperatures into a noise equalization model to equalize the noise of the first image and the rectified image.

[0208] One aspect of the present disclosure provides a method for an electronic device (100) to obtain a stereo image by processing images captured by different types of cameras. The method of operation of the electronic device (100) may include the step (S210) of obtaining a first image using a first camera (111) to which a lens shift type vibration damping structure is applied, and obtaining a second image using a second camera (112) to which a vibration damping structure is not applied. The method of operation of the electronic device (100) may include the step (S220) of determining a crop target area corresponding to the first image captured by the tilted lens of the first camera (111) within the entire area of ​​the second image based on an acceleration value measured using an acceleration sensor (122). The method of operation of the electronic device (100) may include the step (S230) of cropping a crop target area from a second image and resizing the cropped image to obtain a rectified image. The method of operation of the electronic device (100) may include the step (S240) of generating a stereo image using the first image and the rectified image.

[0209] In one embodiment of the present disclosure, the step of determining the crop target area (S220) may include the step of obtaining an acceleration value by measuring the acceleration caused by external force and gravity applied to the electronic device (100) using an acceleration sensor (122), and the step of obtaining information regarding a tilting angle indicating the degree to which the lens of the first camera (111) is tilted based on the acceleration value (S820). The step of determining the crop target area (S220) may include the step of determining the crop target area corresponding to the first image among the entire area of ​​the second image based on the information regarding the tilting angle (S830).

[0210] In one embodiment of the present disclosure, the step of obtaining information regarding the tilting angle (S820) may include the step of calculating the tilting angle by performing an operation to add the acceleration value measured using the acceleration sensor (122) and the angular velocity value due to the rotational inertia of the lens of the first camera (111) measured using the gyro sensor (124).

[0211] In one embodiment of the present disclosure, the step (S830) of determining the crop target area based on the tilting angle may include the step (S1010) of changing the principal point among the intrinsic parameters of the second camera (112) based on information regarding the tilting angle, and the step (S1020) of determining the location of the crop target area among the entire area of ​​the second image based on the changed principal point.

[0212] In one embodiment of the present disclosure, the step of acquiring the rectification image (S230) may include the step of acquiring a rectification image (S1030) having a field of view corresponding to the field of view (FOV) of the first image based on the focal length among the internal parameters of the second camera (112).

[0213] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may further include the step of modeling lens distortion that changes according to the position of an object (S1110), and the step of updating intrinsic parameters of the first camera (111) and the second camera (112) using the modeled lens distortion model (S1120).

[0214] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may further include the step (S1130) of modeling an extrinsic parameter based on a rotation matrix generated by video stabilization and a positional relationship between a first camera (111) and a second camera (112), and the step (S1140) of updating the extrinsic parameter of each of the first camera (111) and the second camera (112) using the modeled extrinsic parameter.

[0215] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may further include the step (S1210) of converting the color temperature of a rectified image to be equal to the color temperature of a first image using a matching curve that converts the cumulative distribution function of the color temperature, and the step (S1220) of inputting the first image and the rectified image, in which the color temperatures have been equalized, into a noise equalization model to equalize the noise of the first image and the rectified image.

[0216] One aspect of the present disclosure provides a computer program product comprising a computer-readable storage medium. The storage medium may include instructions readable by the electronic device (100) for the electronic device (100) to perform the following operations: acquiring a first image using a first camera (111) to which a lens-shift type vibration damping structure is applied, and acquiring a second image using a second camera (112) to which a vibration damping structure is not applied; determining a crop target area corresponding to the first image captured by the tilted lens of the first camera (111) within the entire area of ​​the second image based on an acceleration value measured using an acceleration sensor (122); cropping the crop target area from the second image and resizing the cropped image to acquire a rectified image; and generating a stereo image using the first image and the rectified image.

[0217] One aspect of the present disclosure provides an electronic device (100) for obtaining a stereo image by processing images obtained by different types of cameras. The electronic device (100) of the present disclosure may include a first camera (111) with a lens shift type vibration damping structure, a second camera (112) without a vibration damping structure, an accelerometer (122), at least one processor (130) including a processing circuit, and a memory (140) for storing one or more instructions. By executing one or more of the above commands individually or collectively by the at least one processor (130), the electronic device (100) can acquire a first image using a first camera (111), acquire a second image using a second camera (112), and determine a crop target area among the entire area of ​​the first image captured by the tilted lens of the first camera (111) based on acceleration values ​​measured using an acceleration sensor (122). By executing one or more of the above commands individually or collectively by the at least one processor (130), the electronic device (100) can crop the crop target area from the first image and resize the cropped image to acquire a first rectified image. By executing one or more of the above instructions individually or collectively by the at least one processor (130), the electronic device (100) can crop an area corresponding to the field of view (FOV) of the first rectification image within the entire area of ​​the second image and resize the cropped image to obtain the second rectification image.By executing one or more of the above instructions individually or collectively by the at least one processor (130), the electronic device (100) can generate a stereo image using a first rectification image and a second rectification image.

[0218] In one embodiment of the present disclosure, the electronic device (100) can obtain an acceleration value by measuring the acceleration caused by external force and gravity applied to the electronic device (100) using an acceleration sensor (122) by executing the one or more instructions individually or collectively by the at least one processor (130). The electronic device (100) can obtain information regarding a tilting angle indicating the degree to which the lens of the first camera (111) is tilted, based on the obtained acceleration value. By executing one or more of the above commands individually or collectively by the at least one processor (130), the electronic device (100) can change the principal point among the intrinsic parameters of the first camera (111) based on information regarding the tilting angle, and determine the location of the crop target area among the entire area of ​​the first image based on the changed principal point.

[0219] A program executed by the electronic device (100) described in the present disclosure may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. The program may be executed by any system capable of executing computer-readable instructions.

[0220] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively.

[0221] Software can be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). Computer-readable recording media can be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. The medium is readable by a computer, stored in memory, and can be executed by a processor.

[0222] Computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory' means only that the storage medium does not contain a signal and is tangible, and does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0223] In addition, the program according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product.

[0224] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, the computer program product may be from the manufacturer of the electronic device (100) or an electronic market (e.g., Samsung Galaxy Store). TM It may include a product in the form of a software program that is distributed electronically through ). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the manufacturer of the electronic device (100), a server of an electronic market, or a storage medium of a relay server that temporarily stores the software program.

[0225] A computer program product may include a storage medium of a server or a storage medium of an electronic device (100) in a system composed of an electronic device (100) and / or a server. Alternatively, if there is a third device (e.g., an external device (200), such as a 'head-mounted display device', see FIG. 1a) that is connected to the electronic device (100) through communication, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the electronic device (100) to the third device or from the third device to the electronic device.

[0226] In this case, either the electronic device (100) or one of the third devices may execute a computer program product to perform the method according to the disclosed embodiments. Alternatively, at least one of the electronic device (100) and the third device may execute a computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0227] For example, an electronic device (100) can execute a computer program product stored in memory (140, see FIG. 4) to control another electronic device (e.g., a mobile device) that is connected to the electronic device (100) in communication to perform a method according to the disclosed embodiments.

[0228] As another example, a third device (e.g., an external device (200), such as a 'head-mounted display device', see FIG. 1a) may execute a computer program product to control an electronic device connected to the third device to perform the method according to the disclosed embodiment.

[0229] When the third device executes a computer program product, the third device may download the computer program product from the electronic device (100) and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a pre-loaded state to perform the method according to the disclosed embodiments.

[0230] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, appropriate results can be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or module are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

Claims

1. An electronic device (100) for obtaining a stereo image by processing images obtained by different types of cameras, A first camera (111) with a lens shift type vibration prevention structure applied; A second camera (112) with no vibration damping structure applied; Accelerometer (122); At least one processor (130) including processing circuitry; and Memory (140) for storing one or more instructions; Includes, By executing the above one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) is: A first image is obtained using the first camera (111), and a second image is obtained using the second camera (112). Based on the acceleration value measured using the acceleration sensor (122), a crop target area corresponding to the first image captured by the tilted lens of the first camera (111) is determined among the entire area of ​​the second image, and Cropping the crop target area from the second image and resizing the cropped image to obtain a rectified image, An electronic device (100) that generates a stereo image using the first image and the rectification image.

2. In Paragraph 1, By executing the above one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) is: By using the acceleration sensor (122) above to measure the acceleration caused by external force and gravity applied to the electronic device (100), an acceleration value is obtained, and Based on the above-mentioned acceleration value, information regarding the tilting angle indicating the degree to which the lens of the first camera (111) is tilted is obtained, and An electronic device (100) that determines the crop target area corresponding to the first image among the entire area of ​​the second image based on the information regarding the tilting angle obtained above.

3. In Paragraph 2, 3-axis angular velocity measuring gyro sensor (124); Includes more, By executing the above one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) is: An electronic device (100) that calculates the tilting angle by performing a calculation that adds the acceleration value measured using the acceleration sensor (122) and the angular velocity value due to the rotational inertia of the lens of the first camera (111) measured using the gyroscope sensor (124).

4. In Paragraph 2, By executing the above one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) is: Based on the information regarding the tilting angle above, the principal point among the intrinsic parameters of the second camera (112) is changed, and An electronic device (100) that determines the location of the crop target area within the entire area of ​​the second image based on the above-mentioned changed bar.

5. In any one of paragraphs 1 through 4, By executing the above one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) is: An electronic device (100) that acquires the rectification image having a field of view corresponding to the field of view (FOV) of the first image, based on the focal length among the internal parameters of the second camera (112).

6. In any one of paragraphs 1 through 5, By executing the above one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) is: An electronic device (100) that aligns the first image and the rectification image in a horizontal direction such that the position coordinate values ​​along the Y-axis direction of pixels that match each other in the first image and the rectification image are the same.

7. In any one of paragraphs 1 through 6, By executing the above one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) is: Model lens distortion that changes depending on the object's position, and An electronic device (100) that updates the intrinsic parameters of the first camera (111) and the second camera (112) using the above-modeled lens distortion model.

8. In any one of paragraphs 1 through 7, By executing the above one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) is: Based on the rotation matrix generated by video stabilization and the positional relationship between the first camera (111) and the second camera (112), an extrinsic parameter is modeled, and An electronic device (100) that updates the external parameters of the first camera (111) and the second camera (112), respectively, using the modeled external parameters above.

9. In any one of paragraphs 1 through 8, By executing the above one or more instructions individually or collectively by the at least one processor (130), the electronic device (100) is: Using a matching curve that transforms the cumulative distribution function of the color temperature, the color temperature of the rectification image is transformed to be uniform with the color temperature of the first image, and An electronic device (100) that inputs the first image and the rectification image, in which the color temperature is equalized, into a noise equalization model to equalize the noise of the first image and the rectification image.

10. A method for an electronic device (100) to process images captured by different types of cameras to obtain a stereo image, A step (S210) of acquiring a first image using a first camera (111) to which a lens shift type vibration damping structure is applied, and acquiring a second image using a second camera (112) to which a vibration damping structure is not applied; A step (S220) of determining a crop target area corresponding to the first image captured by the tilted lens of the first camera (111) within the entire area of ​​the second image based on acceleration values ​​measured using an acceleration sensor (122); A step (S230) of cropping the crop target area from the second image and resizing the cropped image to obtain a rectified image; and A step (S240) of generating a stereo image using the first image and the rectification image; A method including 11. In Paragraph 10, The step (S220) of determining the crop target area above is, A step (S810) of obtaining an acceleration value by measuring the acceleration caused by external force and gravity applied to the electronic device (100) using the acceleration sensor (122); A step (S820) of obtaining information regarding a tilting angle indicating the degree to which the lens of the first camera (111) is tilted based on the above-mentioned acceleration value; and Based on the information regarding the tilting angle obtained above, a step (S830) of determining the crop target area corresponding to the first image among the entire area of ​​the second image; A method including 12. In Paragraph 10, The step (S820) of obtaining information regarding the above tilting angle is, A step of calculating the tilting angle by performing an operation to add the acceleration value measured using the acceleration sensor (122) and the angular velocity value due to the rotational inertia of the lens of the first camera (111) measured using the gyro sensor (124); A method including 13. In any one of paragraphs 10 through 12, The step (S830) of determining the crop target area based on the tilting angle is, A step (S1010) of changing the principal point among the intrinsic parameters of the second camera (112) based on the information regarding the tilting angle above; and A step (S1020) of determining the location of the crop target area within the entire area of ​​the second image based on the above-mentioned modified bar; A method including 14. In any one of paragraphs 10 through 13, The step (S230) of acquiring the above-mentioned rectification image is, A step (S1030) of acquiring the rectification image having a field of view (FOV) corresponding to the field of view (FOV) of the first image, based on the focal length among the internal parameters of the second camera (112); A method including 15. In any one of paragraphs 10 through 14, Step of modeling lens distortion that changes depending on the position of the object (S1110); and A step (S1120) of updating the intrinsic parameters of the first camera (111) and the second camera (112) using the above-modeled lens distortion model; A method that further includes.

Images