Electronic device and control method therefor

Abstract

This electronic device comprises a camera, at least one processor, and a memory for storing instructions, wherein, when executed individually or collectively by the at least one processor, the instructions instruct the electronic device to: acquire first internal parameters related to a lens of a projector projecting an image and second internal parameters related to the camera; using the camera to capture a pattern image projected onto a projection surface by the projector; determine pose information about a pose of the electronic device and projection surface information about the projection surface on the basis of the first internal parameters, the second internal parameters and the captured pattern image; acquire, on the basis of the projection surface information, virtual viewpoint information about a virtual viewpoint positioned in front of the projection surface; and generate keystone correction information on the basis of the virtual viewpoint information, wherein the keystone correction information is used to perform keystone correction on at least one image projected onto the projection surface by the projector.

Description

Electronic device and control method thereof

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more specifically, to an electronic device and a method for controlling the same capable of performing keystone correction on an image projected by an external projector.

[0002] A projector refers to an electronic device that projects light onto or toward a projection surface (or screen) to form an image on it. Additionally, mobile electronic devices (e.g., robots) may include projectors. Such mobile electronic devices provide various information to users by utilizing projectors while moving through specific spaces (e.g., homes, restaurants, airports, etc.).

[0003] When using portable projectors, keystone correction can help project images onto the projection surface from various angles. In some methods, the projector can perform keystone correction directly; however, this requires various sensors and suffers from the problem of requiring a relatively large amount of computing resources due to the high computational load.

[0004] Therefore, it is necessary to explore a method to perform keystone correction of the image projected by the projector using other electronic devices.

[0005] According to one embodiment of the present disclosure, an electronic device comprises: a camera; at least one processor; and a memory for storing instructions; wherein, when the instructions are executed individually or collectively by the at least one processor, the electronic device obtains a first intrinsic parameter related to a lens of a projector that projects an image and a second intrinsic parameter related to the camera, captures a pattern image projected onto a projection surface by the projector using the camera, determines pose information for a pose of the electronic device and projection surface information for the projection surface based on the first intrinsic parameter, the second intrinsic parameter and the captured pattern image, obtains virtual viewpoint information for a virtual viewpoint located in front of the projection surface based on the projection surface information, and generates keystone correction information based on the virtual viewpoint information, and the keystone correction information is used to perform keystone correction on at least one image projected onto a projection surface by the projector.

[0006] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may determine a central pixel between a plurality of patterns included in the pattern image and a pattern pixel corresponding to the plurality of patterns using the Fundamental Matrix, and determine pixel matching information between the pattern pixel and the central pixel using the Fundamental Matrix.

[0007] The pose information and the projection plane information can be determined so that the pattern pixel is located on a plane based on the pixel matching information.

[0008] The above virtual viewpoint may be a viewpoint located at a predetermined distance from the projection plane in the frontal direction.

[0009] The virtual viewpoint above may be a viewpoint located at a preset distance from the center pixel.

[0010] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may determine projection area information of the pattern image seen at the virtual viewpoint based on the pose information and the projection surface information, determine a rectangular target projection area within the projection area, and determine the keystone correction information based on the projection area information and the target projection area.

[0011] The size of the target projection area, the projection ratio, and the projection direction can be set according to user input received through the UI.

[0012] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may take multiple pattern images at multiple camera positions using the camera to acquire the plurality of pattern images, and acquire the second intrinsic parameter using triangulation by comparing the position of a specific point within the multiple captured images with the multiple camera positions.

[0013] The first internal parameter above includes information regarding the focal length and principal point of the lens, and the second internal parameter may include information regarding the focal length and principal point of the camera.

[0014] A control method for an electronic device according to one embodiment of the present disclosure comprises: acquiring a first intrinsic parameter related to a lens of a projector that projects an image and a second intrinsic parameter related to a camera of the electronic device; capturing a pattern image projected onto a projection surface by the projector using the camera; determining pose information for a pose of the electronic device and projection surface information for the projection surface based on the first intrinsic parameter, the second intrinsic parameter, and the captured pattern image; acquiring virtual viewpoint information for a virtual viewpoint located in front of the projection surface based on the projection surface information; and generating keystone correction information based on the virtual viewpoint information; wherein the keystone correction information is used to perform keystone correction on at least one image projected by the projector onto a projection surface.

[0015] The step of determining the pose information and the projection plane information may include: a step of determining a center pixel between a plurality of patterns included in the pattern image and a pattern pixel corresponding to the plurality of patterns using a Fundamental Matrix; and a step of determining pixel matching information between the pattern pixel and the center pixel using the Fundamental Matrix.

[0016] The pose information and the projection plane information can be determined so that the pattern pixel is located on a plane based on the pixel matching information.

[0017] The above virtual viewpoint may be a viewpoint located at a predetermined distance from the projection plane in the frontal direction.

[0018] The virtual viewpoint above may be a viewpoint located at a preset distance from the center pixel.

[0019] The step of generating the keystone correction may include determining projection area information of the pattern image visible from the virtual viewpoint based on the pose information and the projection plane information, determining a rectangular target projection area within the projection area, and determining the keystone correction information based on the projection area information and the target projection area.

[0020] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description, which is described together with reference to the accompanying drawings.

[0021] FIG. 1 is a drawing illustrating an image projection system including an electronic device and a projector according to one embodiment of the present disclosure.

[0022] FIG. 2 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure,

[0023] FIG. 3 is a block diagram showing the configuration of a projector according to one embodiment of the present disclosure,

[0024] FIG. 4 is a sequence diagram illustrating an embodiment in which an electronic device obtains keystone correction information using a pattern image projected by a projector, according to one embodiment of the present disclosure.

[0025] FIG. 5 is a drawing illustrating a pattern image projected by a projector according to one embodiment of the present disclosure.

[0026] FIG. 6 is a drawing illustrating a pattern image captured by an electronic device according to one embodiment of the present disclosure.

[0027] FIGS. 7a to 7c are drawings for illustrating an embodiment of acquiring pixel matching information based on various poses of an electronic device according to one embodiment of the present disclosure.

[0028] FIG. 8 is a drawing for explaining a virtual viewpoint according to one embodiment of the present disclosure,

[0029] FIG. 9 is a drawing for illustrating a projection area and a target projection area according to one embodiment of the present disclosure,

[0030] FIGS. 10a to 10d are drawings for illustrating embodiments of setting various target projection areas according to user input according to various embodiments of the present disclosure,

[0031] FIGS. 11a and FIGS. 11b are drawings for explaining the effect of an image projected through keystone correction according to one embodiment of the present disclosure,

[0032] FIG. 12 is a drawing for explaining an embodiment of acquiring internal parameters of an electronic device by capturing pattern images in a plurality of regions according to one embodiment of the present disclosure.

[0033] FIG. 13 is a flowchart illustrating an embodiment in which an electronic device projects an image by performing keystone correction according to one embodiment of the present disclosure, and,

[0034] FIG. 14 is a sequence diagram illustrating an embodiment in which an electronic device performs keystone correction and transmits a keystone-corrected image to a projector, according to one embodiment of the present disclosure.

[0035] The embodiments described herein are subject to various modifications and may have various forms; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals may be used for similar components.

[0036] To avoid unnecessarily obscuring the content of the disclosure, overly detailed descriptions of some elements may be omitted.

[0037] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concept of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to fully convey the technical concept of the present disclosure to those skilled in the art.

[0038] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of the rights. The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0039] In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0040] In the present disclosure, expressions such as “A or B,” “at least one of A and B,” “one or more of A and B,” “at least one of A or B,” “at least one of A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0041] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0042] Where it is stated that a certain component (e.g., a first component) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., a second component), it should be understood that the said certain component may be directly connected to the said other component or connected through another component (e.g., a third component).

[0043] On the other hand, when it is stated that a certain component (e.g., a first component) is "directly connected" or "directly coupled" to another component (e.g., a second component), it may be understood that no other component (e.g., a third component) exists between said certain component and said other component.

[0044] As used in this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware.

[0045] Instead, in some situations, the expression “device configured to do something” may mean that the device is “capable of doing something” together with other devices or components. For example, the phrase “processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing those operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or application processor) capable of performing those operations by executing one or more software programs stored in a memory device.

[0046] In the embodiments, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor, except for a "module" or "part" that needs to be implemented in specific hardware.

[0047] Meanwhile, various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0048] FIG. 1 is a drawing illustrating an image projection system including an electronic device and a projector according to one embodiment of the present disclosure. As shown in FIG. 1, the image projection system may include an electronic device (100) and a projector (200). For example, the electronic device (100) may be implemented as a smartphone, but this is merely one embodiment, and the electronic device (100) may be implemented as various terminal devices capable of capturing the projection surface (10), such as a tablet PC, a wearable device, etc. The projector (200) is a device capable of projecting an image onto the projection surface (10), and may be a mobile projector, but this is merely one embodiment, and the projector may be implemented as various devices such as a robot.

[0049] The projector (200) can project an image toward the projection surface (10). For example, the projector (200) is a movable device, and when the projector (200) moves, if an image is projected toward the projection surface (10) at the moved position, a phenomenon of image distortion may occur if the moved position is not in the front direction (e.g., forward) of the projection surface (10). According to one embodiment, the front direction of the projection surface (10) may mean a direction facing the front of the projection surface (10), and may also be referred to as the forward direction relative to the projection surface (10). To resolve this phenomenon, the projector (200) may allow the user to view a rectangular image through keystone correction. According to one embodiment, keystone correction may be a process of correcting distortion that occurs when the projection surface is not properly aligned with the projector (e.g., not completely perpendicular to the projector). For example, if an image is projected onto a specific projection surface that is not properly aligned with the projector, the image may appear distorted when viewed from a specific point in time. Therefore, keystone correction can be performed by adjusting the shape of the projected image so that the distorted image appears as a rectangular shape (or other desired shape) when viewed from a specific point in time.

[0050] According to the present invention, in order to perform keystone correction, the projector (200) may obtain information for keystone correction (hereinafter referred to as keystone correction information) by linking with (or together with) the electronic device (100).

[0051] For example, the projector (200) can project a pattern image toward the projection surface (10). For example, the pattern image may be an image containing at least one pattern at a preset position, as shown in FIG. 1.

[0052] The electronic device (100) can capture a pattern image projected toward the projection surface (10) at a random or arbitrary location. Then, the electronic device (100) can obtain pose information of the electronic device (100) and information about the projection surface (10) (hereinafter referred to as projection surface information) using the first and second intrinsic parameters and the captured pattern image. For example, the intrinsic parameters may be parameters regarding the focal length or principal point of a lens. Additionally, the pose information of the electronic device (100) may include information regarding the position and orientation (or direction) of the electronic device (100) within a three-dimensional space. For example, the pose information may refer to information regarding the six-axis direction. Based on the pose information of the electronic device (100) and the information regarding the projection surface (10), the electronic device (100) can obtain coordinate information regarding the three-dimensional space where the image projection system is located.

[0053] The electronic device (100) can identify or determine a virtual viewpoint located at a predetermined distance from the projection surface (10) in the frontal direction, and obtain keystone correction information that makes the projected image at the virtual viewpoint appear as a rectangle.

[0054] The electronic device (100) can transmit the acquired keystone correction information toward the projector (200), and the projector (200) can correct the projected image using the keystone correction information.

[0055] For example, the method by which the electronic device (100) obtains keystone correction information will be explained in more detail later.

[0056] As described above, by the electronic device (100) analyzing the pattern image projected by the projector (200) and obtaining keystone correction information, the projector (200) can perform keystone correction in conjunction with the electronic device (100) without using a separate sensor.

[0057] FIG. 2 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure. As shown in FIG. 2, the electronic device (100) may include a communication interface (110), a memory (120), an operation interface (130), a display (140), a camera (150), a sensor (160), and a processor (170). Meanwhile, the configuration shown in FIG. 2 is merely an example of various embodiments, and some configurations may be omitted and new configurations may be added.

[0058] The communication interface (110) can communicate with various external devices. For example, the communication interface (110) can receive information about the first internal parameter of the projector (200) or identification information of the projector (200) from the projector (200). Also, the communication interface (110) can transmit keystone correction information or a keystone-corrected image (e.g., an image on which keystone correction has been performed) to the projector (200).

[0059] For example, the communication interface (110) can communicate with various external devices using various wireless communication technologies or mobile communication technologies. Such wireless communication technologies may include, for example, Bluetooth, Bluetooth Low Energy, CAN communication, Wi-Fi, Wi-Fi Direct, ultrawide band communication (UWB), Zigbee, infrared data association (IrDA), or near field communication (NFC), and mobile communication technologies may include 3GPP, Wi-Max, LTE (Long Term Evolution), 5G, etc.

[0060] The memory (120) may store an operating system (OS) for controlling the overall operation of the components of the electronic device (100) and instructions or data related to the components of the electronic device (100). For example, the memory (120) may include various modules for obtaining keystone correction information. For example, when an event to obtain keystone correction information occurs, the electronic device (100) may load data for various modules stored in non-volatile memory to perform various operations into volatile memory. For example, loading means the operation of bringing data stored in non-volatile memory into volatile memory and storing it so that the processor (170) can access it.

[0061] According to one embodiment, the memory (120) may be implemented as non-volatile memory (e.g., hard disk, SSD (Solid state drive), flash memory), volatile memory (memory within the processor (170)), etc.

[0062] For example, the memory (120) can store information about a second internal parameter of the electronic device (100). Additionally, the memory (120) can store information about a first internal parameter of the projector (200) received from an external source (e.g., outside the electronic device (100)).

[0063] The operation interface (130) may include a button, a lever, a switch, a touch interface, a microphone, etc. For example, the touch interface may be implemented in a way that receives input by touching the screen of the display (140) of the electronic device (100). For example, the operation interface (130) may receive user input to obtain keystone correction information.

[0064] The display (140) can provide various information. For example, the display (140) can provide a UI for setting a target projection area. Additionally, the display (120) can provide information about the captured pattern image.

[0065] A camera (150) is configured to capture a subject and generate an image, wherein the image may include both video and still images. According to one embodiment, the “image” of the present disclosure may include both an image output on a display (140) and an image frame captured by the camera (150). For example, the camera (150) may capture a pattern image projected by a projector (200).

[0066] The sensor (160) can generate an electrical signal or data value corresponding to an internal operating state of the electronic device (100) or an external environmental state. The sensor (160) may include, for example, at least one of a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor. For example, the sensor (160) can store information about the movement and posture of the electronic device (100) using an accelerometer sensor, a gyroscope sensor, etc.

[0067] The processor (170) can control the electronic device (100) according to at least one instruction stored in memory (120).

[0068] For example, the processor (170) may include one or more processors. Specifically, one or more processors may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator. One or more processors may control one or any combination of other components of an electronic device and may perform operations or data processing related to communication. One or more processors may execute one or more programs or instructions stored in memory. For example, one or more processors may perform a method according to one embodiment of the present disclosure by executing one or more instructions stored in memory.

[0069] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. That is, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor). For example, according to one embodiment of the present disclosure, operations such as identifying or determining corners within a handwriting image or correcting spaces within a handwriting image using a neural network model may be performed by a processor that performs parallel operations, such as a GPU or an NPU, and operations such as generating / editing a plan view image or post-processing operations may be performed by a general-purpose processor, such as a CPU.

[0070] One or more processors may be implemented as a single-core processor comprising one core, or as one or more multicore processors comprising multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When one or more processors are implemented as multicore processors, each of the multiple cores included in the multicore processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by multiple cores may be included in the multicore processor. Additionally, each of the multiple cores included in the multicore processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one embodiment of the present disclosure.

[0071] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0072] In embodiments of the present disclosure, the processor (170) may mean a system-on-chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator or machine learning accelerator, etc., but the embodiments of the present disclosure are not limited thereto.

[0073] For example, the processor (170) obtains a first internal parameter related to the lens of the projector (200) that projects an image and a second internal parameter related to the camera of the electronic device (100) by executing at least one instruction stored in the memory (150), captures a pattern image projected onto a projection surface by the projector (200) through the camera (150), identifies or determines pose information and projection surface information of the electronic device (100) based on the first internal parameter, the second internal parameter, and the captured pattern image, obtains information about a virtual viewpoint located in front of the projection surface based on the projection surface information, and obtains keystone correction information based on the information about the virtual viewpoint. In one embodiment, the information about the virtual viewpoint may be referred to as virtual viewpoint information.

[0074] In one or more embodiments, the processor (170) may use the Fundamental Matrix to identify or determine pixel matching information between a pattern pixel corresponding to a plurality of patterns included in a pattern image projected by a projector and a center pixel located between the plurality of patterns, and between a pattern pixel and a center pixel captured by a camera. For example, the fundamental matrix may mean a matrix that connects corresponding points in a pair of images, but the embodiments are not limited thereto. According to some embodiments, the fundamental matrix may be a 3x3 matrix that encodes the intrinsic projective geometry between two viewpoints (e.g., images of the same scene captured at different camera positions) and maps a point in one image to a corresponding epipolar line in another image.

[0075] In one or more embodiments, the processor (170) may identify or determine pose information and projection plane information of an electronic device in which pattern pixels are located on a plane based on pixel matching information. For example, the processor (170) may determine or select pose information and projection plane information such that pattern pixels are located on a plane on a projection plane. This may mean, for example, that the processor (170) may find, identify, select, or determine a plane in which pattern pixels are located (e.g., a location containing all pattern pixels), and select pose information and projection plane information based on this plane (e.g., so that the projection plane includes or coincides with the plane).

[0076] In one or more embodiments, the processor (170) may identify or determine a virtual viewpoint at a point in time located at a predetermined distance in the front direction from a projection plane on which a pattern pixel is located on a plane. For example, the virtual viewpoint may be a point in time located at a predetermined distance from a center pixel.

[0077] In one or more embodiments, the processor (170) may identify or determine information regarding the projection area of ​​a pattern image projected by a projector (200) visible from a virtual viewpoint based on pose information and projection surface information of an electronic device (100), identify or determine a rectangular target projection area within the projection area, and obtain keystone correction information based on the information regarding the projection area and the target projection area. In one embodiment, the information regarding the projection area may also be referred to as projection area information. For example, the target projection area may have its size, projection ratio, and projection direction set according to user input through a UI (e.g., a UI provided through a display (140)).

[0078] In one or more embodiments, the processor (170) can obtain a plurality of captured pattern images by using a camera (150) to capture pattern images from a plurality of camera positions multiple times, and can obtain a second internal parameter through triangulation by comparing the location of a specific point within the plurality of captured pattern images with the plurality of camera positions of the camera (150).

[0079] FIG. 3 is a block diagram showing the configuration of a projector according to one embodiment of the present disclosure. As shown in FIG. 3, the projector (200) may include at least one of a processor (211), a projection module (212), a memory (213), a communication interface (214), an operation interface (215), an input / output interface (216), a speaker (217), a microphone (218), a power module (219), a driving module (220), or a sensor (221). For example, the configuration shown in FIG. 3 is merely an example of various embodiments, and some configurations may be omitted and new configurations may be added.

[0080] The projection module (212) is configured to project an image outward. According to various embodiments of the present disclosure, the projection module (212) can be implemented using various projection methods (e.g., cathode-ray tube (CRT) method, liquid crystal display (LCD) method, digital light processing (DLP) method, laser method, etc.).

[0081] In one embodiment, the projection module (212) can perform various functions to adjust the output image under the control of the processor (211). For example, the projection module (212) can perform functions such as zoom, keystone correction, quick corner keystone (e.g., 4-corner keystone correction), lens shift, etc.

[0082] In particular, the projection module (212) can perform a keystone correction function. If the height is not correct for frontal projection, the screen may be distorted upward or downward. The keystone correction function can be described as a function that corrects distortion of the projected image (e.g., an image projected onto a screen or another projection surface). As another example, if distortion occurs in the left-right direction of the projected image, it can be corrected using horizontal keystone, and if distortion occurs in the up-down direction, it can be corrected using vertical keystone. The quick corner keystone correction function (e.g., 4-corner keystone correction) is a function that corrects the screen when the central area of ​​the screen is normal but the corner areas are unbalanced. The lens shift function is a function that moves the projected image as is when the screen is off the screen.

[0083] In one embodiment, the projector module (212) may include a first projection lens corresponding to a first focal length and a second projection lens corresponding to a second focal length. For example, the first projection lens may be a long-focus projection lens and the second projection lens may be an ultra-short-focus projection lens, but is not limited thereto. A long-focus projection lens is a lens capable of projecting a large screen from a long distance, and can project an image from a position several meters to tens of meters away. An ultra-short-focus projection lens is a lens capable of projecting a large screen from a very short distance, and can project an image from a position several tens of centimeters away. Additionally, the projector unit (212) may further include a switching module capable of switching (or changing, tilting) between the long-focus projection lens and the ultra-short-focus projection lens. That is, the projection module (212) can switch between the long-focus projection lens and the ultra-short-focus projection lens using the switching module under the control of the processor (211).

[0084] The memory (213) may be implemented as internal memory such as ROM (e.g., EEPROM (electrically erasable programmable read-only memory)) or RAM included in the processor (211), or as memory separate from the processor (211). In this case, the memory (213) may be implemented as memory embedded in the projector (200) or as memory that can be attached to the projector (200), depending on the purpose of data storage. For example, data for operating the projector (200) may be stored in memory embedded in the projector (200), and data for the expansion function of the projector (200) may be stored in memory that can be attached to the projector (200).

[0085] The memory (213) may be configured to store or store at least one instruction regarding the projector (200). Additionally, the memory (213) may be configured to store or store an operating system (O / S) for operating the projector (200). Furthermore, the memory (213) may store various software programs or applications for operating the projector (200) according to various embodiments of the present disclosure.

[0086] The memory (213) can store information about the first internal parameter of the projector (200).

[0087] The communication interface (214) is a configuration that performs communication with various types of external devices according to various types of communication methods. The communication interface (214) may include at least one of a wireless communication module and a wired communication module. For example, each communication module may be implemented in the form of at least one hardware chip.

[0088] For example, the communication interface (214) can communicate with an external device. The communication interface (214) can transmit information about the first internal parameter to the electronic device (100) and can receive at least one of keystone correction information or a keystone corrected image (e.g., an image on which keystone correction has been performed) from the electronic device (100).

[0089] The operation interface (215) may include various types of input devices. For example, the operation interface (215) may include a physical button. For example, the physical button may include a function key, a directional key (e.g., a 4-way key), or a dial button. Depending on various embodiments, the physical button may be implemented as a plurality of keys. Depending on other embodiments, the physical button may be implemented as a single key. For example, if the physical button is implemented as a single key, the projector (200) may receive user input when the single key is pressed longer than a predetermined threshold time or when it is pressed more than a predetermined threshold number of times within a predetermined threshold time.

[0090] The input / output interface (216) is configured to input or output at least one of an audio signal and an image signal. The input / output interface (216) can receive at least one of the audio and image signals from an external device and can output control commands to the external device.

[0091] In one embodiment, in various embodiments of the present disclosure, the input / output interface (216) may be implemented as at least one wired input / output interface among HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), USB C-type, DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), and DVI (Digital Visual Interface). According to various embodiments, the wired input / output interface may be implemented as an interface that inputs and outputs only audio signals and an interface that inputs and outputs only image signals, or as a single interface that inputs and outputs both audio signals and image signals.

[0092] The speaker (217) is a component that outputs an audio signal. For example, the speaker (217) may include an audio output mixer, an audio signal processor, and an audio output module. The audio output mixer may synthesize multiple audio signals to be output into at least one audio signal. For example, the audio output mixer may synthesize an analog audio signal and another analog audio signal (e.g., an analog audio signal received from an external source) into at least one analog audio signal. The audio output module may include a speaker or an output terminal. According to various embodiments, the audio output module may include multiple speakers, in which case the audio output module may be placed inside the main body, and the sound radiated by covering at least a part of the diaphragm of the audio output module may pass through a waveguide and be transmitted to the outside of the main body. The audio output module may include multiple audio output units, and by symmetrically arranging the multiple audio output units on the exterior of the main body, sound may be radiated in all directions, that is, in all 360 degrees.

[0093] The microphone (218) is a component for receiving user voice or other sounds and converting them into audio data. The microphone (218) can receive the user's voice when active. For example, the microphone (218) may be formed integrally on the upper side, front side, or side side of the projector (200). The microphone (218) may include various components such as a microphone for collecting analog user voice, an amplifier circuit for amplifying the collected user voice, an A / D conversion circuit for sampling the amplified user voice and converting it into a digital signal, and a filter circuit for removing noise components from the converted digital signal.

[0094] The power module (219) can receive power from an external source and supply power to various components of the projector (200). The power module (219) according to various embodiments of the present disclosure can receive power through various methods.

[0095] The driving module (220) can drive at least one hardware configuration or component included in the projector (200). The driving module (220) can generate physical force and transmit it to at least one hardware configuration or component included in the projector (200). For example, the driving module (220) can generate driving power for a movement operation of the hardware configuration included in the projector (200) (e.g., movement of the projector (200)).

[0096] For example, the driving module (220) can move the position of the projector (200). For example, the driving module (220) can control a moving member to move the projector (200). For example, the driving module (220) can control the moving member using a motor and wheels. In one embodiment, according to the present disclosure, the driving module (220) may be referred to by various terms such as a driving unit, a driving unit, a moving unit, etc.

[0097] The sensor (221) may include at least one sensor. For example, the sensor (221) may include at least one of a tilt sensor that senses the tilt of the projector (200) and an image sensor that captures an image. For example, the tilt sensor may be an accelerometer or a gyroscope, and the image sensor may refer to a camera or a depth camera. In one embodiment, the tilt sensor may be described as a motion sensor. Additionally, the sensor (221) may include various sensors other than the tilt sensor or the image sensor. For example, the sensor (221) may include an illuminance sensor or a distance sensor. The distance sensor may be a Time of Flight (ToF). Additionally, the sensor (121) may include a LiDAR sensor.

[0098] The processor (211) may be implemented as a digital signal processor (DSP) that processes digital signals, a microprocessor, or a time controller (TCON). However, it is not limited thereto, and may include or be defined by one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an ARM (advanced reduced instruction set computer (RISC) machine) processor. Additionally, the processor (211) may be implemented as a System on Chip (SoC) or large scale integration (LSI) with a built-in processing algorithm, or may be implemented in the form of a Field Programmable Gate Array (FPGA). Furthermore, the processor (211) can perform various functions by executing computer executable instructions stored in memory (213).

[0099] For example, the processor (211) can perform keystone correction on an image based on keystone correction information received from an external electronic device (100). Then, the processor (211) can control the projection module (212) to project the keystone-corrected image toward a projection surface.

[0100] FIG. 4 is a sequence diagram illustrating an embodiment in which an electronic device obtains keystone correction information using a pattern image projected by a projector according to one embodiment of the present disclosure. The projector (200) of the present disclosure may be a mobile projector, but is not limited thereto, and may be implemented as various projector devices such as a fixed projector, a robot, etc.

[0101] In the following embodiments, each operation may be performed sequentially, but this is merely one embodiment, and in other embodiments, they may not be performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0102] According to one or more embodiments, S410 to S490 may be understood to be performed in a processor (e.g., processor (170) of FIG. 2) of an electronic device (e.g., electronic device (100) of FIG. 2) and a processor (e.g., processor (211) of FIG. 3) of a projector (e.g., projector (200) of FIG. 3).

[0103] First, the projector (200) can transmit information regarding the first internal parameter to the electronic device (100) (operation S410). For example, the first internal parameter may include information regarding the focal length and principal point of the lens for which the projector (200) projects an image. The focal length may be the distance from the center of the projector lens to the focal point of the projected image (e.g., the point where the image is clearly formed). The principal point refers to the point where the optical axis of the projector lens meets the projection plane, and the position of the image in the horizontal and vertical directions can be determined as the center of the projected image through the principal point. In one or more embodiments, the projector (200) may obtain information regarding the first internal parameter from the previously stored specification information for the projector (200) and then transmit information regarding the first parameter to the electronic device (100). However, not limited thereto, the projector (200) may transmit identification information (e.g., product name, product number, etc.) for the projector (200) to the electronic device (100), and the electronic device (100) may obtain information about the first internal parameter of the projector (200) from an external server based on the identification information for the projector (200).

[0104] The electronic device (100) can obtain a second internal parameter (operation S420). The focal length may be the distance from the optical center of the camera lens to the image sensor (or film). The principal point is the point where the optical axis of the camera meets the image sensor, and may represent the center of the image. In one or more embodiments, the electronic device (100) may obtain information regarding the second internal parameter from the previously stored specification information for the electronic device (100). In one or more embodiments, the electronic device (100) may take multiple images projected by the projector (200) and obtain the second internal parameter through triangulation by comparing the multiple captured images. The above-described embodiments will be explained in more detail later with reference to the drawings.

[0105] The projector (200) can project a pattern image (operation S430). For example, the pattern image may be an image having multiple patterns. In one or more embodiments, the projector (200) may project a pattern image (510) comprising multiple pattern objects having geometric shapes, as shown in FIG. 5. For example, the multiple pattern objects may be placed at a predetermined distance from the center of the pattern image. However, it is not limited thereto and may be implemented as a pattern image of various forms, such as a checkerboard image.

[0106] The electronic device (100) can capture a pattern image projected onto a projection surface to obtain a captured pattern image (operation S440). For example, the electronic device (100) can capture the pattern image at a random or arbitrary point other than the front of the projection surface. Thus, the captured pattern image may be a rectangle (610) having a distorted shape, as shown in FIG. 6, rather than a rectangular shape.

[0107] The electronic device (100) can obtain pose information and projection plane information of the electronic device (100) (operation S450). For example, the pose information of the electronic device (100) may include information (or position information) and attitude (or direction) of the electronic device (100) in three-dimensional space. For example, the position information may indicate the position on the x-axis, y-axis, and z-axis where the electronic device (100) is located in three-dimensional space, and the attitude information may indicate information about the pitch angle rotated around the x-axis, information about the yaw rotated around the y-axis, and information about the roll angle rotated around the z-axis.

[0108] In one embodiment, the electronic device (100) can obtain pose information and projection plane information of the electronic device (100) based on first and second intrinsic parameters and a captured pattern image. For example, the electronic device (100) can use a Fundamental Matrix to identify or determine pixel matching information between a pattern pixel corresponding to a plurality of patterns included in a pattern image projected by a projector (200), a center pixel located between the plurality of patterns, and a pattern pixel and a center pixel captured by the camera of the electronic device (100). For example, the Fundamental Matrix is ​​a 3x3 matrix based on the intrinsic parameters and extrinsic parameters of two cameras, which can identify information regarding the relative position and orientation (e.g., rotation and translation) of the cameras. The intrinsic parameters and extrinsic parameters extracted from the Fundamental Matrix can be expressed as shown in Equation 1 below.

[0109]

[0110] In mathematical formula 1, can represent the two-dimensional pixel coordinates of a camera included in an electronic device (100), and can represent internal parameters, and can represent external parameters, and It can represent three-dimensional spatial coordinates. For example, the external parameter can represent pose information of the electronic device (100).

[0111] In one or more embodiments, the electronic device (100) can use a Fundamental Matrix to identify or determine pixel matching information between a pattern pixel corresponding to a plurality of patterns included in a pattern image projected by a projector (200), a center pixel located between the plurality of patterns, and a pattern pixel and a center pixel captured by a camera of the electronic device (100). For example, the pixel matching information may include information about coordinates where the pixels of the pattern image projected by the projector (200) and the pixels captured by the camera of the electronic device (100) match each other.

[0112] For example, since the electronic device (100) cannot know the pose information of the electronic device (100), it can obtain pixel matching information between the pixels projected by the projector (200) and the pixels captured by the electronic device (100) according to the pose information of various electronic devices (100). Additionally, the electronic device (100) can identify or determine pixel matching information for the pixel corresponding to a plurality of patterns among the pixels and the pixel of the center pixel located between the plurality of patterns. Furthermore, the electronic device (100) can identify or determine the pose information of the electronic device (100) in which the pattern pixels are located on one plane based on the pixel matching information. In another embodiment, the electronic device (100) can identify or determine the pose information of the electronic device (100) in which the pattern pixels and the center pixel are located on one plane by further utilizing the pixel matching information of the center pixel in addition to the pixel matching information for the pattern pixels.

[0113] In one or more embodiments, the electronic device (100) can identify or determine pixel matching information using a random or arbitrary first pose of the electronic device (100). Additionally, the electronic device (100) can identify or determine pose information of the electronic device (100) such that the pattern pixels are located on one plane by adjusting the pose of the electronic device (100) so that the position where the pattern pixels corresponding to at least three patterns are matched is located on a plane.

[0114] In one or more embodiments, the electronic device (100) can identify or determine pixel matching information using a random or arbitrary first pose of the electronic device (100). Additionally, the electronic device (100) can identify or determine pose information of the electronic device (100) such that the pattern pixels (720-1, 720-2) and the center pixel (720-3) are located on a single plane by adjusting the pose of the electronic device (100) so that the position where the pattern pixels corresponding to a plurality of patterns and the center pixel located between the plurality of patterns are matched is located on a plane.

[0115] For example, the electronic device (100) can identify or determine pixel matching information of pattern pixels and center pixels based on pose information of a plurality of electronic devices (100), as shown in FIGS. 7a to 7c. For example, as shown in FIG. 7a, when the pose of the electronic device (100) is a first pose, the electronic device (100) can identify or determine pixel matching information of pattern pixels (710-1, 710-2) corresponding to a plurality of patterns and center pixels (710-3) located between the plurality of patterns. For example, since the center pixel (710-3) is located in front of the pattern pixels (710-1, 710-2), the electronic device (100) can identify or determine that the pattern pixels (710-1, 710-2) and the center pixel (710-3) are not located on the same plane. And, as shown in FIG. 7b, the electronic device (100) can modify the pose of the electronic device (100) to a second pose. And, the electronic device (100) can identify or determine pixel matching information regarding pattern pixels (720-1, 720-2) corresponding to a plurality of patterns and a center pixel (720-3) located between the plurality of patterns. Also, since the center pixel (720-3) is located in front of the pattern pixels (720-1, 720-2), the electronic device (100) can identify or determine that the pattern pixels (720-1, 720-2) and the center pixel (720-3) are not located on a single plane. And, as shown in FIG. 7c, the electronic device (100) can modify the pose of the electronic device (100) to a second pose. And, the electronic device (100) can identify or determine pixel matching information of pattern pixels (730-1, 730-2) corresponding to a plurality of patterns and a center pixel (730-3) located between the plurality of patterns. At this time, the electronic device (100) can identify or determine that the pattern pixels (720-1, 720-2) and the center pixel (720-3) are located on a single plane.

[0116] Accordingly, the electronic device (100) can identify or determine information regarding the third pose as pose information of the electronic device (100) in which the pattern pixels (720-1, 720-2) and the center pixel (720-3) are located on a single plane. Additionally, when the electronic device (100) is in the third pose, the electronic device (100) can identify or determine projection plane information based on information regarding the single plane in which the pattern pixels (720-1, 720-2) and the center pixel (720-3) are located. For example, the projection plane information may include information regarding the angle at which the projection plane is tilted in three-dimensional space, the size of the projection plane, and the position of the pattern and center point within the projection plane.

[0117] For example, the electronic device (100) can identify or determine pose information and projection plane information of the electronic device (100) in which a plurality of pattern pixels (or pattern pixels and center pixels) are located on one plane based on pixel matching information, and thereby can identify or determine pose information and projection plane information of the electronic device (100) and the projector (200) in a three-dimensional space in which the electronic device (100) and the projector (200) are located.

[0118] The electronic device (100) can obtain virtual viewpoint information regarding a virtual viewpoint (operation S460). For example, the electronic device (100) can identify or determine a viewpoint located at a predetermined distance in the frontal direction from a projection surface where pattern pixels are located on a single plane as a virtual viewpoint. For example, the virtual viewpoint may be a virtual point where a user is expected to be located in the frontal direction from the projection surface, and may be a viewpoint located at a predetermined distance from a center pixel. In one embodiment, the virtual viewpoint may be referred to as a target viewpoint, a virtual camera viewpoint, a viewing point, etc.

[0119] For example, as described above, the electronic device (100) can identify or determine projection surface information through the S450 operation. And, the electronic device (100) can identify or determine a virtual viewpoint at a predetermined distance from the center pixel of the projection surface. For example, as shown in FIG. 8, when the projection surface (810) is identified or determined, the electronic device (100) can identify or determine a virtual viewpoint (820) at a predetermined distance from the center pixel of the projection surface (810) in the frontal direction.

[0120] However, the virtual viewpoint being located at a point at a predetermined distance in the frontal direction from the center pixel of the projection plane (810) is merely one embodiment, and the virtual viewpoint may be located at a point at a predetermined distance in the frontal direction from another point of the projection plane (810).

[0121] The electronic device (100) can obtain keystone correction information (operation S470). For example, the electronic device (100) can identify or determine projection area information of a pattern image projected by a projector visible from a virtual viewpoint based on pose information and projection surface information of the electronic device (100), identify or determine a rectangular target projection area within the projection area, and obtain keystone correction information based on the projection area information and information about the target projection area.

[0122] That is, the electronic device (100) can identify or determine location information in a three-dimensional space where a virtual viewpoint is located through pose information and projection surface information of the electronic device (100), and can identify or determine projection area information of a pattern image projected by a projector visible from the virtual viewpoint. For example, the electronic device (100) can identify or determine a projection area (910) which is a distorted rectangular shape visible from the front of the virtual viewpoint, as shown in FIG. 9. And, the electronic device (100) can identify or determine a target projection area having a rectangular shape within the projection area. For example, the electronic device (100) can identify or determine a target projection area (920) located within the vertices of the projection area (910), as shown in FIG. 9.

[0123] In one embodiment, the target projection area (920) may be a projection area where a user can view an image projected in a rectangular shape through keystone correction from the front direction of the projection surface. For example, the target projection area (920) may be the largest rectangular shape within the projection area (910). However, this is merely one embodiment, and the electronic device (100) may set various shapes of target projection areas by setting the size, projection ratio, and projection direction of the target projection area according to user input through a UI (e.g., a UI provided through a display (140)).

[0124] For example, as illustrated in FIG. 10a, the electronic device (100) can set target projection areas (e.g., target projection area 1010-1, target projection area 1010-2, target projection area 1010-3) having various sizes according to user input entered through the UI. As another example, as illustrated in FIG. 10b, the electronic device (100) can set target projection areas (1020-1, 1020-2, 1020-3) to be projected at various locations according to user input received through the UI. As another example, as illustrated in FIG. 10c, the electronic device (100) can set target projection areas (1030-1, 1030-2, 1030-3) rotated in various directions according to user input entered through the UI. As another example, as illustrated in FIG. 10d, the electronic device (100) can set target projection areas (1040-1, 1040-2, 1040-3) of various ratios according to user input entered through the UI.

[0125] The electronic device (100) can obtain a transformation matrix for transforming pixels of a projection area (910) into pixels of a target projection area (920) based on information about a projection area (910) (e.g., projection area information) and a target projection area (920). For example, the transformation matrix can be expressed as shown in Equation 2 below.

[0126]

[0127] The above transformation matrix can be defined as a 3x3 matrix Homography, and it is a matrix capable of defining the transformation relationship of all points through the transformation relationship of four points.

[0128] In the manner described above, the electronic device (100) can obtain keystone correction information including a transformation matrix.

[0129] The electronic device (100) can transmit keystone correction information to the projector (200) (operation S480). In one or more embodiments, the electronic device (100) can transmit keystone correction information including a transformation matrix to the projector (200). However, this is merely one embodiment, and when the electronic device (100) provides an image to the projector, the electronic device (100) can transmit an image in which keystone correction has been performed based on the keystone correction information.

[0130] The projector (200) can perform keystone correction based on received keystone correction information (operation S490). For example, the projector (200) can perform keystone correction on an image to be projected using a transformation matrix included in the keystone correction information, and can project the image with the keystone correction performed onto a projection surface.

[0131] In one embodiment, the projector (200) can project a keystone-corrected image (1110) as shown in FIG. 11a. For example, if a user is in a virtual viewpoint located in front of the projection surface, the user can view an image (1120) having a rectangular shape as shown in FIG. 11b. Thus, even if the projector (200) is not located in front of the projection surface, the user can receive an image having an undistorted shape through keystone correction.

[0132] In one embodiment, the electronic device (100) may store a second internal parameter of the electronic device (100) in advance, but is not limited thereto, and may obtain information about the second internal parameter using a pattern image. For example, in the above-described embodiment, the electronic device (100) took a picture of the pattern image projected by the projector (200) once, but is not limited thereto. For example, in another embodiment, the electronic device (100) may take a picture of the pattern image projected by the projector (200) at a plurality of camera positions to obtain information about the second internal parameter of the electronic device (100).

[0133] For example, as shown in FIG. 12, the electronic device (100) can acquire a first image by capturing a pattern image (1200) at a first camera position (1210) and acquire a second image by capturing a pattern image (1200) at a second camera position (1220). Then, the electronic device (100) can acquire a second intrinsic parameter by using triangulation by comparing the position of a specific point (e.g., a pattern pixel) within the first and second images with the first and second camera positions (1210, 1220).

[0134] For example, the electronic device (100) can model the position of the pattern as shown in mathematical formula 3 below.

[0135]

[0136] In mathematical formula 3, N is the number of images taken, and is the position of the pattern, is the first internal parameter of the projector (200), and is the position of the decoded pattern of the Nth camera (e.g., the position of the pattern when multiple pixels are located on a single plane), and is a second internal parameter of the camera (150) of the electronic device (100), and is the position of the pattern acquired based on the Nth camera, and is the position of the Nth camera relative to the projector (200), and can be a 3D position of the pattern.

[0137] Accordingly, the electronic device (100) can identify or determine a second internal parameter that minimizes the error determined according to the following mathematical formula 4.

[0138]

[0139] For example, when there is only one captured image, point information and pose information for a single camera must be obtained, but when there are multiple captured images, only the camera position information needs to be additionally obtained, so the amount of internal parameter information that can be calculated can be increased.

[0140] Although it has been described that the electronic device (100) and the projector (200) work together to obtain keystone correction information, this is not limited thereto. For example, if the electronic device (100) is implemented as a projector, keystone correction information can be obtained within the electronic device (100).

[0141] FIG. 13 is a flowchart illustrating an embodiment in which an electronic device projects an image by performing keystone correction according to one embodiment of the present disclosure.

[0142] First, the electronic device (100) can obtain a first internal parameter related to a lens that projects an image and a second internal parameter related to a camera (operation S1310). For example, each of the first and second internal parameters may be internal parameters for a lens and a camera of a projector provided within the electronic device (100), which may be stored but not limited thereto, and may be information received from an external server, and may be obtained using triangulation as described in FIG. 12.

[0143] The electronic device (100) can capture a pattern image projected onto a projection surface using a camera (operation S1320).

[0144] The electronic device (100) can identify or determine pose information and projection plane information of the electronic device (100) based on a first internal parameter, a second internal parameter, and a captured pattern image (operation S1330). For example, the electronic device (100) can use a Fundamental Matrix to identify or determine pixel matching information between pattern pixels corresponding to a plurality of patterns included in a pattern image projected by the electronic device (100) and pattern pixels captured by the camera of the electronic device (100). Then, the electronic device (100) can identify or determine pose information of the electronic device (100) in which the pattern pixels are located on one plane based on the pixel matching information.

[0145] The electronic device (100) can obtain virtual viewpoint information for a virtual viewpoint located in front of a projection surface based on information about the projection surface (operation S1340). For example, the electronic device (100) can identify or determine a viewpoint located at a predetermined distance in the front direction from the projection surface, where pattern pixels are located on a single plane, as a virtual viewpoint.

[0146] The electronic device (100) can obtain keystone correction information based on information about a virtual viewpoint (operation S1350). For example, the electronic device (100) can identify or determine projection area information of a pattern image projected by a projector visible at a virtual viewpoint based on pose information and projection surface information of the electronic device (100), identify or determine a rectangular target projection area within the projection area, and obtain keystone correction information based on the projection area information and information about the target projection area.

[0147] The electronic device (100) can perform keystone correction on an image to be projected based on keystone correction information to obtain a keystone corrected image (which may be called a corrected image) (operation S1360).

[0148] The electronic device (100) can project a keystone-corrected image onto a projection surface (operation S1370).

[0149] In addition, in the above-described embodiment, it was explained that the electronic device (100) acquires information regarding keystone correction and transmits it to the projector (200), and then the projector (200) performs keystone correction; however, this is merely one embodiment, and the electronic device (100) can acquire keystone correction information, perform keystone correction on an image based on the acquired information regarding keystone correction, and then transmit the keystone-corrected image to the projector (200).

[0150] FIG. 14 is a sequence diagram illustrating an embodiment in which an electronic device performs keystone correction and transmits a keystone-corrected image to a projector, according to one embodiment of the present disclosure.

[0151] Meanwhile, operations S1410 to S1470 illustrated in FIG. 14 may be identical, similar, or corresponding to operations S410 to S470 described in FIG. 4. For example, operation S1410 corresponds to S410, operation S1420 corresponds to S420, operation S1430 corresponds to S430, operation S1440 corresponds to S440, operation S1450 corresponds to S450, operation S1460 corresponds to S460, and operation S1470 corresponds to S470. Therefore, repeated or redundant descriptions may be omitted.

[0152] The electronic device (100) can perform keystone correction based on keystone correction information (operation S1480). For example, instead of directly transmitting the keystone correction information to the projector (200), the electronic device (100) can first perform keystone correction on the image to be projected by the projector (200) to obtain a keystone-corrected image (i.e., a corrected image).

[0153] And, the electronic device (100) can transmit the keystone-corrected image to the projector (200) (operation S1490).

[0154] The projector (200) can project a keystone-corrected image (operation S1495), thereby enabling the projector (200) to project a keystone-corrected image even without high computing resources.

[0155] Meanwhile, the method according to various embodiments of the present disclosure may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0156] A method according to various embodiments of the present disclosure may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., a TV) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions.

[0157] In one embodiment, a device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory storage medium" simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium. For example, the "non-transitory storage medium" may include a buffer in which data is stored temporarily.

[0158] When the above instruction is executed by a processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter.

[0159] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In an electronic device, camera; At least one processor; and Memory for storing instructions; including, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Obtaining a first intrinsic parameter related to the lens of a projector that projects an image and a second intrinsic parameter related to the camera, Using the above camera, a pattern image projected onto a projection surface by the above projector is captured, and Based on the first internal parameter, the second internal parameter, and the captured pattern image, pose information for the pose of the electronic device and projection surface information for the projection surface are determined, and Based on the above projection surface information, virtual viewpoint information for a virtual viewpoint located in front of the projection surface is obtained, and Keystone correction information is generated based on the above virtual viewpoint information, and The above keystone correction information is an electronic device used to perform keystone correction on at least one image projected by the projector onto a projection surface.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Using a Fundamental Matrix, a center pixel between multiple patterns included in the pattern image and a pattern pixel corresponding to the multiple patterns are determined, and An electronic device that determines pixel matching information between the pattern pixel and the center pixel using the above Fundamental Matrix.

3. In Paragraph 2, An electronic device in which the pose information and the projection plane information are determined such that the pattern pixel is positioned on a plane based on the pixel matching information.

4. In Paragraph 3, The above virtual viewpoint is an electronic device that is a viewpoint located at a predetermined distance in the frontal direction from the above projection plane.

5. In Paragraph 4, The above virtual viewpoint is an electronic device that is a viewpoint located at a preset distance from the above center pixel.

6. In Paragraph 4, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Based on the pose information and the projection surface information, the projection area information of the pattern image visible from the virtual viewpoint is determined, and A rectangular target projection area is determined within the above projection area, and An electronic device that determines the keystone correction information based on the projection area information and the target projection area.

7. In Paragraph 6, An electronic device in which the size of the target projection area, the projection ratio, and the projection direction are set according to user input received through the UI.

8. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, To acquire the above multiple pattern images, the camera is used to capture multiple pattern images at multiple camera positions, and An electronic device that obtains the second intrinsic parameter using triangulation by comparing the position of a specific point within the multiple images captured above with the multiple camera positions.

9. In Paragraph 1, The first internal parameter above includes information regarding the focal length and principal point of the lens, and The above second internal parameter is an electronic device that includes information regarding the focal length and principal point of the camera.

10. In a method for controlling an electronic device, A step of obtaining a first intrinsic parameter related to the lens of a projector that projects an image and a second intrinsic parameter related to the camera of the electronic device; A step of capturing a pattern image projected onto a projection surface by the projector using the camera above; A step of determining pose information for the pose of the electronic device and projection surface information for the projection surface based on the first internal parameter, the second internal parameter, and the captured pattern image; A step of obtaining virtual viewpoint information for a virtual viewpoint located in front of the projection surface based on the projection surface information; and The method includes the step of generating keystone correction information based on the virtual viewpoint information above; The above keystone correction information is a control method used to perform keystone correction on at least one image projected by the projector onto a projection surface.

11. In Paragraph 10, The step of determining the pose information and the projection plane information is: A step of determining a central pixel between a plurality of patterns included in the pattern image and a pattern pixel corresponding to the plurality of patterns using a Fundamental Matrix; and A control method comprising the step of determining pixel matching information between the pattern pixel and the center pixel using the above Fundamental Matrix.

12. In Paragraph 11, A control method in which the pose information and the projection plane information are determined such that the pattern pixel is positioned on a plane based on the pixel matching information.

13. In Paragraph 12, A control method in which the virtual viewpoint above is a viewpoint located at a predetermined distance in the frontal direction from the projection plane.

14. In Paragraph 13, A control method in which the virtual viewpoint above is a viewpoint located at a preset distance from the center pixel.

15. In Paragraph 13, The step of generating the above keystone correction is, Based on the pose information and the projection surface information, the projection area information of the pattern image visible from the virtual viewpoint is determined, and A rectangular target projection area is determined within the above projection area, and A control method comprising the step of determining the keystone correction information based on the projection area information and the target projection area.

Images