Electronic device, and method for providing 3D image

Abstract

This electronic device comprises: a display for providing a multi-view image that includes a plurality of images viewable at different viewpoints; a sensor; a memory for storing instructions; and at least one processor including a processing circuit, wherein, when executed individually or collectively by the at least one processor, the instructions instruct the electronic device to: identify a viewing location of a user on the basis of sensing data acquired using the sensor; identify an image at the viewpoint corresponding to the viewing location of the user from among the plurality of images viewable at different viewpoints; adjust the brightness of the images at remaining viewpoints that exclude some viewpoints including the viewpoint of the identified image, so as to acquire an output image; and display the output image on the display.

Description

Electronic device and method of providing 3D images

[0001] One or more embodiments of the present disclosure relate to electronic devices, for example, to an electronic device that renders and provides a 3D image and a method for providing a 3D image.

[0002] The purpose of display technology is to convey natural three-dimensional space or virtual spatial information unfolding before the eyes to humans more accurately and realistically. To create natural and immersive images that make one feel as if they are actually in that place, 3D stereoscopic video technology is being developed in various ways. 3D display technology includes glasses-based and glasses-free types, and encompasses stereoscopic 3D displays (e.g., glasses-based, two-viewpoint), multi-viewpoint, and holographic 3D display technologies. Due to the discomfort caused by wearing 3D glasses, glasses-free technology is being researched.

[0003] Conventional stereoscopic 3D displays contain two viewpoint image information points for viewing stereoscopic images. Due to limitations in the field of view, the viewpoint changes discontinuously, leading to problems such as unnatural motion parallax and focus-convergence mismatch, which cause eye strain and dizziness. A viewpoint image refers to the image seen when a user looks at a 3D display in actual physical space. To address these two issues, various methods to secure a wider field of view are being researched. From a hardware perspective, research is being conducted on multi-viewpoint methods that can provide more natural 3D effects by attempting to increase the number of viewpoints. From a software perspective, rendering methods are being researched that track the user's position and provide optimal 3D stereoscopic images based on that position by rearranging pixels appropriate to that location.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] An electronic device according to one embodiment comprises: a display configured to provide a multi-view image including a plurality of images viewable at different points in time; a sensor; a memory for storing instructions; and at least one processor including a processing circuitry. When the instructions are executed individually or collectively by the at least one processor, the electronic device identifies a user's viewing position based on sensing data acquired through the sensor, identifies an image at a point in time corresponding to the user's viewing position among the plurality of images viewable at different points in time, obtains an output image by adjusting the brightness of the remaining images excluding a portion of the image including the identified image, and displays the output image on the display.

[0006] According to one embodiment, the multi-view image may be an image in which the plurality of images viewable at the different viewpoints are sequentially and repeatedly arranged. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to adjust the brightness of the remaining images excluding the identified image.

[0007] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may acquire the output image by adjusting the brightness of the remaining image so that the brightness of the remaining image, excluding the identified image, is reduced.

[0008] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may acquire the output image by adjusting the brightness of the remaining image so that the brightness is gradually reduced in proportion to the distance from the identified image.

[0009] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device acquires the output image by adjusting the brightness of the remaining image so that the brightness is gradually reduced based on a weight that decreases according to the distance from the identified image, and the magnitude of the reduction of the weight may increase in proportion to the distance from the identified image.

[0010] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may convert the remaining image from an RGB color space image to a YUV color space image, maintain the U and V values ​​in the YUV color space image and adjust the Y value, then convert the YUV color space image with the adjusted Y value back into an RGB color space image, and obtain the output image based on the converted RGB color space image.

[0011] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may obtain the output image by blurring the remaining image while adjusting the brightness of the remaining image, excluding some time points including the identified image.

[0012] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may acquire the output image by adjusting the brightness of the remaining image, excluding some time points including the identified image, when a private mode is selected.

[0013] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may provide a UI for setting a viewable viewing angle, and in the private mode, identify the remaining viewpoint images excluding the identified image based on the viewable viewing angle set through the UI.

[0014] According to one embodiment, the display may include: a display panel that displays a multi-view image including a plurality of images viewable at different points in time; and a viewing area separation unit disposed on the front of the display panel and providing an optical view corresponding to a different point in time for each viewing area.

[0015] A control method for an electronic device comprising a display configured to provide a multi-view image including a plurality of images viewable at different viewpoints according to one embodiment comprises: an operation of identifying a user's viewing position based on sensing data acquired through a sensor; an operation of identifying an image at a viewpoint corresponding to the user's viewing position among the plurality of viewpoint images viewable at different viewpoints; an operation of acquiring an output image by adjusting the brightness of the remaining images excluding a portion of the viewpoint including the identified image; and an operation of displaying the output image on the display.

[0016] A non-transient computer-readable medium storing computer instructions that cause an electronic device to perform an operation when executed by a processor of an electronic device, wherein the electronic device includes a display configured to provide a multi-view image comprising a plurality of images viewable at different viewpoints according to one embodiment, wherein the instructions include: an operation in which the electronic device identifies a user's viewing position based on sensing data acquired through a sensor; an operation in which an image at a viewpoint corresponding to the user's viewing position among the plurality of images viewable at different viewpoints; an operation in which an output image is obtained by adjusting the brightness of the remaining images excluding a portion of the viewpoint including the identified image; and an operation in which the output image is displayed on the display.

[0017] The above and other aspects and features of specific embodiments of the present disclosure will become more apparent from the following description taken together with the accompanying drawings.

[0018] FIG. 1 is a drawing for explaining the operation of an electronic device according to one or more embodiments.

[0019] FIG. 2 is a block diagram of an electronic device according to one embodiment.

[0020] FIG. 3 is a drawing for explaining an example of implementing a display according to one embodiment of the present invention.

[0021] FIG. 4 is a flowchart illustrating a method for providing a 3D image according to one embodiment.

[0022] FIG. 5 is a drawing for explaining an example of a method for providing a 3D image according to one embodiment.

[0023] FIG. 6 is a drawing for explaining a method of operation according to an example of an electronic device for providing a 3D image according to one embodiment.

[0024] FIG. 7 is a drawing for explaining a method of providing an optical view according to one embodiment.

[0025] FIGS. 8A, FIGS. 8B, and FIGS. 8C are drawings for explaining a method for adjusting the brightness of an optical view according to one embodiment.

[0026] The terms used in the embodiments of this disclosure have been selected to be as widely used as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the description section of the disclosure. Therefore, the terms used in this disclosure should be defined based on their meanings and the overall content of this disclosure, rather than merely their names (such as analyzing calls, messages, schedules, etc.).

[0027] In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of the feature (e.g., a numerical value, function, operation, or component, etc.) and do not exclude the presence of additional features.

[0028] The expression "at least one of A or / and B" should be understood as representing either "A" or "B" or "A and B".

[0029] Expressions such as "first," "second," "first," or "second" used in this specification may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0030] Where it is stated that a component (e.g., a first component) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., a second component), it should be understood that the component may be directly connected to the other component or connected through the other component (e.g., a third component).

[0031] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as “comprising” or “consisting of” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0032] In the embodiments, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor, except for a "module" or "part" that needs to be implemented in specific hardware.

[0033] In the present disclosure, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0034] The various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0035] Embodiments of the present disclosure will be described in more detail below with reference to the attached drawings.

[0036]

[0037] *28 FIG. 1 is a drawing for explaining the operation of an electronic device according to one or more embodiments.

[0038] According to one embodiment, the electronic device (100) may be implemented as various types of display devices such as a TV, monitor, kiosk, tablet PC, digital photo frame, mobile phone, LFD (large format display), Digital Signage, DID (Digital Information Display), video wall, projector display, etc. However, depending on the case, it may also be implemented as a video processing device (e.g., set-top box, one connected box) that is connected to the display device to provide video.

[0039] According to one embodiment, the electronic device (100) may be implemented as a glasses-free 3D display device based on user position tracking. For example, the electronic device (100) may provide an optimized 3D image by tracking the position of the user's eyes or head in real time. For example, the electronic device (100) may identify the user's viewing position in real time based on data obtained through a sensor (140) capable of tracking the user's viewing position in real time. For example, the sensor (140) may include at least one of an RGB camera and a depth camera.

[0040] According to one example, the electronic device (100) can identify the user's real-time viewing position by calibrating the captured image acquired through the camera. For example, calibration may be a process of converting 3D world coordinates into 2D image coordinates by estimating the intrinsic and extrinsic parameters of the camera.

[0041] According to one example, the electronic device (100) may generate a multi-view image based on the user's real-time viewing position and provide the generated multi-view image. For example, the electronic device (100) may acquire a multi-view image in which multiple viewpoint images are arranged according to a Linear Mapping method. The Linear Mapping method may generate a multi-view image by sequentially repeating a set number of viewpoint images. For example, the electronic device (100) may generate a multi-view image according to the Linear Mapping method such that viewpoint images from viewpoint 0 to viewpoint 7 are matched to an optical view in the order of viewpoints 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3... In this case, the section where 0, 1, 2, 3, 4, 5, 6, 7 are arranged in order may correspond to the regular viewpoint section. The optical view may be a visual image provided in the user's viewing area provided by an optical method. For example, the electronic device (100) may use OpenGL for multi-view image rendering and the commercial library FaceAPI for tracking the user's face.

[0042] According to one embodiment, when 3D content is activated, there may be cases where display information that one does not wish to expose to others is viewed when using a personal display. Accordingly, various embodiments are described below that provide a private mode that limits the viewing area of ​​a 3D display by adjusting the brightness of the image based on a function that identifies the user's viewing position and provides the user with an optimal viewing area.

[0043] FIG. 2 is a block diagram of an electronic device according to one embodiment.

[0044] Referring to FIG. 2, an electronic device (100) according to one embodiment may include a processor (110), a display (120), a memory (130), a sensor (140), a communication circuit (150), a user interface (160), and a speaker (170). One or more embodiments of the present disclosure may be implemented even if some of the configurations shown in FIG. 2 are omitted or substituted. At least some of the illustrated configurations may be operatively, electrically, and / or functionally connected to one another.

[0045] In one embodiment, the hardware of the electronic device (100) being operatively coupled may mean that a direct or indirect connection between the hardware is established via wired or wireless means so that the second hardware is controlled by the first hardware among the hardware. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the hardware of FIG. 2 (e.g., at least some of the processor (110), memory (130), and display (120)) may be included in a single integrated circuit, such as a system on a chip (SoC). The type and / or number of hardware included in the electronic device (100) is not limited to that shown in FIG. 2. For example, the electronic device (100) may include only some of the hardware components shown in FIG. 2.

[0046] According to one embodiment, a processor (110) of an electronic device (100) may include hardware for processing data based on one or more instructions. The hardware for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). The number of processors (110) may be one or more. For example, the processor (110) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0047] The processor (110) can control the operations of the electronic device (100) by executing instructions stored in the memory (130). For example, the processor (110) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0048] A CPU (central processing unit) is a general-purpose processor capable of performing not only general operations but also artificial intelligence operations, and it can efficiently execute complex programs through a multi-layered cache structure. The CPU is advantageous for serial processing methods, which enable the organic linkage between previous and next calculation results through sequential computation. General-purpose processors are not limited to the examples mentioned above, except for cases specified as the aforementioned CPU.

[0049] A GPU (graphic processing unit) is a processor designed for massive computations, such as floating-point operations used in graphics processing, and can perform large-scale computations in parallel by integrating a large number of cores. In particular, GPUs may be advantageous over CPUs for parallel processing methods such as convolution operations. Additionally, GPUs can be used as co-processors to complement the functions of CPUs. Processors for massive computation are not limited to the examples mentioned above, except for cases specified as GPUs.

[0050] A Neural Processing Unit (NPU) is a processor specialized for artificial intelligence computations using artificial neural networks, and each layer constituting the neural network can be implemented in hardware (e.g., silicon). In this case, since the NPU is designed specifically according to the specifications required by the vendor, it has a lower degree of flexibility compared to CPUs or GPUs, but it can efficiently process the artificial intelligence computations required by the vendor. Meanwhile, as a processor specialized for artificial intelligence computations, the NPU can be implemented in various forms such as Tensor Processing Units (TPUs), Intelligence Processing Units (IPUs), and Vision Processing Units (VPUs). Artificial intelligence processors are not limited to the examples mentioned above, except for cases specified as the aforementioned NPU.

[0051] According to one embodiment, the memory (130) of the electronic device (100) may include a hardware component for storing data and / or instructions that are input and / or output to the processor (110). Depending on the purpose of data storage, the memory (130) may be implemented in the form of a memory embedded in the electronic device (100') or in the form of a memory that is detachable from the electronic device (100). For example, data for operating the electronic device (100) may be stored in a memory embedded in the electronic device (100'), and data for the expansion function of the electronic device (100) may be stored in a memory that is detachable from the electronic device (100). Meanwhile, the memory embedded in the electronic device (100) may be implemented as at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD).In addition, the memory that can be attached to and detached from the electronic device (100') can be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.) or an external memory that can be connected to a USB port (e.g., USB memory).

[0052] According to one embodiment, within the memory (130) of the electronic device (100), one or more instructions (or commands) representing operations and / or operations to be performed on data by the processor (110) may be stored. A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the electronic device (100) and / or the processor (110) may perform various operations when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. In the following, the statement that an application is installed on an electronic device (100) means that one or more instructions provided in the form of an application are stored in the memory (130) of the electronic device (100), and that the one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the electronic device (100)) that is executable by the processor (110) of the electronic device (100).

[0053] At least one processor (110) controls the processing of input data according to a predefined operation rule or AI model (artificial-intelligence model) stored in memory (130). The predefined operation rule or AI model is characterized by being created through learning. Being created through learning means that a predefined operation rule or AI model with desired characteristics is created by applying a learning algorithm to a number of learning data. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is performed, or it may be performed through a separate server / system.

[0054] An AI model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs the layer's operation through the result of the operation of the previous layer and at least one defined operation. Examples of neural networks include convolutional neural networks (CNN), recurrent neural networks (RNN), deep neural networks (DNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN), deep Q-networks, and Transformers; however, the neural networks in this disclosure are not limited to the aforementioned examples except where specified.

[0055] A learning algorithm is a method of training a specific target device (e.g., a robot) using a number of learning data to enable the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithms in this disclosure are not limited to the aforementioned examples except where specified.

[0056] According to one embodiment, a display (120) of an electronic device (100) can output visualized information to a user. For example, the display (120) can be controlled by a controller, such as a GPU (graphic processing unit), to output visualized information to a user. The display (120) may include LED (Light Emitting Diodes), micro LED, Mini LED, OLED (Organic Light Emitting Diodes) display, LCD (Liquid Crystal Display), PDP (Plasma Display Panel), QD (Quantum dot) display and / or QLED (Quantum dot light-emitting diodes). According to one example, the display (120) may be implemented as a flat display, a curved display, a folding and / or rolling flexible display.

[0057] According to one embodiment, the display (120) may be implemented as a 3D display (3-dimensional display) that provides a 3D image to a user. For example, the display (120) may include a structure such as a lenticular lens or a parallax barrier placed on the front of a display panel containing a plurality of pixels.

[0058] A sensor (140) of an electronic device (100) according to one embodiment can sense various information. The sensor (140) can be implemented as various types of sensors.

[0059] According to one example, the sensor (140) may include a camera. The camera may capture surrounding subjects and convert image information into digital data to provide to the processor (110). The electronic device (100) may include at least one camera on the front and / or the rear, opposite direction, where the display (120) is included in the housing. According to one embodiment, the camera may include a lens assembly comprising at least one lens that collects light emitted from an external environment (or subject), an image sensor (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that converts the light collected through the lens assembly into an electrical signal to generate image data, and an image signal processor that performs various processing on the image data acquired from the image sensor. At least some of the above-described configurations of the camera may be omitted or replaced with other configurations. The camera may provide images of the external environment captured in real time to the processor (110) through an interface (e.g., a mobile industry processor interface). For example, the camera may include at least one of an RGB camera, an ultra-wide angle camera, a depth camera, or an IR camera.

[0060] According to one example, the sensor (140) may include at least one sensor among a time of flight (ToF) sensor, an ultrasonic sensor, a radio detection and ranging (RADAR) sensor, a photodiode sensor, a proximity sensor, a passive infrared (PIR) sensor, a pinhole sensor, a pinhole camera, an infrared human body detection sensor, a complementary metal oxide semiconductor (CMOS) image sensor, a thermal detection sensor, a light sensor, and a motion detection sensor.

[0061] The sensor (140) may include a touch sensor that detects touch actions, such as a touch film, a touch sheet, or a touch pad.

[0062] The sensor (140) may further include at least one sensor capable of sensing ambient illuminance, ambient temperature, and the direction of incidence of light. In this case, the sensor (140) may be implemented as an illuminance sensor, a temperature sensing sensor, and a light intensity sensing layer.

[0063] The sensor (140) may further include at least one of an acceleration sensor (or gravity sensor), a geomagnetic sensor, and a gyro sensor. For example, the acceleration sensor may be a 3-axis acceleration sensor. The 3-axis acceleration sensor may measure gravitational acceleration by axis and provide raw data to the processor (140). The geomagnetic sensor or the gyro sensor may be used to obtain attitude information. Here, the attitude information may include at least one of roll information, pitch information, or yaw information.

[0064] A communication circuit (150) of an electronic device (100) according to one embodiment may include hardware for supporting the transmission and / or reception of electrical signals between the electronic device (100) and an external device (e.g., a server). For example, the communication circuit (150) may communicate with an external device, an external storage medium (e.g., a USB memory stick), an external server (e.g., a web hard drive), etc., through a communication method such as Bluetooth, AP-based Wi-Fi (Wi-Fi, Wireless LAN network), Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), Optical, Coaxial, etc. According to one example, the communication circuit (150) can communicate with other electronic devices, external servers and / or remote control devices, etc.

[0065] The user interface (160) of the electronic device (100) according to one embodiment may be implemented as a device such as a button, touch pad, mouse and keyboard, or as a touch screen capable of performing the display function and operation input function described above.

[0066] A speaker (170) of an electronic device (100) according to one embodiment may be configured to output various audio data as well as various notification sounds or voice messages. A processor (110) may control the speaker (170) to output feedback or various notifications in the form of audio according to various embodiments of the present disclosure.

[0067] FIG. 3 is a drawing for explaining an example of implementing a display according to one embodiment of the present invention.

[0068] According to one embodiment, the display (120) may include a display panel and a viewing area separation unit.

[0069] A viewing area separator is positioned on the front of the display panel to provide different viewing points, i.e., multi-views, for each viewing area. In this case, the viewing area separator can be implemented as a lenticular lens or a parallax barrier.

[0070] For example, the viewing area separation unit may be implemented as a lenticular lens comprising a plurality of lens regions. Accordingly, the lenticular lens can refract an image displayed on a display panel through the plurality of lens regions. Each lens region is formed with a size corresponding to at least one pixel, so that light passing through each pixel can be dispersed differently according to the viewing area.

[0071] As another example, the viewing area separation section can be implemented as a parallax barrier. The parallax barrier is implemented as a transparent slit array containing multiple barrier regions. Accordingly, light is blocked through slits between the barrier regions, allowing images at different viewpoints to be emitted for each viewing area.

[0072] In FIG. 3, an example is given in which the viewing area separation part (122) is implemented as a lenticular lens array.

[0073] According to FIG. 3, the display panel (121) includes a plurality of pixels divided into a plurality of columns. Pixel data corresponding to images at different viewpoints may be sequentially and repeatedly arranged in the plurality of pixels. According to one example, as shown in FIG. 3, pixel data corresponding to viewpoint 0 and viewpoint 7 may be sequentially and repeatedly arranged. In this case, the electronic device (100) may provide an 8-viewpoint image (310). For example, the electronic device (100) may generate an output image by combining each viewpoint image at the final stage of rendering based on an index image in which viewpoint information corresponding to each pixel is recorded.

[0074] According to one example, light corresponding to each viewpoint image 0, 1, 2, 3, 4, 5, 6, 7 formed on a display panel (121) is projected onto a viewing area separation unit (122), and the viewing area separation unit (122) disperses the projected light of each viewpoint image 0, 1, 2, 3, 4, 5, 6, 7 and transmits it toward the viewer. For example, the viewing area separation unit (122) can create exit pupils at the viewer's position, i.e., at the viewing distance. For example, as shown in FIG. 3, if the viewing area separation unit (122) is implemented as a lenticular lens array, the thickness and diameter of the lenticular lenses, and if implemented as a parallax barrier, the spacing of the slits, etc., can be designed so that the exit pupils created by each row are separated by an average binocular center distance of less than 65 mm. The separated image lights each form an optical view (or viewing view). For example, as shown in FIG. 3, when the first to seventh views are formed and the user's left and right eyes are positioned in the third and fourth views, respectively, the user can view the 3D image.

[0075] FIG. 4 is a flowchart illustrating a method for providing a 3D image according to one embodiment.

[0076] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0077] According to one embodiment, operations 410 to 440 can be understood as being performed in the processor (110) of the electronic device (100).

[0078] According to FIG. 4, in operation 410, an electronic device (100) according to one embodiment can identify the user's viewing location based on sensing data obtained through a sensor (430).

[0079] According to one example, the electronic device (100) can identify at least one of a user's face (or head) or eyes within a captured image acquired by a camera. For example, the electronic device (100) can identify an object region included in the captured image through at least one of object recognition, object detection, object tracking, or image segmentation. For example, the electronic device (100) can identify the user by using techniques such as semantic segmentation, which separates and extracts objects included in the input image by type as needed; instance segmentation, which recognizes objects by separating them even if they are of the same type; and a rectangular bounding box containing the detected object when detecting objects included in the image.

[0080] According to one example, the electronic device (100) can identify a user's face from a captured image and identify the user's eyes from the user's face. For example, various conventional methods can be used as face region detection methods. Specifically, direct recognition methods and statistical methods can be used. Direct recognition methods create rules using physical features such as the contours, skin color, and the size or distance between components of a face image, and compare, inspect, and measure according to those rules. Statistical methods can detect face regions according to a pre-trained algorithm. That is, it is a method of converting unique features included in an input face into data and comparing and analyzing them with a prepared large database (shapes of faces and other objects). In particular, face regions can be detected according to a pre-trained algorithm, and methods such as MLP (Multi Layer Perceptron) and SVM (Support Vector Machine) can be used. User eye regions can be identified in a similar way.

[0081] According to one example, the electronic device (100) can identify a user's eyes from a captured image using a learned artificial intelligence model. For example, the artificial intelligence model may be implemented as a neural network including multiple neural network layers. The artificial intelligence model may be implemented as a Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, but is not limited thereto.

[0082] According to one example, the user's viewing location information may include at least one of the user's movement distance information and movement direction information. For example, the user's viewing location information may include movement distance information and movement direction information in a three-dimensional space using a coordinate system of the X-axis, Y-axis, and Z-axis.

[0083] In operation 420, an electronic device (100) according to one embodiment can identify a viewpoint image corresponding to the user's viewing position among a plurality of different viewpoint images. The plurality of different viewpoint images may be a plurality of images taken or generated at different angles.

[0084] According to one example, the electronic device (100) may receive multiple different viewpoint images from an external device (e.g., a server) or generate multiple different viewpoint images using various technologies. For example, multiple viewpoint images may be obtained through at least one of multi-camera shooting, computer graphics rendering, or image processing technology. For example, multi-camera shooting may generate multiple different viewpoint images by simultaneously capturing images from various angles using multiple cameras. For example, computer graphics rendering may generate images of various viewpoints through software based on a 3D model. For example, image processing technology may extract depth information from a single image and generate multiple viewpoint images based on the extracted depth information.

[0085] According to one example, the electronic device (100) can identify a viewpoint image provided to the user's viewing position among a plurality of viewpoint images according to the optical structure of a viewing area separation part (e.g., a lenticular lens or a perellax barrier).

[0086] In operation 430, an electronic device (100) according to one embodiment may obtain an output image by adjusting the brightness (or luminance) of the remaining viewpoint images, excluding some viewpoint images that include a viewpoint image identified based on the user's viewing position. For example, the output image may be an image in which a plurality of different viewpoint images are sequentially repeated.

[0087] According to one example, the electronic device (100) can display the brightness of the remaining viewpoint images, excluding the identified viewpoint image, by adjusting the brightness.

[0088] According to one example, the electronic device (100) can adjust the brightness of the remaining viewpoint images so that the brightness of the remaining viewpoint images is reduced, excluding the viewpoint image identified based on the user's viewing position.

[0089] According to one example, the electronic device (100) can adjust the brightness of the remaining viewpoint images so that the brightness gradually decreases in proportion to the distance from the identified viewpoint image.

[0090] According to one example, the electronic device (100) can adjust the brightness of the remaining viewpoint image so that the brightness gradually decreases based on a weight that decreases according to the distance from the viewpoint image identified based on the user's viewing position.

[0091] For example, the magnitude of the weight reduction may be equal in proportion to the distance from the identified viewpoint image. For example, the electronic device (100) may adjust the brightness of the remaining viewpoint images so that the Y value reference of the YUV image is reduced by the same amount according to the distance from each viewpoint image. For example, the electronic device (100) may adjust the brightness of the remaining viewpoint images so that the Y value reference of the YUV image is reduced in the order of -10, -20, -30 according to the distance from each viewpoint image.

[0092] For example, the magnitude of the weight reduction may increase proportionally to the distance from the identified viewpoint image. For example, the electronic device (100) may adjust the brightness of the remaining viewpoint images so that the reference Y value of the YUV image decreases by different magnitudes that gradually increase according to the distance from each viewpoint image. For example, the electronic device (100) may adjust the brightness of the remaining viewpoint images so that the reference Y value of the YUV image decreases in the order of -10, -25, -45... according to the distance from each viewpoint image.

[0093] According to one embodiment, the electronic device (100) can convert the image of the remaining viewpoint from an RGB (Red, Green, Blue) color space image (or RGB domain image) to a YUV (Luminance, Chrominance) color space image (or YUV domain image) in order to adjust the brightness of the image of the remaining viewpoint.

[0094] RGB is represented by three values ​​indicating the intensity of the red, green, and blue components in each channel. In the case of an 8-bit image, the channels have values ​​between 0 and 255; for example, (255, 0, 0) can represent red, (0, 255, 0) can represent green, and (0, 0, 255) can represent blue. YUV is a color space composed of three elements: luminance (Y), chrominance (U, V), and chromaticity. Since Y is the brightness component, it contains luminance information, while U and V can contain chromaticity information regarding color. For each pixel, the U and V values ​​can represent the difference between the color of that pixel and the color of surrounding pixels.

[0095] According to one example, the electronic device (100) can convert an RGB color space image into a YUV color space image using at least one of a preset formula, a preset formula, a preset rule, or an algorithm. For example, the electronic device (100) can normalize the RGB values ​​of each pixel included in the RGB color space image to values ​​from 0 to 1, and obtain the YUV values ​​of each pixel according to the formula “Y=0.299×R+0.587×G+0.114×B, U=0.492×(BY), V=0.877×(RY)”.

[0096] According to one example, an electronic device (100) maintains the U value and V value in a YUV color space image, adjusts the Y value (or Y signal), converts the YUV color space image with the adjusted Y value into an RGB color space image, and obtains a remaining viewpoint image with adjusted brightness based on the converted RGB color space image.

[0097] According to one example, the electronic device (100) can obtain an output image based on an identified viewpoint image (RGB image) and the remaining viewpoint image in the reconverted RGB color space.

[0098] According to one embodiment, the electronic device (100) can obtain an output image by blurring the remaining viewpoint images while adjusting the brightness of some viewpoint images, excluding some viewpoint images identified based on the user's viewing position. For example, blurring can be performed by averaging the color values ​​of surrounding pixels to reduce the difference between individual pixels.

[0099] According to one embodiment, when a private mode is selected, the electronic device (100) can obtain an output image by adjusting the brightness of the remaining viewpoint images, excluding some viewpoint images that include a viewpoint image identified based on the user's viewing position.

[0100] According to one embodiment, the electronic device (100) may provide a UI for setting a viewable viewing angle. According to one example, the electronic device (100) may provide a UI for setting a viewable viewing angle through at least one of a viewing angle selection slider, a manual input field, or viewing angle presets. According to one example, when a private mode is selected, the electronic device (100) may identify a viewpoint image among a plurality of viewpoint images to adjust brightness based on the viewable viewing angle set through the UI.

[0101] According to one embodiment, the electronic device (100) can automatically set the optimal audiovisual quality by analyzing the image provided on the screen.

[0102] According to one embodiment, the electronic device (100) can automatically apply an optimized viewing angle according to at least one of lighting or viewing distance.

[0103] According to one embodiment, the electronic device (100) can recommend a viewing angle identified through a UI based on at least one of screen analysis, lighting, or viewing distance.

[0104] According to one embodiment, the electronic device (100) may provide a preview screen that simulates a situation in which the brightness of the outer edge of the selected viewing angle decreases whenever a recommended viewing angle is selected.

[0105] In operation 440, the electronic device (100) according to one embodiment can display the output image obtained in operation 430 on the display (120).

[0106] FIG. 5 is a drawing for explaining an example of a method for providing a 3D image according to one embodiment.

[0107] According to one embodiment, the electronic device (100) can track the user's viewing position and render a viewpoint image that is optimal for the user's viewing position and provide it in real time. A viewpoint image may be an image of the present viewed from a point in a physical real space. The electronic device (100) can generate a viewpoint image corresponding to each viewpoint. For example, the electronic device (100) can provide eight viewpoint images as illustrated in FIG. 5. For example, the eight viewpoint images can be obtained by positioning a virtual camera at a viewpoint location defined in three-dimensional space during display design and saving the scene displayed on the current screen while changing the viewpoint.

[0108] According to one embodiment, the electronic device (100) can limit the viewable viewing angle by lowering the brightness of the image provided to a location other than the user's viewing location according to the user's viewing location, as shown in FIG. 5. Accordingly, privacy protection for the 3D image is possible by restricting others from viewing the 3D image. For example, the privacy function may be activated only when necessary by allowing the user to select a privacy mode when desired.

[0109] FIG. 6 is a drawing for explaining a method of operation according to an example of an electronic device for providing a 3D image according to one embodiment.

[0110] According to FIG. 6, the electronic device (100) can identify the user's viewing location by tracking the user's face based on a captured image obtained through a sensor (130), for example, a camera (621).

[0111] According to one embodiment, the electronic device (100) can perform a rendering process (610) that generates an output image based on the user's viewing position.

[0112] According to one example, the electronic device (100) can generate a multi-view image including multiple viewpoint images (622).

[0113] According to one example, the electronic device (100) can identify an image corresponding to the user's viewing position among a plurality of viewpoint images based on the user's viewing position. For example, the electronic device (100) can identify two viewpoints (e.g., viewpoint 3 and viewpoint 4) corresponding to the user's left and right eyes among the eight viewpoints shown in FIG. 5.

[0114] According to one example, the electronic device (100) can adjust the brightness of the remaining viewpoint images, excluding the image corresponding to the user's viewing position (e.g., viewpoints 3 and 4 in FIG. 5) (623). For example, the electronic device (100) may be designed so that the viewpoint images 3 and 4 are provided at the current user's viewing position. In this case, the electronic device (100) may adjust the brightness of the viewpoint images 2 and 5 to -10, the brightness of the viewpoint images 1 and 6 to -30, and the brightness of 0 and 7 to -50, so that they gradually decrease. Here, -10, -30, and -50 may be examples of numerical expressions of the degree of brightness adjustment for convenience of explanation.

[0115] According to one example, the electronic device (100) can obtain a multi-view output image by combining brightness-adjusted viewpoint images (e.g., Texture 0 to Texture 7) (624) stored in video memory (625). For example, the electronic device (100) can combine multiple viewpoint images (e.g., Texture 0 to Texture 7) using a fragment shader. A fragment shader may be a shader program used in graphics programming to determine the final color of each pixel. For example, it is used in graphics APIs such as OpenGL and Vulkan, and can generate an output image by calculating the color of each fragment (pixel).

[0116] According to one example, the electronic device (100) can combine each viewpoint image with brightness adjusted based on an index image (630). For example, the index image (630) may be an image in which viewpoint information corresponding to each pixel is recorded as an index. For example, in the case of an 8-viewpoint image, indices from 0 to 7 may be used as viewpoint information. However, it is not limited thereto, and the index may be set to various values ​​that can distinguish each viewpoint.

[0117] According to one example, the electronic device (100) can output an output image obtained through a rendering process (610) to a display (110).

[0118] In the embodiments described above, it was explained that one viewpoint image is provided to each of the user's left and right eyes, but according to one embodiment, a single optical view can be provided in which multiple viewpoint images are provided to each of the user's left and right eyes. An optical view is a visual image provided in the user's viewing area by an optical method, and when different optical views are provided to the user's left and right eyes, the user can view a 3D image.

[0119] FIG. 7 is a drawing for explaining a method of providing an optical view according to one embodiment.

[0120] According to one embodiment, the electronic device (100) can provide a plurality of optical views that allow viewing of 3D images in the user's viewing area.

[0121] According to an embodiment illustrated in FIG. 7, the electronic device (100) may provide eight optical views (711 to 718). According to one example, each optical view (711 to 718) may consist of five viewpoint images (or sub-views). That is, eight optical views (711 to 718) may be generated using a total of 40 (1 to 40) viewpoint images (720). For example, the first view (711) may be provided by synthesizing 1 to 5 viewpoint images (or sub-views). In this case, if the parallax of adjacent optical views is A, the parallax of adjacent viewpoint images may be A / 5. However, this is merely an example, and the number of optical views and the number of viewpoint images constituting each optical view may vary depending on the implementation example.

[0122] FIGS. 8a to 8c are drawings for explaining a method for adjusting the brightness of an optical view according to one embodiment.

[0123] According to FIG. 8a, the electronic device (100) can adjust the brightness (or luminance) of the remaining views (711, 712, 715, 716, 717, 718), excluding the third view (713) and the fourth view (714), to a brightness darker than that of the third view (713) and the fourth view (714), when the views perceived by the user's left and right eyes are the third view (713) and the fourth view (714). For example, the electronic device (100) can adjust the brightness of the remaining views, excluding the third view (713) and the fourth view (714), to a brightness equal to a ratio of 1 / n (where n is a number greater than 1) of the brightness of the third view (713) and the fourth view (714).

[0124] According to FIG. 8b, when the views perceived by the user's left and right eyes are the third view (713) and the fourth view (714), the electronic device (100) can adjust the brightness (or luminance) of the remaining views (711, 712, 715, 716, 717, 718), excluding the third view (713) and the fourth view (714), to a brightness darker than that of the third view (713) and the fourth view (714), but to a different brightness. For example, the electronic device (100) can provide brightness that gradually decreases as it moves further away from the third view (713) and the fourth view (714). For example, the electronic device (100) can adjust the brightness of the second view (712) and the fifth view (715), which are closest to the third view (713) and the fourth view (714), to a ratio of 1 / n1 (where n1 is a number greater than 1) to the brightness of the third view (713) and the fourth view (714), and the first view (711) and the sixth view (716) to a ratio of 1 / n2 (where n2 is a number less than n1) to the brightness of the third view (713) and the fourth view (714). For example, the electronic device (100) can adjust the brightness of (711, 712, 715, 716, 717, 718) so that the brightness gradually decreases at the same rate as it moves further away from the third view (713) and the fourth view (714).

[0125] According to FIG. 8c, when the views perceived by the user's left and right eyes are the third view (713) and the fourth view (714), the electronic device (100) can adjust the brightness (or luminance) of the sub-views constituting the remaining views (711, 712, 715, 716, 717, 718), excluding the third view (713) and the fourth view (714), to a brightness darker than that of the third view (713) and the fourth view (714), but to a different brightness. For example, the electronic device (100) can adjust the brightness so that it gradually decreases sequentially starting from the sub-view (10) that is close to the third view (713) among the 10 sub-views (1 to 10) constituting the second view (712) and the first view (711). For example, the brightness can be adjusted to gradually decrease in the order of 10→9→8→7→6→5→4→3→2→1. For example, the electronic device (100) can adjust the brightness to gradually decrease sequentially starting from the sub-view (21) that is close to the fourth view (714) among the 20 sub-views (21 to 40) constituting the fifth view (715) to the eighth view (718). For example, the brightness can be adjusted to gradually decrease in the order of 21→22→23...→38→39→40.

[0126] In the above-described embodiments, it was explained that the brightness values ​​of the subviews constituting the view are all adjusted or not all adjusted; however, this is merely an example, and even among subviews constituting the same view, some may have their brightness values ​​adjusted while others may not.

[0127] According to the various embodiments described above, it is possible to identify the user's viewing position in a glasses-free 3D display and provide an optimal 3D image. In addition, the brightness of the 3D image can be adjusted at positions other than the user's viewing position to limit the visibility of the image to others. Accordingly, privacy protection for the 3D image becomes possible.

[0128] An electronic device according to one or more embodiments disclosed in this disclosure may be a device of various forms. An electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. An electronic device according to an embodiment of this disclosure is not limited to the devices described above.

[0129] The term “module” as used in one or more embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0130] One or more embodiments of the present disclosure may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0131] According to one embodiment, the method according to one or more embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read only memory (CD-ROM)) or an application store (e.g., Play Store). TMIt can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0132] According to one or more embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one or more embodiments, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one or more embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, A display configured to provide a multi-view video including multiple videos viewable at different points in time; Sensor; Memory for storing instructions; and at least one processor including processing circuitry; and When the instructions stored in the memory are executed individually or collectively by the above at least one processor, the electronic device, Identifying the user's viewing location based on sensing data acquired through the above sensor, and Identifying the video at a time corresponding to the user's viewing position among the plurality of videos viewable at the different times, and An output image is obtained by adjusting the brightness of the remaining images, excluding the images viewable at some points including the identified images above, and An electronic device that displays the above output image on the above display.

2. In Paragraph 1, The above multi-view image is, It includes a video in which multiple videos viewable at different points in time are sequentially repeated, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that adjusts the brightness of the remaining images excluding the identified image above.

3. In Paragraph 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that obtains an output image by adjusting the brightness of the remaining image so that the brightness of the remaining image, excluding the identified image, is reduced.

4. In Paragraph 3, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device for obtaining an output image by adjusting the brightness of the remaining image so that the brightness gradually decreases in proportion to the distance from the identified image.

5. In Paragraph 3, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, The output image is obtained by adjusting the brightness of the remaining image so that the brightness gradually decreases based on a weighting factor that decreases based on the distance from the identified image, and The magnitude of the reduction in the above weight is, An electronic device that increases in proportion to the distance from the identified image above.

6. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Convert the remaining images above from RGB color space images to YUV color space images, and After maintaining the U and V values ​​in the above YUV color space image and adjusting the Y value, the above YUV color space image with the adjusted Y value is re-converted into an RGB color space image, and An electronic device that obtains the output image based on the above-mentioned reconverted RGB color space image.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that obtains an output image by blurring the remaining image while adjusting the brightness of the image, excluding some time points including the identified image.

8. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that, when private mode is selected, adjusts the brightness of the remaining images excluding some time points including the identified images to obtain the output images.

9. In Paragraph 8, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Provides a UI for setting the viewing angle, and An electronic device that, in the above private mode, identifies the remaining viewpoint images excluding the identified viewpoint image based on the viewable viewing angle set through the UI.

10. In Paragraph 1, The above display is, A display panel displaying a multi-view image including a plurality of images viewable at different points in time; and An electronic device comprising: a viewing area separation unit disposed on the front of the display panel and providing an optical view corresponding to a different time point for each viewing area.

11. A method for controlling an electronic device comprising a display configured to provide a multi-view image including a plurality of images viewable at different viewpoints, An operation to identify the user's viewing location based on sensing data acquired through a sensor; An operation of identifying an image at a time corresponding to the user's viewing position among the plurality of images viewable at the above different time points; An operation to obtain an output image by adjusting the brightness of the remaining image excluding some time points including the identified image above; and A control method comprising the operation of displaying the output image on the display.

12. In Paragraph 11, The above multi-view image is, It includes a video in which the plurality of videos viewable at the different points in time are sequentially and repeatedly arranged, and The operation of acquiring the above output image is, A control method comprising: adjusting the brightness of the remaining images excluding the identified image and displaying them on the display.

13. In Paragraph 12, The output image above is, A control method obtained by adjusting the brightness of the remaining image so that the brightness of the remaining image, excluding the identified image, is reduced.

14. In Paragraph 13, The output image above is, A control method obtained by adjusting the brightness of the remaining image so that the brightness gradually decreases in proportion to the distance from the identified image.

15. A non-transient computer-readable medium storing computer instructions that cause an electronic device to perform an operation when executed by a processor of an electronic device comprising a display configured to provide a multi-view image including a plurality of images viewable at different viewpoints, said electronic device The above instruction is that the electronic device, An operation to identify the user's viewing location based on sensing data acquired through a sensor; An operation of identifying an image at a time corresponding to the user's viewing position among the plurality of images viewable at the above different time points; An operation to obtain an output image by adjusting the brightness of the remaining image excluding some time points including the identified image above; and A non-transient computer-readable medium that enables the operation of displaying the above output image on the above display.

Images