Electronic device and method for providing multi-screen

Abstract

This electronic device may: if calibration for gaze tracking starts, identify a display position of a calibration point on the basis of division lines corresponding to a plurality of screens included in a multi-screen; display the calibration point on a display on the basis of the identified display position; if a gaze of a user gazing at the calibration point is identified on the basis of a captured image acquired through a camera, store, in a memory, calibration information acquired on the basis of the identified gaze of the user and the display position of the calibration point; and if the multi-screen is provided on the display, perform, on the basis of the calibration information, gaze tracking of the user for the plurality of screens included in the multi-screen.

Description

Method of providing electronic devices and multi-screens

[0001] One or more embodiments of the present disclosure relate to electronic devices, for example, to an electronic device providing a multi-screen and a method for providing a multi-screen.

[0002] Driven by advancements in electronic technology, various types of electronic devices are being developed and distributed. In particular, display devices used in diverse settings, such as homes, offices, and public spaces, have been continuously evolving over the past few years.

[0003] Multi-screens can display multiple contents simultaneously by providing multiple screens at the same time. For example, multi-screens can display multiple contents received from multiple input sources on different screens at the same time.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] According to one embodiment, the electronic device comprises: a display; a camera; a memory for storing instructions; and one or more processors including processing circuitry; wherein, when the instructions are executed individually or collectively, the electronic device identifies a display location of a calibration point based on a dividing line corresponding to a plurality of screens included in a multi-screen when calibration for eye tracking is initiated, displays the calibration point on the display based on the identified display location, and when the gaze of the user looking at the calibration point is identified based on a captured image acquired through the camera, stores calibration information acquired based on the identified user's gaze and the display location of the calibration point in the memory, and when the multi-screen is provided to the display, performs eye tracking of the user for the plurality of screens included in the multi-screen based on the calibration information.

[0006] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may identify a calibration point at the intersection of a plurality of dividing lines corresponding to the plurality of screens included in the multi-screen, or identify a calibration point at the center of a single dividing line corresponding to the plurality of screens.

[0007] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may identify the number of calibration points based on the larger number between the number of the first horizontal split screen and the number of the second vertical split screen among the plurality of screens, minus 1.

[0008] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may identify a threshold range of eye-tracking disabled areas based on the position of a dividing line included in the multi-screen when the multi-screen is provided to the display, and ignore the user's gaze when the user's gaze is located in the eye-tracking disabled area. When the instructions are executed individually or collectively by the at least one processor, the electronic device may play a first image of a first screen corresponding to the first area when the user's gaze is located in a first area outside the eye-tracking disabled area, and play a first image of a screen corresponding to the second area when the user's gaze moves from the first area to a second area outside the eye-tracking disabled area.

[0009] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may disable the remaining screens among the plurality of screens included in the multi-screen, excluding the first screen where the user's gaze is located, when the multi-screen is provided to the display.

[0010] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device monitors an event occurring on the remaining screens that are disabled, excluding the first screen, while the remaining screens are disabled, and if an event is identified on the second screen among the remaining screens, the second screen is enabled to play an image on the second screen.

[0011] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may reduce the image brightness output from the remaining screen among the plurality of screens to a preset brightness and reduce the sound volume to a preset size, thereby disabling the remaining screen.

[0012] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may enable the remaining screens if another user is identified based on a captured image acquired through the camera while the remaining screens, excluding the first screen, are deactivated.

[0013] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device can identify a user through face recognition of a captured image acquired through the camera when the multi-screen function is executed, and if calibration information corresponding to the identified user is not stored in the memory, the calibration for eye tracking on the multi-screen can be initiated.

[0014] According to one embodiment, when calibration for eye tracking is initiated, the method comprises: identifying a display position of a calibration point based on a dividing line corresponding to a plurality of screens included in a multi-screen; displaying the calibration point based on the identified display position; when the user's gaze directed at the calibration point is identified based on a captured image acquired through the camera, storing calibration information acquired based on the identified user's gaze and the display position of the calibration point; and when the multi-screen is provided, performing eye tracking of the user for the plurality of screens included in the multi-screen based on the calibration information.

[0015] According to one embodiment, a non-transient computer-readable medium storing computer instructions that cause the electronic device to perform an operation when executed by a processor of the electronic device comprises: an operation of identifying a display position of a calibration point based on a dividing line corresponding to a plurality of screens included in a multi-screen when calibration for eye tracking is started; an operation of displaying the calibration point based on the identified display position; an operation of storing calibration information obtained based on the identified gaze of the user and the display position of the calibration point when the gaze of the user looking at the calibration point is identified based on a captured image obtained through the camera; and an operation of performing eye tracking of the user for the plurality of screens included in the multi-screen based on the calibration information when the multi-screen is provided.

[0016] The above and other aspects and features of specific embodiments of the present disclosure will become more apparent from the following description taken together with the accompanying drawings.

[0017] FIG. 1 is a drawing for explaining a method of providing a multi-screen according to one embodiment.

[0018] FIG. 2 is a block diagram showing the configuration of an electronic device according to one embodiment.

[0019] FIG. 3 is a flowchart illustrating a method for controlling an electronic device according to one embodiment.

[0020] FIGS. 4a and FIGS. 4b are drawings for explaining a method for identifying calibration points according to one embodiment.

[0021] FIGS. 5A and FIGS. 5B are drawings for explaining a method for identifying calibration points according to one embodiment.

[0022] FIGS. 6a to 6c are drawings for explaining a gaze tracking method according to one embodiment.

[0023] FIGS. 7a to 7c are drawings for explaining a gaze tracking method according to one embodiment.

[0024] FIGS. 8A and FIGS. 8B are drawings for explaining a method of using calibration information according to one embodiment.

[0025] FIG. 9 is a drawing for explaining a method of providing eye tracking in a multi-screen according to one embodiment.

[0026] The present disclosure will be described in detail below with reference to the attached drawings.

[0027] The terms used in the embodiments of this disclosure have been selected to be as widely used as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the description section of the disclosure. Therefore, the terms used in this disclosure should be defined based on their meanings and the overall content of this disclosure, rather than merely their names (such as analyzing calls, messages, schedules, etc.).

[0028] In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of the above features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0029] The expression "at least one of A and / or B" should be understood as representing either "A" or "B" or "A and B".

[0030] Expressions such as "first," "second," "first," or "second" used in this specification may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0031] Where it is stated that a component (e.g., a first component) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., a second component), it should be understood that the component may be directly connected to the other component or connected through the other component (e.g., a third component).

[0032] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as “comprising” or “consisting of” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0033] In the embodiments, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor, except for a "module" or "part" that needs to be implemented in specific hardware.

[0034] In the present disclosure, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0035] The various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0036] Embodiments of the present disclosure will be described in more detail below with reference to the attached drawings.

[0037] FIG. 1 is a drawing for explaining a method of providing a multi-screen according to one embodiment.

[0038] According to FIG. 1, the electronic device (100) can be implemented as a TV, but is not limited thereto, and can be implemented as various types of display devices such as a monitor, PC, kiosk, tablet PC, digital photo frame, mobile phone, HMD (Head mounted Display), NED (Near Eye Display), LFD (large format display), Digital Signage, DID (Digital Information Display), video wall, projector display, etc., or as an image processing device that provides images to a display device (e.g., set-top box, one connected box).

[0039] According to one embodiment, the electronic device (100) may provide a multi-screen as illustrated in FIG. 1. The multi-screen may provide multiple screens simultaneously to display multiple contents at the same time. For example, the multi-screen may display multiple contents received from multiple input sources at the same time on different screens.

[0040] According to one embodiment, the electronic device (100) can control multiple screens in a multi-screen environment by tracking the user's gaze. For example, the electronic device (00) can activate and control a screen selected through the user's gaze among multiple screens. The gaze tracking technology may be a technology that tracks the user's eye movements and corresponds them to a specific point on the screen.

[0041] To improve the accuracy of eye tracking in a multi-screen environment, calibration that considers the unique characteristics and positions of each individual's eyes is required. Calibration is a process designed to enhance eye tracking accuracy by accurately mapping the user's eye movements to coordinates on the screen.

[0042] FIG. 2 is a block diagram showing the configuration of an electronic device according to one embodiment.

[0043] According to FIG. 2, the electronic device (100) includes at least one processor (110), memory (120), and display (130).

[0044] According to one embodiment, the electronic device (100) may include at least one of a processor (110), a memory (120), a display (130), or a camera (140). The processor (110), the memory (120), and the display (130) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus.

[0045] In one embodiment, the hardware of the electronic device (100) being operatively coupled may mean that a direct or indirect connection between the hardware is established via wired or wireless means so that the second hardware is controlled by the first hardware among the hardware. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the hardware of FIG. 2 (e.g., at least some of the processor (110), memory (120), and display (130)) may be included in a single integrated circuit, such as a system on a chip (SoC). The type and / or number of hardware included in the electronic device (100) is not limited to that shown in FIG. 2. For example, the electronic device (100) may include only some of the hardware components shown in FIG. 2.

[0046] According to one embodiment, the processor (110) of the electronic device (100) may include hardware for processing data based on one or more instructions. The hardware for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), a graphic processing unit (GPU), a neural processing unit (NPU), and / or an application processor (AP). The number of processors (110) may be one or more. For example, the processor (110) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0047] The processor (110) can control the operations of the electronic device (100) by executing instructions stored in memory (120). For example, the processor (110) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0048] A CPU (central processing unit) is a general-purpose processor capable of performing not only general operations but also artificial intelligence operations, and it can efficiently execute complex programs through a multi-layered cache structure. The CPU is advantageous for serial processing methods, which enable the organic linkage between previous and next calculation results through sequential computation. General-purpose processors are not limited to the examples mentioned above, except for cases specified as the aforementioned CPU.

[0049] A GPU (graphic processing unit) is a processor designed for massive computations, such as floating-point operations used in graphics processing, and can perform large-scale computations in parallel by integrating a large number of cores. In particular, GPUs may be advantageous over CPUs for parallel processing methods such as convolution operations. Additionally, GPUs can be used as co-processors to complement the functions of CPUs. Processors for massive computation are not limited to the examples mentioned above, except for cases specified as GPUs.

[0050] A Neural Processing Unit (NPU) is a processor specialized for artificial intelligence computations using artificial neural networks, and each layer constituting the neural network can be implemented in hardware (e.g., silicon). In this case, since the NPU is designed specifically according to the specifications required by the vendor, it has a lower degree of flexibility compared to CPUs or GPUs, but it can efficiently process the artificial intelligence computations required by the vendor. Meanwhile, as a processor specialized for artificial intelligence computations, the NPU can be implemented in various forms such as Tensor Processing Units (TPUs), Intelligence Processing Units (IPUs), and Vision Processing Units (VPUs). Artificial intelligence processors are not limited to the examples mentioned above, except for cases specified as the aforementioned NPU.

[0051] According to one embodiment, the memory (120) of the electronic device (100) may include a hardware component for storing data and / or instructions that are input and / or output to the processor (110). Depending on the purpose of data storage, the memory (120) may be implemented in the form of a memory embedded in the electronic device (100) or in the form of a memory that is detachable from the electronic device (100). For example, data for operating the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for the expansion function of the electronic device (100) may be stored in a memory that is detachable from the electronic device (100). Meanwhile, the memory embedded in the electronic device (100) may be implemented as at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD).In addition, the memory that can be attached to and detached from the electronic device (100) can be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.) or an external memory that can be connected to a USB port (e.g., USB memory).

[0052] According to one embodiment, within the memory (120) of the electronic device (100), one or more instructions (or commands) representing operations and / or operations to be performed on data by the processor (110) may be stored. A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the electronic device (100) and / or the processor (110) may perform various operations when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. In the following, the statement that an application is installed on an electronic device (100) means that one or more instructions provided in the form of an application are stored in the memory (120) of the electronic device (100), and that the one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the electronic device (100)) that is executable by the processor (110) of the electronic device (100).

[0053] At least one processor (110) controls the processing of input data according to a predefined operation rule or artificial intelligence model stored in memory (120). The predefined operation rule or artificial intelligence model is characterized by being created through learning. Being created through learning means that a predefined operation rule or artificial intelligence model with desired characteristics is created by applying a learning algorithm to a number of learning data. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is performed, or it may be performed through a separate server / system.

[0054] According to one embodiment, a display (130) of an electronic device (100) can output visualized information to a user. For example, the display (130) can be controlled by a controller, such as a GPU (graphic processing unit), to output visualized information to a user. The display (130) may include LED (Light Emitting Diodes), micro LED, Mini LED, OLED (Organic Light Emitting Diodes) display, LCD (Liquid Crystal Display), PDP (Plasma Display Panel), QD (Quantum dot) display and / or QLED (Quantum dot light-emitting diodes). According to one example, the display (130) may be implemented as a flat display, a curved display, a folding and / or rolling flexible display.

[0055] According to one embodiment, a camera (140) of an electronic device (100) can convert a captured image into an electrical signal and generate image data based on the converted signal. For example, the camera (140) may include at least one of an RGB camera, a depth camera, and an ultra-wide angle camera.

[0056] A communication circuit (150) of an electronic device (100) according to one embodiment may include hardware for supporting the transmission and / or reception of electrical signals between the electronic device (100) and an external device (e.g., a server). For example, the communication circuit (150) may communicate with an external device, an external storage medium (e.g., a USB memory stick), an external server (e.g., a web hard drive), etc., through a communication method such as Bluetooth, AP-based Wi-Fi (Wi-Fi, Wireless LAN network), Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), Optical, Coaxial, etc. According to one example, the communication circuit (150) can communicate with other electronic devices, external servers and / or remote control devices, etc.

[0057] A speaker (160) of an electronic device (100) according to one embodiment may be configured to output various audio data as well as various notification sounds or voice messages. A processor (110) may control the speaker (160) to output feedback or various notifications in the form of audio according to various embodiments of the present disclosure.

[0058] The user interface (170) of the electronic device (100) according to one embodiment may be implemented as a device such as a button, touch pad, mouse, and keyboard, or as a touch screen capable of performing the display function and operation input function described above.

[0059] A sensor (180) of an electronic device (100) according to one embodiment can sense various information. The sensor (170) can be implemented as various types of sensors. For example, the sensor (170) may include at least one sensor among a time of flight (ToF) sensor, an ultrasonic sensor, a radio detection and ranging (RADAR) sensor, a proximity sensor, a passive infrared (PIR) sensor, a pinhole sensor, an infrared human body detection sensor, a thermal detection sensor, a light sensor, and a motion detection sensor.

[0060] In addition, the electronic device (100) may further include a microphone.

[0061] The microphone is configured to receive user voice or other sounds and convert them into audio data. However, according to another embodiment, the electronic device (100) may receive user voice input through an external device via a communication circuit (150).

[0062] Meanwhile, depending on the implementation example of the electronic device (100), a speaker, a tuner, and a demodulator may be additionally included. The tuner (not shown) can receive RF (Radio Frequency) broadcast signals by tuning a channel selected by the user or all previously stored channels among the RF broadcast signals received through the antenna. The demodulator (not shown) may receive and demodulate a digital IF signal (DIF) converted by the tuner and perform channel decoding, etc. According to one embodiment, an input video received through the tuner may be provided to the processor (110) after being processed through the demodulator (not shown).

[0063] FIG. 3 is a flowchart illustrating a method for controlling an electronic device according to one embodiment.

[0064] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0065] According to one embodiment, operations 310 to 360 can be understood as being performed in the processor (110) of the electronic device (100).

[0066] In operation 310, the electronic device (100) according to one embodiment can identify whether calibration for eye tracking is initiated. Calibration may be a process to increase the accuracy of eye tracking by accurately mapping the user's eye movements to coordinates on the screen. According to one example, calibration may be performed according to at least one of a user command or the execution of a multi-screen function. For example, when the multi-screen function is executed, the electronic device (100) may identify the user through face recognition of a captured image acquired through the camera (140), and if calibration information corresponding to the identified user is not stored in the memory (120), it may initiate calibration for eye tracking on the multi-screen.

[0067] According to one example, the electronic device (100) may initiate calibration for eye tracking in a multi-screen environment. The multi-screen may provide multiple screens simultaneously to display multiple contents at the same time. For example, the multi-screen may display multiple contents received from multiple input sources at the same time on different screens. For example, the electronic device (100) may initiate calibration by considering the entire screen of the display (110) as a single area.

[0068] When calibration is started (310:Y), in operation 320, the electronic device (100) according to one embodiment can identify the display location of a calibration point based on a dividing line corresponding to a plurality of screens included in a multi-screen. The calibration point may be a reference point displayed on the screen to accurately measure the user's eye movements in an eye-tracking system.

[0069] According to one example, the electronic device (100) can identify the display location of a calibration point at the intersection of a plurality of dividing lines corresponding to a plurality of screens included in a multi-screen.

[0070] According to one example, the electronic device (100) can identify the display position of a calibration point at the center of a single dividing line corresponding to a plurality of screens included in a multi-screen.

[0071] According to one example, since rectangular content may be provided on a rectangular screen, screen division may be based on two directions: horizontal and vertical. According to one example, the electronic device (100) may identify the number of calibration points based on the number obtained by subtracting 1 from the larger number between the number of first divided screens in the horizontal direction and the number of second divided screens in the vertical direction among a plurality of screens included in a multi-screen. For example, the number of calibration points may be identified based on the following mathematical formula 1.

[0072] [Mathematical Formula 1]

[0073] Number of calibration points = MAX(Number of horizontal split screens, Number of vertical split screens) - 1

[0074] According to one example, by setting calibration points based on screen split lines included in a multi-screen, it is possible to increase eye tracking accuracy while reducing the number of calibration points.

[0075] In operation 330, the electronic device (100) according to one embodiment may display a calibration point on the display (130). For example, the electronic device (100) may display a calibration point at a screen location identified in operation 420.

[0076] In operation 340, an electronic device (100) according to one embodiment can identify the gaze of a user looking at a calibration point based on a captured image acquired through a camera (140). For example, the electronic device (100) can simultaneously display calibration points at a plurality of calibration locations identified on a screen and identify the gaze of a user looking at each calibration point. For example, the electronic device (100) can sequentially display calibration points at a plurality of calibration locations identified on a screen and identify the gaze of a user looking at each calibration point. For example, the electronic device (100) can provide a guide that guides at least one of the method of looking at the calibration point and the order of looking. For example, the electronic device (100) can guide the user to look at each calibration point in order.

[0077] In operation 350, the electronic device (100) according to one embodiment can store calibration information obtained based on the user's line of sight and the display position of the calibration point.

[0078] According to one example, the electronic device (100) may store the acquired calibration information in memory (120) by mapping the location of each calibration point and the user's gaze information. For example, the electronic device (100) may generate a calibration table (or calibration matrix) by mapping the screen coordinates of each calibration point and the user's gaze position and store it in memory (120). For example, the user's gaze information may identify gaze information or a confidence score for gaze information, including at least one of 2D gaze coordinates, pupil size, or head pose estimation.

[0079] In operation 360, when a multi-screen is provided to an upper display (130), the electronic device (100) according to one embodiment can perform eye tracking of a plurality of screens included in the multi-screen based on calibration information stored in memory (120).

[0080] According to one example, the electronic device (100) can acquire user gaze information by inputting a captured image of a user into an artificial intelligence model. For example, the electronic device (100) may preprocess the captured image and input the preprocessed image into the artificial intelligence model, or input feature information obtained from the preprocessed image into the artificial intelligence model. Preprocessing of the captured image may include at least one of face detection, eye region segmentation, or light removal. The feature information may include at least one of pupil detection, iris and eyelid features, or eye periphery contour information.

[0081] According to one embodiment, an artificial intelligence model can process input data to output user gaze information. For example, the gaze position information may identify gaze information including at least one of 2D gaze coordinates, 3D gaze vector, fixation and gaze time, eye movement data, pupil size, blink information, and head pose estimation, and a confidence score for the gaze information. For example, the head pose information may be used as an auxiliary tool for gaze estimation by outputting the position and orientation of the head as 3D coordinates.

[0082] According to one example, an artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs the layer's operation through the result of the operation of the previous layer and at least one defined operation. Examples of neural networks include convolutional neural networks (CNN), recurrent neural networks (RNN), deep neural networks (DNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN), deep Q-networks, and Transformers, and the neural networks in this disclosure are not limited to the examples described above except where specified.

[0083] A learning algorithm is a method of training a specific target device (e.g., a robot) using a number of learning data to enable the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithms in this disclosure are not limited to the aforementioned examples except where specified.

[0084] According to one embodiment, the electronic device (100) can perform eye tracking of a user on a plurality of screens included in a multi-screen based on eye information output from an artificial intelligence model and calibration information stored in memory (120). According to one example, when 2D eye coordinates are output from an artificial intelligence model, the electronic device (100) can identify the screen that the user is looking at among the plurality of screens included in the multi-screen by comparing the user's eye position (or eye coordinates) corresponding to the screen coordinates of each calibration point included in the calibration information.

[0085]

[0086] According to one embodiment, when a multi-screen is provided on a display (130), the electronic device (100) can identify a threshold range of eye-tracking disabled areas based on the position of a dividing line included in the multi-screen. According to one example, the electronic device (100) can ignore the user's gaze if the user's gaze is located in the eye-tracking disabled area.

[0087] According to one embodiment, the electronic device (100) can play a first image of a first screen corresponding to the first area when the user's gaze is located in a first area outside the gaze tracking disabled area. According to one example, the electronic device (100) can play a first image of a screen corresponding to the second area when the user's gaze moves from the first area to a second area outside the gaze tracking disabled area.

[0088] According to one embodiment, when a multi-screen is provided on a display (130), the electronic device (100) can disable the remaining screens among the multiple screens included in the multi-screen, excluding the first screen where the user's gaze is located. According to one example, the electronic device (100) can disable the remaining screens by adjusting at least one of the image brightness or sound output from the remaining screens among the multiple screens. According to one example, the electronic device (100) can disable the remaining screens by reducing the image brightness output from the remaining screens among the multiple screens to a level below a preset brightness and reducing the sound level (or volume level) to a level below a preset size. For example, at least one of the preset brightness and preset size may be preset during the manufacture of the electronic device (100) or may be set by user input. For example, at least one of the preset brightness and preset size may be changed by user input.

[0089] An electronic device (100) according to one embodiment can monitor events occurring on the remaining screens that are deactivated while the remaining screens, excluding the first screen among the plurality of screens, are deactivated. According to one example, if a sound of a size greater than or equal to a set level is detected on the second screen among the remaining screens, the electronic device (100) can activate the second screen and play the video on the second screen. For example, the electronic device (100) can maintain the activation state of the first screen when the second screen is activated.

[0090] For example, if a soccer match video is playing on the second screen, the audio volume may momentarily increase when a "goal" occurs. In this case, the second screen can be activated to play the video on the second screen.

[0091] According to one embodiment, the electronic device (100) can activate the remaining screens when another user is identified based on a captured image obtained through a camera (140) while the remaining screens, excluding the first screen among the plurality of screens, are deactivated.

[0092] FIGS. 4a and FIGS. 4b are drawings for explaining a method for identifying calibration points according to one embodiment.

[0093] According to one embodiment, the electronic device (100) can identify the display position of a calibration point based on a dividing line corresponding to a plurality of screens included in a multi-screen.

[0094] According to one example, when a first screen (411) and a second screen (412) are arranged in a left-right direction on a multi-screen as illustrated in FIG. 4a, the electronic device (100) can identify the display location of a calibration point based on a vertical dividing line (413) connecting the dividing points of the first screen (411) and the second screen (412). For example, the electronic device (100) can identify the center point (414) of the vertical dividing line (413) as the display location of the calibration point.

[0095] According to one example, when calibration is performed at a calibration point (414) shown in FIG. 4a, the electronic device (100) can input a captured image of a user gazing at the calibration point (414) into a learned artificial intelligence model and store the acquired gaze coordinates (e.g., x-coordinates) in memory (120).

[0096] Afterward, when the electronic device (100) is provided with a multi-screen including a left screen (411) and a right screen (412) as shown in the uppermost drawing of FIG. 4a as the multi-screen function is executed, the electronic device (100) can identify the screen where the user's gaze is located on the multi-screen based on the stored gaze coordinates (e.g., x coordinates). For example, the electronic device (100) can input a user-captured image into an artificial intelligence model to obtain gaze coordinates, and if the obtained gaze coordinates are to the left of the stored x coordinates, it can identify that the gaze is located on the left screen, and if the obtained gaze coordinates are to the right of the stored x coordinates, it can identify that the gaze is located on the left screen. For example, assuming the case where the previously stored x coordinate corresponding to the calibration point (414) is 40, the user-captured image is input into an artificial intelligence model, and if the obtained gaze coordinates are x coordinates = 30, it can be determined that the user is looking at the left screen, and if x coordinates = 45, it can be determined that the user is looking at the right screen.

[0097] According to one example, as illustrated in FIG. 4b, when a first screen (421), a second screen (422), a third screen (423), and a fourth screen (424) are arranged vertically and horizontally on a multi-screen, the electronic device (100) can identify the display location of a calibration point based on a horizontal dividing line (425) and a vertical dividing line (426) connecting the dividing points of the first screen (421), the second screen (422), the third screen (423), and the fourth screen (424). For example, the electronic device (100) can identify the intersection point (427) of the horizontal dividing line (425) and the vertical dividing line (426) as the display location of the calibration point.

[0098] According to one example, when calibration is performed at a calibration point (427) shown in FIG. 4a, the electronic device (100) can input a captured image of a user gazing at the calibration point (427) into a learned artificial intelligence model and store the acquired gaze coordinates (e.g., x coordinate and y coordinate) in memory (120).

[0099] Afterward, when the electronic device (100) is provided with a multi-screen including four screens (421, 422, 423, 424) as shown in the uppermost drawing of FIG. 4b as the multi-screen function is executed, the electronic device (100) can identify the screen where the user's gaze is located on the multi-screen based on stored gaze coordinates (e.g., x coordinate and y coordinate). For example, the electronic device (100) can input a captured image of the user into an artificial intelligence model to obtain gaze coordinates, and identify the screen where the user's gaze is located among the four screens (421, 422, 423, 424) based on the x and y coordinates where the obtained gaze coordinates are stored.

[0100] FIGS. 5A and FIGS. 5B are drawings for explaining a method for identifying calibration points according to one embodiment.

[0101] Referring to the first drawing of FIG. 5a, when a vertical dividing line (511) is identified based on a plurality of screens included in a multi-screen, the electronic device (100) can identify the center point (512) of the vertical dividing line (511) as the display position of the calibration point.

[0102] Referring to the second drawing of FIG. 5a, when a horizontal dividing line (521) is identified based on a plurality of screens included in a multi-screen, the electronic device (100) can identify the center point (522) of the horizontal dividing line (521) as the display position of the calibration point.

[0103] Referring to the third drawing of FIG. 5a, when the electronic device (100) identifies a vertical dividing line (531) and a horizontal dividing line (532) based on a plurality of screens included in a multi-screen, the intersection point (533) of the vertical dividing line (531) and the horizontal dividing line (532) can be identified as the display position of the calibration point.

[0104] Referring to the first drawing of FIG. 5b, when a plurality of vertical dividing lines (541, 543) are identified based on a plurality of screens included in a multi-screen, the electronic device (100) can identify different points (542, 544) on the plurality of vertical dividing lines (541, 543) as display positions of a plurality of calibration points.

[0105] Referring to the second drawing of FIG. 5b, when the electronic device (100) identifies a plurality of vertical dividing lines (551, 552) and one horizontal dividing line (553) based on a plurality of screens included in a multi-screen, the intersection points (554, 555) of each of the plurality of vertical dividing lines (551, 552) and the horizontal dividing line (553) can be identified as display positions of a plurality of calibration points.

[0106] Referring to the third figure of FIG. 5b, when an electronic device (100) identifies one vertical dividing line (561) and a plurality of horizontal dividing lines (562, 563) based on a plurality of screens included in a multi-screen, it can identify the intersection points (564, 565) of each of the plurality of horizontal dividing lines (562, 563) and the vertical dividing line (561) as display positions of a plurality of calibration points.

[0107] Referring to the fourth figure of FIG. 5b, when a plurality of horizontal dividing lines (571, 573) are identified based on a plurality of screens included in a multi-screen, the electronic device (100) can identify different points (572, 574) on the plurality of horizontal dividing lines (571, 573) as display positions of a plurality of calibration points.

[0108] According to one example, when calibration is performed at a plurality of calibration points (542, 544) shown in the first drawing of FIG. 5b, the electronic device (100) may input a captured image of a user gazing at each of the plurality of calibration points (542, 544) into a learned artificial intelligence model and store the acquired plurality of gaze coordinates (e.g., x1 coordinate and x2 coordinate) in memory (120).

[0109] Afterward, when the electronic device (100) is provided with a multi-screen display (130) containing three screens arranged horizontally (e.g., left screen, center screen, and right screen) as the multi-screen function is executed, it can identify the screen where the user's gaze is located on the multi-screen based on a plurality of stored gaze coordinates (e.g., x1 coordinate and x2 coordinate). For example, let us assume a case where the stored x-coordinate corresponding to the first calibration point (542) is -20 and the stored x-coordinate corresponding to the second calibration point (544) is +25. In this case, it can be determined that the user is looking at the left screen among the three screens if the x-coordinate obtained by inputting the user's captured video into an artificial intelligence model is a value smaller than -20, the center screen if it is greater than -20 and less than 25, and the right screen if it is greater than 25.

[0110] As described above, in most multi-screen cases, eye tracking control can be achieved with only one or two calibration points.

[0111] FIGS. 6a to 6c are drawings for explaining a gaze tracking method according to one embodiment.

[0112] According to one embodiment, a threshold range of eye tracking disable areas can be identified based on the location of a dividing line identified based on a plurality of screens included in a multi-screen. According to one example, the electronic device (100) can ignore the user's gaze when the user's gaze is located in the eye tracking disable area.

[0113] According to one example, as shown in FIG. 6a, an electronic device (100) can identify a screen (612) where the user (10) is looking, identified based on a captured image obtained through a camera (140) among a plurality of screens (611, 612, 613, 614) included in a multi-screen, as a screen selected by the user. For example, the electronic device (100) can play the image of the screen (612) where the user is looking (or is focused by the user's gaze) and disable the remaining screens (611, 613, 614). For example, the electronic device (100) can disable the remaining screens by reducing the brightness of the image output from the remaining screens (611, 613, 614) to a level below a preset brightness and reducing the sound volume to a level below a preset volume.

[0114] According to one example, as shown in FIG. 6a, a threshold range of eye tracking disable areas (621, 622) can be identified based on the location of a plurality of dividing lines identified based on a plurality of screens (611, 612, 613, 614). For example, the electronic device (100) can ignore the gaze of the user (10) when the user's gaze is located in the eye tracking disable area (621, 622).

[0115] According to one example, as shown in FIG. 6b, when the user (10)’s gaze is positioned on one of the disabled screens (611, 613, 614) (611), the electronic device (100) can play the image of the screen (611) selected by the user (10)’s gaze as shown in FIG. 6c and disable the screen (612) where the image was previously played. For example, the electronic device (100) can perform an action corresponding to the user (10)’s gaze position only when the user (10)’s gaze is positioned in an area that is not a gaze tracking disabled area (621, 622).

[0116] According to the above-described embodiment, the eye tracking disabled area is distinguished as a buffer area, thereby preventing unwanted changes in eye focus by the user.

[0117] FIGS. 7a to 7c are drawings for explaining a gaze tracking method according to one embodiment.

[0118] An electronic device (100) according to one embodiment can monitor events occurring on the remaining screens that are deactivated while the remaining screens, excluding the first screen among the plurality of screens, are deactivated. According to one example, if an event is identified on the second screen among the remaining screens, the electronic device (100) can activate the second screen and play the video on the second screen. For example, if a sound of a size greater than a preset level is output on the second screen, the electronic device (100) can activate the second screen and play the video on the second screen.

[0119] According to one example, as shown in FIG. 7a, the electronic device (100) tracks the gaze of the user (10) based on a captured image obtained through a camera (140) so that the image is played on the first screen (711) where the user (10)'s gaze is located and the remaining screens (712, 713, 714) are disabled.

[0120] According to one example, as shown in FIG. 7b, the electronic device (100) can activate the second screen (712) and play an image of the second screen when a sound of a size greater than a preset size is output from the second screen (712).

[0121] According to FIG. 7c, for example, the electronic device (100) can continue to activate the second screen (712) to play an image when the user (10)’s gaze is positioned on the activated first screen (712). For example, the electronic device (100) can reactivate the second screen (712) when the user (10)’s gaze is continuously positioned on the first screen (711).

[0122] According to the above-described embodiment, while the user is immersed in the video of a specific screen on a multi-screen, they do not miss events (e.g., notifications) occurring on other inactive screens.

[0123] FIGS. 8A and FIGS. 8B are drawings for explaining a method of using calibration information according to one embodiment.

[0124] According to one embodiment, when the multi-screen function is executed, the electronic device (100) can identify a user through face recognition of a captured image obtained through a camera (140) and identify whether calibration information corresponding to the identified user is stored in memory (120). According to one example, if the calibration information corresponding to the user is not stored in memory (120), the electronic device (100) can start calibration for eye tracking on the multi-screen. According to one example, if the calibration information corresponding to the user is stored in memory (120), the electronic device (100) can execute the multi-screen function by tracking the user's gaze.

[0125] According to FIGS. 8a and 8b, in one example, when a user is identified through face recognition in a captured image acquired through a camera (140), the electronic device (100) can identify whether calibration information corresponding to the identified user is stored in memory (120). In one example, calibration information corresponding to a plurality of users (812, 813, 814, 815, 816, 817) may be stored in memory (120).

[0126] According to one example, as shown in FIG. 8a, when the electronic device (100) identifies the first user (811) in the captured image, calibration can be performed because calibration information corresponding to the first user (811) is not stored in the memory (120).

[0127] According to one example, as shown in FIG. 8b, when a second user (813) is identified in a captured image, the electronic device (100) can execute a multi-screen function by tracking the gaze of the second user (813) based on the calibration information stored in the memory (120), since calibration information corresponding to the second user (813) is stored in the memory (120).

[0128] FIG. 9 is a drawing for explaining a method of providing eye tracking in a multi-screen according to one embodiment.

[0129] According to one embodiment, the electronic device (100) can activate the remaining screens when another user is identified based on a captured image obtained through a camera (140) while the remaining screens, excluding the first screen among the multiple screens included in the multi-screen, are deactivated.

[0130] According to one example, as shown in FIG. 9, the electronic device (100) can track the gaze of the user (911) based on a captured image obtained through a camera (140) and disable the remaining screens among the multiple screens included in the multi-screen, excluding the screen where the user (911) is looking.

[0131] According to one example, as shown in FIG. 9, the electronic device (100) can activate the remaining screens when other users (912, 913) other than the user (911) are identified.

[0132] According to the various embodiments described above, user convenience can be improved by reducing the number of calibration points for eye tracking in a multi-screen environment.

[0133] In a multi-screen environment, screens other than the one focused by the user's gaze can be disabled to enhance user concentration on the focused screen, while continuously monitoring events occurring on the disabled screens to provide notifications about those events.

[0134] By providing an eye-tracking disabled area within the boundary regions of multiple screens included in a multi-screen setup, the problem of the desired screen not being focused by the user's gaze can be reduced.

[0135] The methods according to the various embodiments of the present disclosure described above may be implemented in the form of an application that can be installed on an existing electronic device. Alternatively, the methods according to the various embodiments of the present disclosure described above may be performed using a deep learning-based artificial neural network (or deep artificial neural network), that is, a learning network model.

[0136] The methods according to the various embodiments of the present disclosure described above can be implemented by software upgrades or hardware upgrades alone for existing electronic devices.

[0137] The various embodiments of the present disclosure described above may also be performed through an embedded server equipped in an electronic device or an external server of the electronic device.

[0138] According to a specific example of the present disclosure, the various embodiments described above may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions. When instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or by using other components under the control of the processor. Instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means only that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

[0139] Additionally, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0140] Additionally, each component (e.g., module or program) according to the various embodiments described above may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in the various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the functions performed by each of the respective components prior to integration in the same or similar manner. The operations performed by the module, program, or other components according to the various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added.

[0141] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In an electronic device, display; camera; Memory for storing instructions; and One or more processors including processing circuitry; and The above one or more processors, When the above instructions are executed individually or collectively, the electronic device, When calibration for eye tracking begins, the display position of the calibration point is identified based on dividing lines corresponding to multiple screens included in the multi-screen, and Based on the identified display location, the calibration point is displayed on the display, and When the gaze of the user looking at the calibration point is identified based on the captured image acquired through the camera, calibration information acquired based on the identified gaze of the user and the display position of the calibration point is stored in the memory, and An electronic device that, when the multi-screen is provided to the display, performs eye tracking of the user for the plurality of screens included in the multi-screen based on the calibration information.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Identifying calibration points at the intersection of multiple dividing lines corresponding to the multiple screens included in the multi-screen, or An electronic device for identifying a calibration point at the center of a single dividing line corresponding to the plurality of screens.

3. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device for identifying the number of calibration points based on the number obtained by subtracting 1 from the larger number between the number of a first divided screen in the horizontal direction and the number of a second divided screen in the vertical direction among the plurality of screens.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When the above multi-screen is provided to the display, it is identified as a threshold range eye-tracking disabled area based on the position of the dividing line included in the multi-screen, and An electronic device that ignores the gaze of the user when the user's gaze is located in the gaze tracking disabled area.

5. In Paragraph 4, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When the user's gaze is located in a first area outside the aforementioned eye tracking disabled area, a first image of a first screen corresponding to the first area is played, and An electronic device that plays a first image of a screen corresponding to the second area when the user's gaze moves from the first area to a second area outside the gaze tracking disable area.

6. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that, when the above multi-screen is provided to the display, disables the remaining screens among the plurality of screens included in the multi-screen, excluding the first screen where the user's gaze is located.

7. In Paragraph 6, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, While the remaining screens excluding the first screen are disabled, the event occurring on the disabled remaining screens is monitored, and An electronic device that, when an event is identified on the second screen among the remaining screens, activates the second screen to play an image on the second screen.

8. In Paragraph 6, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that reduces the image brightness of the remaining screen among the plurality of screens to a preset brightness level and reduces the sound level to a preset level to disable the remaining screen.

9. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that enables the remaining screens, excluding the first screen, when another user is identified based on a captured image obtained through the camera while the remaining screens are deactivated.

10. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When the multi-screen function is executed, the user is identified through face recognition of the captured video acquired through the camera, and An electronic device that initiates the calibration for eye tracking on the multi-screen when calibration information corresponding to the identified user is not stored in the memory.

11. An operation to identify the display position of a calibration point based on a dividing line corresponding to a plurality of screens included in a multi-screen when calibration for eye tracking begins; An operation of displaying the calibration point based on the identified display position; When the gaze of the user looking at the calibration point is identified based on the captured image acquired through the camera, the operation of storing calibration information acquired based on the identified gaze of the user and the display position of the calibration point; and A control method comprising: an operation to perform eye tracking of the user for the plurality of screens included in the multi-screen based on calibration information when the multi-screen is provided.

12. In Paragraph 11, The operation of identifying the display position of the above calibration point is, Identifying calibration points at the intersection of multiple dividing lines corresponding to the multiple screens included in the multi-screen, or A control method comprising: an operation of identifying a calibration point at the center of a single dividing line corresponding to the plurality of screens.

13. In Paragraph 11, The operation of identifying the display position of the above calibration point is, A control method comprising: identifying the number of calibration points based on the larger number between the number of a first divided screen in the horizontal direction and the number of a second divided screen in the vertical direction among the plurality of screens, by subtracting 1.

14. In Paragraph 11, When the above multi-screen is provided to the display, the operation of identifying a threshold range of eye-tracking disabled areas based on the position of a dividing line included in the multi-screen; and A control method further comprising: an action of ignoring the user's gaze when the user's gaze is located in the gaze tracking disabled area.

15. A non-transient computer-readable medium storing computer instructions that cause said electronic device to perform an operation when executed by a processor of said electronic device, When calibration for eye tracking begins, an operation to identify the display position of a calibration point based on dividing lines corresponding to multiple screens included in a multi-screen; An operation of displaying the calibration point based on the identified display position; When the gaze of the user looking at the calibration point is identified based on the captured image acquired through the camera, the operation of storing calibration information acquired based on the identified gaze of the user and the display position of the calibration point; and A non-transient computer-readable medium comprising: an operation of performing eye tracking of the user for the plurality of screens included in the multi-screen based on calibration information when the multi-screen is provided.

Images