Display device and control method therefor

Abstract

This display device: receives, from an external device and through a communication circuit, first sensing data generated by a second motion of the external device which is in accordance with a first motion of a living body; obtains living body motion information corresponding to the first motion on the basis of the first sensing data; obtains cursor movement information corresponding to the living body motion information; and controls a display so as to display a screen on which a cursor is moved from a first position to a second position on the basis of the cursor movement information.

Description

Display device and control method thereof

[0001] The present disclosure relates to a display device and a method for controlling the same, and more specifically, to a display device and a method for controlling the same that display a screen in which a cursor is moved based on biological motion of the display device.

[0002] With the advancement of electronic technology, the development of technology for various types of electronic devices is becoming active. Among these, electronic devices that analyze and provide users with their physical data, such as wearable devices or display devices, are also advancing.

[0003] In particular, various electronic devices capable of providing control functions based on the body movements of a user wearing a wearable device are being developed recently. For example, a wearable device that can be used with a specific electronic device can detect the user's body movements, and the electronic device can provide the user with functions corresponding to the user's body movements by using the sensing data acquired through detection by the wearable device.

[0004] A display device is provided. The display device includes at least one processor comprising a memory for storing instructions, a communication circuit, a display for displaying a cursor, and a processing circuitry. When the instructions are executed individually or collectively by the at least one processor, the display device may receive, through the communication circuit, first sensing data generated by a second motion of the external device corresponding to a first motion of the biological body from an external device, and based on the first sensing data, obtain biological motion information corresponding to the first motion, obtain cursor movement information corresponding to the biological motion information, and control the display to display a screen in which the cursor has moved from a first position to a second position based on the cursor movement information.

[0005] The display device further includes a UWB sensor, and when the instructions are executed individually or collectively by the at least one processor, the display device may obtain the distance between the external device and the display device from the UWB sensor, and if the distance is determined to be less than a predetermined distance, receive the first sensing data through the communication circuit.

[0006] The display device further includes at least one motion sensor for sensing a third motion of the display device, and when the instructions are executed individually or collectively by the at least one processor, the display device may acquire the bio-motion information when it is identified that the second sensing data acquired through the at least one motion sensor matches the first sensing data.

[0007] When the above instructions are executed individually or collectively by the at least one processor, the display device may obtain the bio-motion information from a first neural network model trained to output bio-motion information according to previous sensing data and a first previous motion of the bio-.

[0008] The first neural network model is trained to output bio-motion information based on the previous sensing data, a second previous motion, and the previous sensing data may include a first previous sensing data obtained by sensing the second previous motion of the external device and a second previous sensing data obtained by sensing the first previous motion.

[0009] The second prior sensing data includes image data obtained by capturing the first prior motion multiple times in multiple directions, and the first neural network model can be trained to output bio-motion information including a direction corresponding to the second prior motion among the multiple directions.

[0010] When the above instructions are executed individually or collectively by the at least one processor, the display device may obtain the cursor movement information from a second neural network model trained to output cursor movement information according to previous sensing data and a first previous motion of the biological body.

[0011] The second neural network model above can be trained to output cursor movement information according to the first previous motion of the test cursor and the biological body, based on previous sensing data obtained by sensing the second previous motion of the external device corresponding to the movement of the test cursor displayed on the display.

[0012] The display device further includes a UWB sensor, and when the instructions are executed individually or collectively by the at least one processor, the display device may acquire the distance between the external device and the display device based on first initial sensing data acquired through the UWB sensor, identify whether the cursor is activated based on whether the distance is less than a predetermined distance, and receive the first sensing data when the cursor is identified as activated.

[0013] The display device further includes at least one motion sensor for sensing a third motion of the display device, and when the instructions are executed individually or collectively by the at least one processor, the display device receives, through the communication circuit, second initial sensing data generated by a gesture for activating the cursor from the external device when it is determined that the distance is less than a predetermined distance, and when the third initial sensing data obtained by sensing a motion corresponding to the gesture through the at least one motion sensor and the second initial sensing data are identified as matching a predetermined gesture pattern, the cursor is identified as being activated and the display can be controlled to display the cursor.

[0014] A method for controlling a display device is provided. The control method may include the steps of receiving first sensing data generated by a second motion of the external device corresponding to a first motion of the biological body from an external device, acquiring biological motion information corresponding to the first motion based on the first sensing data, acquiring cursor movement information corresponding to the biological motion information, and displaying a screen in which the cursor is moved from a first position to a second position based on the cursor movement information.

[0015] The step of receiving the first sensing data involves obtaining the distance between the external device and the display device from the UWB sensor of the display device, and if the distance is determined to be less than a predetermined distance, the first sensing data can be received.

[0016] The step of acquiring the above bio-motion information may be performed by acquiring the bio-motion information when the second sensing data acquired through at least one motion sensor sensing the third motion of the display device is identified as matching the first sensing data.

[0017] The step of acquiring the above bio-motion information may be to acquire the bio-motion information from a first neural network model trained to output bio-motion information according to previous sensing data and a first previous motion of the bio-.

[0018] The first neural network model is trained to output bio-motion information based on the previous sensing data, a second previous motion, and the previous sensing data may include a first previous sensing data obtained by sensing the second previous motion of the external device and a second previous sensing data obtained by sensing the first previous motion.

[0019] The second prior sensing data includes image data obtained by capturing the first prior motion multiple times in multiple directions, and the first neural network model can be trained to output bio-motion information including a direction corresponding to the second prior motion among the multiple directions.

[0020] The step of acquiring the cursor movement information may be performed by acquiring the cursor movement information from a second neural network model trained to output cursor movement information according to previous sensing data and a first previous motion of the biological body.

[0021] The second neural network model above can be trained to output cursor movement information according to the first previous motion of the test cursor and the biological body, based on previous sensing data obtained by sensing the second previous motion of the external device corresponding to the movement of the test cursor displayed on the display device.

[0022] The method further includes the step of obtaining a distance between the external device and the display device based on first initial sensing data obtained through a UWB sensor of the display device, and the step of identifying whether the cursor is activated based on whether the distance is less than a predetermined distance, wherein the step of receiving the first sensing data can receive the first sensing data when the cursor is identified as being activated.

[0023] A non-transient computer-readable recording medium is provided for storing computer instructions that cause the display device to perform an operation when executed by a processor of the display device. The operation may include receiving first sensing data generated by a second motion of the external device corresponding to a first motion of the biological body from an external device; acquiring biological motion information corresponding to the first motion based on the first sensing data; acquiring cursor movement information corresponding to the biological motion information; and displaying a screen in which the cursor is moved from a first position to a second position based on the cursor movement information.

[0024]

[0025] FIG. 1 is a diagram for schematically explaining the operation of a display device according to one or more embodiments of the present disclosure.

[0026] FIG. 2 is a block diagram for explaining the configuration of a display device according to one or more embodiments of the present disclosure.

[0027] FIG. 3 is a detailed block diagram for explaining the detailed configuration of a display device according to one or more embodiments of the present disclosure.

[0028] FIG. 4 is a drawing for explaining the operation of an electronic device according to one or more embodiments of the present disclosure.

[0029] FIG. 5 is a drawing for explaining sensing data according to one or more embodiments of the present disclosure.

[0030] FIG. 6 is a drawing for illustrating aspects of a first neural network model and a second neural network model according to one or more embodiments of the present disclosure.

[0031] FIG. 7 is a drawing for illustrating aspects of a first neural network model according to one or more embodiments of the present disclosure.

[0032] FIG. 8 is a flowchart for illustrating aspects of a first neural network model according to one or more embodiments of the present disclosure.

[0033] FIG. 9 is a drawing for illustrating aspects of a second neural network model according to one or more embodiments of the present disclosure.

[0034] FIG. 10 is a drawing for illustrating aspects of a second neural network model according to one or more embodiments of the present disclosure.

[0035] FIG. 11 is a flowchart illustrating aspects of a second neural network model according to one or more embodiments of the present disclosure.

[0036] FIG. 12 is a drawing for explaining the operation of activating a cursor according to one or more embodiments of the present disclosure.

[0037] FIG. 13 is a flowchart for explaining a control method of a display device according to one or more embodiments of the present disclosure.

[0038] The embodiments described herein are subject to various modifications and may have various forms; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals may be used for similar components.

[0039] In describing the present disclosure, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the present disclosure, such detailed description is omitted.

[0040] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concept of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to fully convey the technical concept of the present disclosure to those skilled in the art.

[0041] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of the rights. The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0042] In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0043] In the present disclosure, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0044] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0045] When it is stated that a certain component (e.g., a first component) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., a second component), it should be understood that the said certain component may be directly connected to the said other component or connected through another component (e.g., a third component).

[0046] On the other hand, when it is stated that a certain component (e.g., a first component) is "directly connected" or "directly coupled" to another component (e.g., a second component), it may be understood that no other component (e.g., a third component) exists between said certain component and said other component.

[0047] As used in this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware.

[0048] Instead, in some situations, the expression “device configured to do something” may mean that the device is “capable of doing something” in conjunction with other devices or components. For example, the phrase “processor configured (or set) to perform A, B, and C” may refer to a dedicated processor for performing those operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or application processor) capable of performing those operations by executing one or more software programs stored in a memory device.

[0049] In the embodiments, a 'module' or 'part' performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of 'modules' or a plurality of 'parts' may be integrated into at least one module and implemented by at least one processor, except for the 'module' or 'part' that needs to be implemented in specific hardware.

[0050] Meanwhile, various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present disclosure is not limited by the relative sizes or spacing depicted in the attached drawings.

[0051] Hereinafter, embodiments according to the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement them.

[0052] FIG. 1 is a drawing for explaining the operation of a display device according to one or more embodiments of the present disclosure.

[0053] According to FIG. 1, a display device (100) and an external device (10) are shown.

[0054] The display device (100) can be a device that outputs images through a display. The display device (100) can be implemented as a device such as a mobile phone or a smartphone, but is not limited thereto, and can be implemented as various types of display devices equipped with a display, such as a TV, monitor, laptop PC, tablet PC, video wall, LFD (large format display), Digital Signage, DID (Digital Information Display), projector display, kiosk, electronic whiteboard, refrigerator, air conditioner, etc. In addition, it can be implemented as various display devices such as a set-top box, PC, or home server that is used by connecting to an external display device (e.g., a TV or monitor, etc.) without directly having a display.

[0055] Alternatively, the display device (100) may not be implemented only as the various independent devices described above, but may also be implemented in the form of a display panel or module applicable to such devices.

[0056] The external device (10) can be implemented as a display device that can be used with the display device (100). For example, the external device (10) can be implemented as a wearable device.

[0057] A wearable device is a device implemented in a form that is worn by a user or attached to or inserted into the skin, and may be a smart watch, smart band, smart glass, smart ring, HMD (head mounted display), etc. However, it is not limited to these, and any device that is a display device implemented in a form that is worn by a user or attached to or inserted into the skin may be used.

[0058] When the external device (10) is implemented as a wearable device, specifically a smart ring, the external device (10) can be implemented in a form that can be worn on the user's finger.

[0059] In this case, the external device (10) can acquire sensing data by detecting the movement (arrow) of the finger (e.g., index finger) using the external device (10). Here, the movement of the finger may refer to the movement of the fingertip (20). The fingertip may refer to the end of the distal phalanx of the finger wearing the external device (10).

[0060] Here, the sensing data may correspond to data obtained by an external device (10) detecting acceleration, angular acceleration, velocity, displacement, etc., of the user's movement.

[0061] The external device (10) may correspond to a display device (100) that collects sensing data. Here, the display device (100) may be implemented as a device capable of pairing with the external device (10). Here, pairing may refer to a process in which an external device (10), such as a smart ring, enables mutual recognition and data exchange, etc., through wireless communication with a display device (100), such as a smartphone.

[0062] When the display device (100) receives data regarding the movement of a finger (arrow) from an external device (10), the display device (100) can display a screen in which the cursor (30) has been moved to correspond to the movement of the finger. Here, the data received by the display device (100) may include information regarding the direction and speed of the user's finger movement.

[0063] Here, the cursor (30) may refer to a graphic element indicating a selection location in a user interface. The cursor (30) can be moved on the screen via touch input or control by an external device.

[0064] In this way, the display device (100) can obtain information regarding the movement of the cursor (30) by processing sensing data received from an external device (10). Specific operations regarding this will be explained in detail through Fig. 2, etc., which will be described later.

[0065] FIG. 2 is a block diagram for explaining the configuration of a display device according to one or more embodiments of the present disclosure.

[0066] According to FIG. 2, the display device (100) may include a memory (110), a communication circuit (120), a display (130), and a processor (140).

[0067] As the display device (100) has been described in FIG. 1, a redundant description will be omitted.

[0068] The memory (110) is electrically connected to at least one processor (140) and can store data necessary for various embodiments of the present disclosure. For example, the memory (110) may be implemented as an internal memory such as ROM (Read-Only Memory) (e.g., EEPROM (electrically erasable programmable read-only memory)) or RAM (Random Access Memory) included in the processor (140), or it may be implemented as a memory separate from at least one processor (140).

[0069] Depending on the purpose of data storage, the memory (110) may be implemented in the form of a memory embedded in the display device (100) or in the form of a memory that can be attached to and detached from the display device (100). For example, data for driving the display device (100) may be stored in a memory embedded in the display device (100), and data for the expansion function of the display device (100) may be stored in a memory that can be attached to and detached from the display device (100). When implemented as memory embedded in a display device (100), the memory (110) may be at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD).

[0070] Meanwhile, although the display device (100) is depicted as being composed of a single memory in the illustrated example, when distinguishing between volatile memory and non-volatile memory, the display device (100) may be described as including multiple memories.

[0071] A memory (110) according to one or more embodiments may store at least one instruction. Here, the at least one instruction may correspond to at least one command that causes the display device (100) to display a screen where the cursor position has been moved. In addition, the memory (110) may store information necessary for the operation of the display device (100).

[0072] For example, the memory (110) can store at least one neural network model. The neural network model is composed of machine learning (deep learning) technology that uses an algorithm to self-classify / learn the features of input data, and element technologies that utilize machine learning algorithms to mimic functions such as cognition and judgment of the human brain.

[0073] Here, neural network models may also be referred to as learning models, AI models, deep learning models, etc.

[0074] The elemental technologies may include, for example, at least one of linguistic understanding technology that recognizes human language / characters, visual understanding technology that recognizes objects like human vision, reasoning / prediction technology that judges information to logically reason and predict, and knowledge representation technology that processes human experience information into knowledge data.

[0075] For example, the neural network model may correspond to at least one of a first neural network model and a second neural network model for displaying a screen where the cursor has moved. The first neural network model and the second neural network model will be described in detail in the following section.

[0076] The neural network model is not limited to the example described above, and the neural network model can be implemented as various models for the display device (100) to display the screen where the cursor has moved.

[0077] The communication circuit (120) can perform data communication with an external electronic device under the control of the processor (140). For example, the communication circuit (120) can communicate with an external electronic device through a network. For example, the processor (140) can receive data from an external electronic device through the communication circuit (120) and transmit data to an external electronic device through the communication circuit (120).

[0078] The communication circuit (120) may include hardware components to support the transmission and / or reception of electrical signals between the display device (100) and an external device. For example, the communication circuit (120) may support the transmission and / or reception of electrical signals based on various types of protocols such as Ethernet, LAN (local area network), WAN (wide area network), Wi-Fi (wireless fidelity), Bluetooth, BLE (bluetooth low energy), ZigBee, LTE (long term evolution), 5G NR (new radio) and / or 6G.

[0079] According to one or more embodiments, the display device (100) may receive first sensing data generated by a second motion of an external device corresponding to a first motion through a communication circuit (120). Here, the first motion may refer to a motion of a living body. For example, the motion of a living body may refer to the movement of a part of a user's living body, such as a finger, to move a cursor displayed on the display device (100).

[0080] Here, the second motion may correspond to the movement of an external device in accordance with the first motion. For example, if an external device worn on a finger moves due to the first motion of the user's finger, the second motion may refer to this movement.

[0081] Here, the first sensing data may correspond to data obtained by an external device sensing a second motion. For example, the external device may detect the movement of the external device (itself) caused by the second motion and obtain first sensing data including information regarding velocity, acceleration, displacement, etc.

[0082] Meanwhile, the display device (100) may include a display (130).

[0083] The display (130) is configured to display the operating status, notification messages, UI screens, etc. of the display device (100). The display can be implemented as various types of displays such as LCD (Liquid Crystal Display), OLED (Organic Light Emitting Diodes) display, PDP (Plasma Display Panel), etc. The display may also include a driving circuit, a backlight unit, etc., which can be implemented in forms such as a-si TFT (amorphous silicon thin film transistor), LTPS (low temperature poly silicon) TFT, OTFT (organic TFT), etc. Meanwhile, the display can be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, a three-dimensional display, etc. Alternatively, the display may be implemented with only one or more light-emitting elements.

[0084] The display device (100) can change the display state of the display according to various states, such as when the display device (100) is turned on, when it is operating normally, when there is insufficient power, or when there is an error, so that the user can intuitively understand the state of the display device (100).

[0085] The processor (140) can perform overall control operations of the display device (100).

[0086] The processor (140) may be implemented as a digital signal processor (DSP), microprocessor, or time controller (TCON) that processes digital signals. However, it is not limited thereto, and may include or be defined by one or more of a central processing unit (CPU), microcontroller unit (MCU), microprocessing unit (MPU), controller, application processor (AP), graphics-processing unit (GPU), communication processor (CP), or ARM processor. Additionally, the processor (140) may be implemented as a System on Chip (SoC) or large-scale integration (LSI) with built-in processing algorithms, or may be implemented in the form of a Field Programmable Gate Array (FPGA). Furthermore, at least one processor (140) can perform various functions by executing computer executable instructions stored in memory. Meanwhile, although FIG. 2 illustrates that the display device (100) includes only one processor, the implementation The device may also include multiple processors (e.g., CPU + GPU, CPU + DSP).

[0087] Meanwhile, when the instructions stored in the memory (110) described above are executed individually or collectively by the processor (140), the display device (100) may be made to perform the following operations.

[0088] According to one or more embodiments, the display device (100) may receive first sensing data generated by a second motion of an external device corresponding to a first motion of a living organism from an external device through a communication circuit. As this has been explained previously, a redundant explanation will be omitted.

[0089] According to one or more embodiments, the display device (100) can acquire bio-motion information corresponding to a first motion based on first sensing data. Here, since the first sensing data has been described in detail above, a redundant description is omitted.

[0090] Here, the biological motion information may correspond to motion information corresponding to the first motion. Here, since the first motion has been explained previously, a redundant explanation is omitted. The biological motion information may include physical information regarding the movement of the biological body, such as information regarding the acceleration, velocity, and displacement of the first motion of the biological body (e.g., fingertip).

[0091] Here, the displacement of the living organism may refer to the distance and direction in which the living organism has moved. For example, the displacement may be expressed as virtual coordinates based on the position of the display device (100). That is, the displacement may be expressed in the form of a vector based on the new position moved from the initial position of the living organism.

[0092] For example, the display device (100) can obtain information about the movement of the fingertip from the first sensing data about the motion of the external device. Here, the fingertip may correspond to the fingertip of the finger wearing the external device.

[0093] That is, the motion of the fingertip may not match the motion of the external device, but the display device (100) can recognize the motion of the fingertip using first sensing data for the motion of the external device.

[0094] Specifically, the display device (100) can recognize the motion of the fingertip from the motion of an external device using mapping information. For example, the display device (100) can pattern specific parameters (e.g., acceleration, velocity, tilt, etc.) included in previously acquired first sensing data (first prior sensing data) and map each of the plurality of patterns according to fingertip movement types (e.g., up, down, left, right, etc.).

[0095] Mapping information can be stored in a form where these multiple patterns are mapped according to fingertip movement types.

[0096] Here, patterning may mean that the shape of the waveform of a specific parameter is classified into a similar form from the first sensing data obtained by the external device sensing the various movements of the external device multiple times.

[0097] Meanwhile, the display device (100) can acquire bio-motion information using a neural network model learned from previous sensing data.

[0098] According to one or more embodiments, the display device (100) can acquire bio-motion information through a first neural network model. Here, the first neural network model may correspond to a model trained to output bio-motion information corresponding to a first previous motion of the body based on previous sensing data.

[0099] Here, the previous sensing data may correspond to previous data obtained by an external device (10) or other device sensing the motion of a biological object (e.g., a user's finger) prior to the current time. Here, the previous data may be used as training data to train a first neural network model.

[0100] Here, the biological motion information may correspond to motion information corresponding to the first previous motion. Here, the first previous motion may refer to the motion of the biological body prior to the current point in time.

[0101] Specifically, the first neural network model can be trained using previous sensing data (training data) obtained by sensing both the motion of an external device and the corresponding motion of a living organism.

[0102] For example, the first neural network model may correspond to a model trained to output bio-motion information corresponding to the second previous motion based on previous sensing data including the first previous sensing data and the second previous sensing data.

[0103] Here, the first prior sensing data may correspond to data obtained by sensing the second prior motion of an external device. Here, the second prior motion may correspond to the motion of the external device when the external device moves in accordance with the first prior motion described above. Here, the first prior motion may correspond to the motion of a biological body for collecting training data.

[0104] Specifically, the motion for collecting training data may correspond to a motion performed prior to the second motion described above, and may correspond to data to be input to the first neural network model during the process of training the first neural network model.

[0105] Here, the second prior sensing data may correspond to data obtained by sensing the second first prior motion. Here, the first prior motion is a motion performed prior to the first motion described above, and may correspond to a motion of a living organism for collecting training data.

[0106] Specifically, the motion for collecting training data may correspond to a motion performed prior to the first motion described above, and may correspond to data to be input to the first neural network model during the process of training the first neural network model.

[0107] For example, the user may move a biological body, such as a finger, while the display device (100) is training the first neural network model. Here, the finger may correspond to a finger wearing an external device, such as a smart ring.

[0108] The display device (100) can acquire second prior sensing data by sensing the movement of the living organism. The display device (100) can receive first prior sensing data regarding the movement of the external device according to the movement of the living organism from the external device.

[0109] The display device (100) can train a first neural network model using data including first prior sensing data and second prior sensing data as training data. Here, the first neural network model can be trained to output information about finger movements (biometric motion information). In this process, a label for the first neural network model can be obtained based on the second prior sensing data.

[0110] For example, the second prior sensing data may include image data. Here, the image data may correspond to image data obtained by capturing the first prior motion multiple times in multiple directions.

[0111] Here, each of the multiple directions may correspond to the direction in which the organism moves. For example, the multiple directions may include up, down, left, right, etc.

[0112] At this time, the display device (100) can direct the user's direction through the display (130). The display device (100) can display a UI (e.g., arrows), and the user can move their finger while looking at the UI. Accordingly, the user's fingertip can move from an initial position to a specific position according to the directed direction.

[0113] The display device (100) can acquire image data by sensing the fingertip multiple times while the fingertip moves from an initial position to a specific position. For example, the display device (100) can sense the fingertip through a camera.

[0114] Here, the display device (100) can sense the fingertip at predetermined intervals based on user operation input for capturing the fingertip, but this is merely an example and the display device (100) can sense the fingertip at predetermined intervals when it detects motion of the fingertip.

[0115] The display device (100) can sense multiple images by sensing the fingertip multiple times at predetermined intervals. Here, each of the multiple images may correspond to an image in which the fingertip is sequentially captured as the fingertip moves. The display device (100) can acquire image data containing these multiple images.

[0116] The display device (100) can obtain training data by labeling the first previous sensing data. Here, labeling may refer to the process of connecting the first previous sensing data with a correct answer value (label).

[0117] Here, the correct answer value can be obtained from image data containing multiple images of a specific direction. For example, a display device (100) can identify the direction of the fingertip by analyzing multiple images.

[0118] The display device (100) can obtain data labeled for each of the multiple directions by repeating the process described above for each of the multiple directions.

[0119] As described above, the display device (100) can train a first neural network model based on the second previous sensing data and the first previous sensing data.

[0120] The first neural network model may correspond to a model trained to output bio-motion information including a direction corresponding to a second previous motion among a plurality of directions.

[0121] The display device (100) can acquire bio-motion data corresponding to the second motion of an external device by inputting the sensing data (first sensing data) obtained by sensing the second motion into the first neural network model learned in this way.

[0122] All or part of the above-described processes may be performed on a display device and an available external device (e.g., a smart ring) or other external device (e.g., a server device, etc.).

[0123] Meanwhile, the display device (100) can obtain cursor movement information based on the bio-motion data obtained as above through a separate neural network model.

[0124] According to one or more embodiments, the display device (100) can obtain cursor movement information through a second neural network model. Here, the second neural network model may correspond to a model trained to output cursor movement information corresponding to a first previous motion of the living organism based on previous sensing data.

[0125] Here, the first previous motion of the organism may correspond to the previous motion of the organism for training the second neural network model. As this has been explained previously, a redundant explanation will be omitted.

[0126] According to one or more embodiments, the display device (100) can acquire cursor movement information corresponding to bio-motion information.

[0127] Here, cursor movement information may refer to data including information such as the starting coordinates, target coordinates, movement path, speed, and direction during the process of cursor movement.

[0128] In the starting coordinates and target coordinates, the coordinates may refer to specific locations on the display (130).

[0129] Here, the cursor can be moved by user input, etc. Here, user input may correspond to an input signal input through an external device. That is, user input may correspond to a signal input according to the motion of an external device caused by the motion of the user's biological body. Here, the external device may convert the motion of the external device into electrical data and transmit it to the display device (100) in the form of an input signal.

[0130] Meanwhile, the display device (100) can obtain cursor movement information from bio-motion information using mapping information. For example, the display device (100) can pattern specific parameters (e.g., acceleration, velocity, tilt, etc.) included in previously obtained first sensing data (first previous sensing data) and map each of the plurality of patterns according to the type of cursor movement (e.g., up, down, left, right, etc.).

[0131] Mapping information can be stored by mapping these multiple patterns according to the type of bio-motion information.

[0132] Here, patterning may mean that the shape of the waveform of a specific parameter is classified into a similar form from the first sensing data obtained by the external device sensing the various movements of the external device multiple times.

[0133] Alternatively, the display device (100) may obtain bio-motion information from the first prior sensing data and map the type of bio-motion information (e.g., type by direction of movement of the finger tip) to the type of cursor movement.

[0134] Mapping information can be stored by mapping the types of bio-motion information according to the types of bio-motion information.

[0135] Meanwhile, the display device (100) can acquire bio-motion information using a neural network model learned from previous sensing data.

[0136] According to one or more embodiments, the display device (100) can obtain cursor movement information through a second neural network model. Here, the second neural network model may correspond to a model trained to output cursor movement information corresponding to a first previous motion of the living organism based on previous sensing data.

[0137] The second neural network model may correspond to a model trained using previous sensing data and other data obtained therefrom as training data. Here, the external device may acquire previous sensing data according to the first previous motion of the biological body.

[0138] For example, if the external device moves in accordance with the first previous motion of the biological body, the external device can acquire previous sensing data by sensing the movement of the external device. As other previous sensing data has been explained previously, a redundant explanation will be omitted.

[0139] Meanwhile, cursor movement information corresponding to the first previous motion may include information regarding the (target) position, movement path, speed, etc., to which the cursor must move according to the previous motion of the living organism.

[0140] The second neural network model can be trained using training data on cursor movement to output such cursor movement information.

[0141] Here, learning data for the movement of the cursor can be obtained from the user's previous input for moving the cursor to a specific position (or direction). Here, the user's previous input may correspond to an input signal for moving a cursor temporarily displayed on a display (130), etc.

[0142] According to one or more embodiments, the second neural network model may correspond to a model trained to output cursor movement information corresponding to a first previous motion for moving and displaying a test cursor based on previous sensing data.

[0143] Here, the previous sensing data may correspond to data obtained by sensing a second previous motion of an external device. Here, the second previous motion may correspond to a previous motion corresponding to the movement of a test cursor displayed on a display device.

[0144] Here, the test cursor may refer to a cursor that can be displayed on the display (130) for the collection of training data. The display device (100) may display a screen through the display (130) in which the test cursor moves. The display device (100) may induce the user to move a biological object, such as a finger, according to the movement of the test cursor through the screen.

[0145] For example, a user may look at a screen where a test cursor is being moved and move a finger to move the test cursor as shown on the screen. Here, the finger may correspond to a finger wearing an external device. At this time, while holding the display device (100), the user may move the fingertip while in contact with the fingertip on the back of the display device (100).

[0146] Specifically, when a test cursor is moved from a specific starting position to a specific target position, the user can move their fingertip to move the cursor from that starting position to that target position. That is, the user can move their fingertip as if controlling the cursor with their fingertip by using the back of the display device (100) as if it were a trackpad (or touchpad).

[0147] The user may tap or double-tap a specific location on the back of the display device (100) before moving the fingertip. Then, the user may move the fingertip while it is in contact with the back of the display device (100) to move the test cursor. When the user finishes moving the fingertip, they may tap or double-tap again.

[0148] An external device or display device (100) can sense the motion of a fingertip to obtain previous sensing data. Specifically, the external device or display device (100) can recognize the start of the fingertip motion through a tap or double tap, and then recognize the end of the fingertip motion through a subsequent tap or double tap.

[0149] Here, an external device or display device (100) stores a pattern corresponding to a tap or double tap in advance, and when previous sensing data matching the stored pattern is acquired, it can recognize the start or end of the bio-motion.

[0150] The display device (100) can label the previous sensing data acquired by the external device or the display device (100). For example, if the display device (100) acquires previous sensing data according to a screen that guides the movement of a test cursor in a specific direction, the display device (100) can label the acquired previous sensing data in a specific direction.

[0151] The display device (100) can acquire learning data by repeatedly labeling previous sensing data for each of the multiple directions or multiple movement lengths.

[0152] The second neural network model can be trained to acquire cursor movement information corresponding to the first previous motion. For example, the cursor movement information corresponding to the first previous motion may refer to information regarding the direction, speed, distance, etc. of cursor movement determined by the characteristics of a specific user's bio-motion (movement direction, speed, distance, etc.).

[0153] That is, if the second neural network model is trained to acquire such cursor movement information, the second neural network model can acquire cursor movement information to move the cursor to an appropriate degree according to the characteristics of a specific user's bio-motion. Here, the characteristics of bio-motion can be acquired from sensing data.

[0154] According to one or more embodiments, the display device (100) can control the display (130) to display a screen in which the position of the cursor is moved from a first position to a second position based on cursor movement information.

[0155] Here, the first position is the initial coordinate prior to cursor movement, which may refer to the position displayed on the screen before the cursor moves. For example, the first position may refer to the position initially displayed on the screen when the cursor is activated. The activation of the cursor will be described in detail later in FIG. 3.

[0156] The second position is the final coordinate after cursor movement, and may refer to a newly displayed position on the screen after the cursor has moved. The display device (100) can obtain the second position by using information regarding the degree of cursor movement (movement direction and distance, etc.) included in the cursor movement information from the first position.

[0157] The screen moved to the second position may refer to an updated screen that reflects the state in which the cursor has changed from the first position to the second position according to cursor movement information. Here, the updated screen may refer to a screen that has changed from a screen where the cursor is displayed at the first position to a screen where the cursor is displayed at the second position.

[0158] As described above, the display device (100) can change the position of the cursor based on the movement of the living body wearing the external device. Even if the user does not wear the external device directly on the part of the living body (e.g., fingertip) that they actually want to move, the display device (100) can predict the movement of the part of the living body from the movement of the external device.

[0159] Since the display device (100) can move a cursor to correspond to the movement of a predicted biological part, the user can be provided with a user experience of controlling the display device (100) through a virtual trackpad even if the display device (100) is not equipped with a configuration such as a trackpad (touchpad).

[0160] Although FIG. 2 shows that the display device (100) includes only basic configurations, the display device (100) may include various additional configurations in addition to the configurations described above.

[0161] FIG. 3 is a detailed block diagram for explaining the detailed configuration of a display device according to one or more embodiments of the present disclosure.

[0162] According to FIG. 3, the display device (100) may include a memory (110), a communication circuit (120), a display (130), a processor (140), an ultra-wide band (UWB) sensor (150) and a motion sensor (160).

[0163] The display device (100) and the memory (110), communication circuit (120), display (130), and processor (140) have been described in FIG. 2, so a redundant description will be omitted.

[0164] The UWB sensor (150) may be a sensor for detecting the distance to an object around the display device (100) using a UWB signal. Here, the distance to the object may refer to the distance to an external device (or a separate UWB sensor provided in the external device). Additionally, the UWB sensor (150) may detect changes in the distance from the UWB sensor (150) to the object.

[0165] Specifically, the UWB sensor (150) can be implemented as an IR-UWB (Impulse-Radio Ultra Wideband) radar sensor. The IR-UWB radar sensor can identify the distance from the sensor (150) to the object or the location of the object by radiating an ultra-wideband impulse signal to the object and measuring the time until the signal reflected from the object is received.

[0166] That is, the UWB sensor (150) may be a radar sensor that transmits a signal using a wide frequency bandwidth (e.g., 7500 MHz, etc.) with low power and receives the reflected signal to measure the location of an object. At this time, the IR-UWB radar sensor can detect even minute changes in movement, so it can be used to measure minute movements of a person's finger wearing an external device such as a smart ring.

[0167] The display device (100) may include at least one UWB sensor (150), but is not limited thereto and may be implemented with a plurality of UWB sensors (150). For convenience, the following description assumes a case where the UWB sensor (150) is implemented with at least one UWB sensor (150).

[0168] Meanwhile, although the distance to an external device can be detected through a UWB sensor (150) in the present disclosure, this is merely one embodiment, and the movement of the body can be detected through various sensors for detecting the distance to an external device or the body, such as a distance sensor or a LiDAR sensor.

[0169] According to one or more embodiments, the display device (100) can obtain the distance between the external device and the display device (100) through a UWB sensor (150). Here, the distance between the external device and the display device (100) may correspond to the distance between the UWB sensors of the external device and the display device (100), respectively.

[0170] The display device (100) can identify whether the acquired distance is less than a predetermined distance. Here, the predetermined distance may correspond to a reference distance for determining whether a living organism wearing an external device is in contact with or located at a close distance to the display device (100).

[0171] For example, if the external device is implemented as a smart ring, the display device (100) can identify whether the fingertip portion of the finger wearing the smart ring is in contact with or located at a close distance to the back of the display device (100).

[0172] The display device (100) can receive first sensing data through the communication circuit (120) if the acquired distance is less than a predetermined distance. For example, if the acquired distance is less than a predetermined distance, the display device (100) may recognize it as being close, such as when the finger tip of a finger wearing a smart ring is in contact with the back of the display device (100) or when it is less than a preset range. The preset range may consider value-to-value analysis per unit of time, similarity of multiple values ​​over a certain period of time, or similar factors.

[0173] On the other hand, if the distance obtained by the display device (100) is greater than a predetermined distance, the fingertip of the finger wearing the smart ring can be recognized as being located at a distance from the display device (100).

[0174] For example, when the user is not holding the display device (100) (such as when the display device (100) is placed on a desk or in a pants pocket), the movement of the finger may not correspond to a movement for moving the cursor.

[0175] That is, if the external device is far away from the display device (100), the sensing data acquired by the external device (smart ring, etc.) may not be received. Through this, the display device (100) can prevent malfunctions caused by unintentional movement when the external device is located at a distance greater than a reference distance. Here, malfunction may refer to a phenomenon in which the cursor moves to an unintended position due to unintentional movement.

[0176] Meanwhile, a motion sensor (160) may refer to a sensor that detects a user's movement or change in position. The motion sensor (160) may include an IMU sensor, an infrared-based motion detection sensor, a LiDAR sensor, an ultrasonic sensor, etc. Here, an IMU (Inertial Measurement Unit) sensor is a sensor that analyzes movement and direction through inertial measurement and can be utilized as one of the various motion sensors (160).

[0177] Specifically, the IMU sensor may include an accelerometer, a gyroscope, a magnetometer, etc. The accelerometer measures the acceleration value of the user's segment (such as a body part), and the gyroscope can measure rotational speed. In addition, the magnetometer can perform the function of identifying the user's direction using the Earth's magnetic field.

[0178] The display device (100) may include at least one motion sensor (160), but is not limited thereto and may be implemented with a plurality of motion sensors (160). For convenience, the following description assumes a case where the motion sensor (160) is implemented as at least one motion sensor (160).

[0179] According to one or more embodiments, the motion sensor (160) can sense a third motion of the display device (100). For example, when a third motion (first motion) occurs due to biological motion, the display device (100) can sense the third motion.

[0180] Here, the bio-motion may correspond to tapping or swiping, etc. Here, the third motion according to the bio-motion may correspond to a motion in which the display device (100) shakes according to the bio-motion, such as tapping the back of the display device (100). However, it is not limited thereto.

[0181] Here, "swipe" may refer to a gesture of moving the screen surface by continuously pushing it in a specific direction using a fingertip or similar object. "Tapping" refers to a touch gesture of quickly pressing and releasing the screen surface for a short period of time using a fingertip or similar object. The gestures that a user can perform while the cursor at the second position is displayed are not limited to the examples described above.

[0182] The display device (100) can identify whether the first sensing data and the second sensing data match. Here, the second sensing data may correspond to data obtained by sensing a third motion through the motion sensor (160). Here, the first sensing data may correspond to data obtained through a separate motion sensor provided in an external device.

[0183] Here, the fact that the first sensing data and the second sensing data are matched may mean a state in which specific characteristics of the two sensing data (e.g., waveform pattern, frequency component, temporal change, etc.) exhibit identical or similar aspects.

[0184] For example, the display device (100) can extract a change in acceleration over time from each of the first sensing data and the second sensing data. Here, the change in acceleration over time may refer to a change during the period in which the third motion (or the first motion, etc.) occurs and ends.

[0185] The display device (100) can identify that the first sensing data and the second sensing data are matched if the change in acceleration of each of the first sensing data and the second sensing data shows a similar pattern.

[0186] Here, exhibiting a similar pattern may mean that values ​​such as dynamic time distortion (DTW) distance, correlation coefficient, or mean squared error (MSE) between signals (e.g., acceleration over time) converge to below a set threshold.

[0187] Meanwhile, if the display device (100) identifies that the first sensing data and the second sensing data match, it can obtain bio-motion information based on the first sensing data and the second sensing data.

[0188] Here, the display device (100) can obtain bio-motion information by inputting data including first sensing data and second sensing data into the first neural network model described above.

[0189] For example, the display device (100) can obtain bio-motion information corresponding to the motion of the display device (100) as well as the motion of the external device by using the second sensing data as well as the first sensing data.

[0190] As described above, by first identifying whether the second sensing data acquired by the display device (100) itself senses matches the first sensing data, it is possible to prevent the cursor from malfunctioning due to unintentional motion of the body (e.g., when the user moves their finger while not holding the display device (100)).

[0191] Meanwhile, in a normal situation (e.g., when a user touches the back of the display device (100) with a fingertip while holding the display device (100)), the display device (100) can identify that the first sensing data and the second sensing data match.

[0192] In this case, the display device (100) can recognize that the biological motion is an intentional movement and obtain biological movement information based on the first sensing data and the second sensing data.

[0193] Meanwhile, the display device (100) can switch the cursor to an active state when a specific condition is satisfied while the cursor is in a deactivated state. Here, a deactivated state of the cursor may mean a state in which the cursor is not displayed and thus functions using the cursor cannot be used.

[0194] According to one or more embodiments, the display device (100) can obtain the distance between the external device and the display device based on first initial sensing data obtained through the UWB sensor (150). Here, the first initial sensing data may correspond to sensing data obtained through the UWB sensor before there is a motion to activate the cursor.

[0195] For example, the first initial sensing data may include information regarding the time of flight (ToF), time of arrival (ToA), round trip time (RTT), etc. For example, the time of flight may refer to the time it takes for a signal transmitted from the UWB sensor (150) to be reflected by a target object (external device) and received again.

[0196] The display device (100) can obtain the distance to an external device (or a UWB sensor of an external device) by using the signal propagation time, etc.

[0197] The display device (100) can identify whether the distance between the external device and the display device (100) is less than a predetermined distance. Here, the predetermined distance has been specifically explained above, so a redundant explanation will be omitted.

[0198] The display device (100) can identify whether the cursor is activated based on whether the distance described above is less than a predetermined distance.

[0199] For example, if the distance is less than a predetermined distance, the display device (100) may receive second initial sensing data from an external device through a communication circuit (120). Here, the second initial sensing data may correspond to data generated by a gesture to activate a cursor.

[0200] For example, when a user taps (or double-taps) the back of the display device (100) with a fingertip to activate the cursor, the external device can acquire second initial sensing data by sensing the movement of the external device caused by the movement of the fingertip. The external device can transmit the acquired second initial sensing data to the display device (100).

[0201] Meanwhile, the display device (100) can identify that the cursor is activated when the third initial sensing data and the second initial sensing data are identified as matching a predetermined gesture pattern.

[0202] Here, the third initial sensing data may correspond to data obtained by sensing motion corresponding to a gesture through at least one motion sensor (160). Here, the motion corresponding to a gesture may correspond to the motion of the display device (100) according to a gesture (e.g., tapping) to activate the cursor.

[0203] The defined gesture pattern here may correspond to a pattern obtained by repeatedly sensing various types of user gestures (up, down, left, right, etc.) and analyzing the resulting sensing data. Here, the pattern may refer to aspects of specific characteristics of the sensing data (e.g., waveforms of velocity, acceleration, tilt, frequency, temporal change, etc.).

[0204] For example, the display device (100) can sense the movement of the fingertip through the motion sensor (160) when the user moves the fingertip multiple times in the up, down, left, and right directions, respectively. Additionally, an external device can sense the user's fingertip movement as described above. Accordingly, the display device (100) and the external device can each acquire previous sensing data.

[0205] The display device (100) can extract a pattern that appears in common in the previous sensing data acquired by the display device (100) and the previous sensing data acquired by the display device (100) acquired by an external device.

[0206] For example, if the display device (100) shows changes in acceleration and angular velocity over time in each of the two previously described prior sensing data multiple times in a similar pattern (waveform), it can obtain a gesture pattern (e.g., an average feature of multiple similar features) from the features of the waveforms that appear similarly multiple times. Here, since similar patterns have been explained previously, a redundant explanation will be omitted.

[0207] Meanwhile, the fact that the third initial sensing data and the second initial sensing data match a predetermined gesture pattern may mean that each of the third initial sensing data and the second initial sensing data exhibits a similar aspect to the predetermined gesture pattern.

[0208] The display device (100) can identify that the cursor is activated when it is identified that the second initial sensing data and the third initial sensing data show similar patterns.

[0209] When the display device (100) identifies that the cursor is active, it can control the display (130) to display the cursor.

[0210] For example, the display device (100) can control the display (130) to display the cursor at an initial position. For example, the display device (100) can display the cursor at the center point of the screen on a screen where the cursor is not displayed (a screen where the cursor is disabled). In this case, the display device (100) can repeat the process of removing and displaying the cursor displayed on the screen so that the user can visually recognize that the cursor is activated. However, it is not limited thereto.

[0211] Meanwhile, the display device (100) can receive first sensing data when it is identified that the cursor is activated. The display device (100) can display a screen in which the cursor has moved to a second position by performing the above-described process, such as obtaining bio-motion information from the received first sensing data.

[0212] In this way, the display device (100) can activate the cursor only when both of the two conditions are satisfied. Here, the two conditions may correspond to a condition where the distance between the display device (100) and an external device is less than a predetermined distance, and a condition where the second initial sensing data and the third initial sensing data are matched.

[0213] That is, there may be cases where there is unintentional motion of the display device (100), such as when the user does not move their fingers when holding the display device (100). If the display device (100) does not perform the initial activation process as described above, the cursor moves due to such unintentional motion, so an unintended action by the user may occur on the display device (100).

[0214] The display device (100) can prevent such unintentional actions from occurring by initially activating the cursor only when two conditions are satisfied. The display device (100) can move the cursor to a position more suitable for the user's intention by having the user perform a gesture to move the cursor to a desired position after seeing the cursor displayed at the initial position of the cursor (e.g., the center position of the screen).

[0215] Meanwhile, although FIG. 3 illustrates a display device (100) including various additional configurations, some of the illustrated configurations may be implemented in an omitted form. Additionally, other configurations that are not illustrated may also be included.

[0216] FIG. 4 is a drawing for explaining the operation of a display device according to one or more embodiments of the present disclosure.

[0217] According to FIG. 4, the first state (410) and the second state (420) of the display device (100) are shown.

[0218] The first state (410) represents the state before the cursor (413) moves, and the second state may represent the state after the cursor (423) moves.

[0219] In both the first state (410) and the second state, the external device (10) may be worn on the index finger. Here, the index finger may refer to the finger that the user moves to move the cursor through the external device (10) when controlling the display device (100) using the finger.

[0220] Figure 4 shows the case where the external device (10) is worn on the index finger, but this is merely an example and it is obvious that it can also be worn on other fingers (thumb, middle finger, ring finger, etc.).

[0221] According to the first state (410), the user can grasp the display device (100) while the fingertip (412) of the index finger is positioned at the initial point. In this case, while the fingertip (412) is positioned at the initial point, the external device (10) worn on the index finger can be positioned at the first device position (411). At this time, the display device (100) can perform the process of activating the cursor (413) as described above.

[0222] In this case, the display device (100) can display the cursor (413) at a first cursor position when the cursor is activated. Here, the first cursor position may correspond to the position where the cursor (413) is first displayed when activated.

[0223] When the user moves the index finger (or fingertip (412)) to move the cursor (413), the display device (100) can change to a second state (420).

[0224] For example, in the second state (420), after the movement of the index finger, the fingertip (412) may correspond to a state where it has moved from an initial position to a new position. Depending on the movement of the fingertip, the position of the external device (10) may move from the first device position (411) to the second device position (421).

[0225] In this case, on the screen of the display device (100), the moved cursor (423) may be displayed at a second cursor position. Here, the second position may correspond to a position on the screen that corresponds to the position of the fingertip (412) after movement.

[0226] The external device (10) senses the motion of the external device (10) while moving from the first device position (411) to the second device position (421) to obtain first sensing data, and the display device (100) can obtain fingertip movement information from the first sensing data. Here, the fingertip movement information may correspond to the aforementioned bio-motion information.

[0227] The display device (100) can obtain cursor information from the movement information of the fingertip and display a screen in which the cursor has moved from a first position to a second position according to the cursor movement information. Since the specific operations performed by the display device (100) to display the screen in which the cursor has moved to a second position have been specifically described in FIGS. 2 and 3, a redundant description will be omitted.

[0228] FIG. 5 is a drawing for explaining sensing data according to one or more embodiments of the present disclosure.

[0229] According to FIG. 5, the first sensing data (510) and the second sensing data (520) are shown.

[0230] Each of the first sensing data (510) and the second sensing data (520) may represent a change in a specific parameter value over time. For example, the specific parameter value may correspond to acceleration. Here, acceleration may be obtained from a sensor (motion sensor or UWB sensor) equipped in the display device (100) or an external device.

[0231] Meanwhile, each of the first sensing data (510) and the second sensing data (520) may exhibit similar characteristics in the motion interval (530). Here, the motion interval may refer to the interval during which the user moves the finger tip to move the cursor. The case of similar characteristics may refer to the case where the waveform of parameters, such as acceleration over time, is similar.

[0232] The display device (100) can compare the first sensing data (510) and the second sensing data (520) to determine whether they match each other. Here, if it is determined that similar patterns appear in the first sensing data (510) and the second sensing data (520) respectively during the motion interval (530), the display device (100) can determine that the first sensing data (510) and the second sensing data (520) match each other.

[0233] For example, the display device (100) can move the cursor to correspond to the motion of the user's fingertip only when it identifies that the first sensing data (510) and the second sensing data (520) are matched.

[0234] That is, the display device (100) can prevent the cursor on the display device (100) from moving due to the motion of the fingertip when the user is not using the display device (100), such as when an external device is far away from the display device (100).

[0235] Meanwhile, the display device (100) can move and display a cursor using sensing data acquired by the display device (100) and an external device. In this case, the display device (100) can acquire information about the position where the cursor is to be moved using one or more neural network models, and move and display the cursor to a position suitable for the user's motion.

[0236] This will be explained in detail in Fig. 6.

[0237] FIG. 6 is a drawing for illustrating aspects of a first neural network model and a second neural network model according to one or more embodiments of the present disclosure.

[0238] According to FIG. 6, the display device (100) can move a cursor or execute a function corresponding to a UI using a first neural network model (620) and a second neural network model (640).

[0239] As the first neural network model (620) and the second neural network model (640) have been described previously, a redundant description will be omitted.

[0240] The first neural network model (620) can obtain bio-motion information (630) using the first sensing data (611), the second sensing data (612), and the third sensing data (613) as input data. Here, since the first sensing data (611), the second sensing data (612), and the bio-motion information (630) have been described in detail previously, a redundant description will be omitted.

[0241] The third sensing data (613) may correspond to sensing data obtained by the display device (100) through a UWB sensor. Here, the UWB sensor may correspond to a sensor different from the sensor (e.g., motion sensor) used by the display device (100) to obtain the second sensing data (612).

[0242] The UWB sensor can sense the distance (and changes in distance) between the display device (100) and an external device. For example, the third sensing data (613) may include information about the distance between the display device (100) and an external device. Other specific details regarding the UWB sensor have been described in detail previously, so a redundant description will be omitted.

[0243] Meanwhile, the first neural network model (620) can also be learned based on third prior sensing data obtained by the display device (100) sensing the prior motion of an external device. That is, the first neural network model (620) can be learned using not only the first prior sensing data and second prior sensing data obtained through a UWB sensor, etc., but also the third prior sensing data as learning data.

[0244] In this case, the first neural network model (620) can be trained to output bio-motion information corresponding to a specific distance or a specific distance change between the display device (100) and the external device.

[0245] The first neural network model (620) can be trained to output bio-motion information by using other data as training data in addition to the first sensing data (611) to the third sensing data (613) described above.

[0246] For example, the first neural network model (620) can also be trained based on image data obtained through a camera.

[0247] Specifically, the display device (100) can capture a specific movement of a finger to acquire a plurality of images and estimate the direction of movement of the finger from the plurality of images. The display device (100) can acquire new training data by labeling at least one of the first sensing data (611) to the third sensing data (613) according to the estimated direction of movement of the finger. The display device (100) can train the first neural network model (620) using the training data acquired by labeling.

[0248] Meanwhile, the second neural network model (640) can output cursor movement information (650) using bio-motion information (630) as input data. Here, the second neural network model (640) can be trained to output cursor movement information (650) corresponding to bio-motion based on the first sensing data and the second sensing data of bio-motion.

[0249] And the display device (100) can execute a function corresponding to the UI by using the acquired cursor movement information to move the cursor according to the cursor movement information, and then swiping or tapping at the position where the user moved (660).

[0250] For example, the display device (100) can move and display the cursor using cursor movement information. Here, the display device (100) can display a screen in which the cursor has been moved from a first position to a second position.

[0251] And when the cursor at the second position of the display device (100) is positioned on a specific UI, the user can perform actions such as swiping or tapping. Here, actions such as swiping or tapping may refer to gestures that allow the user to select a specific UI and perform a specific action. Here, since swiping or tapping has been explained in detail above, a redundant explanation will be omitted.

[0252] The display device (100) can receive sensing data generated by a user's swipe or tapping from an external device. Alternatively, the display device (100) can acquire sensing data by sensing a user's swipe or tapping.

[0253] The display device (100) can recognize that there is a swipe or tap by the user and execute a function corresponding to the UI. For example, if the display device (100) displays an icon of a specific application at a second location, the display device (100) can execute the application. However, it is not limited thereto.

[0254] In this way, the display device (100) can move the cursor or execute a function corresponding to the UI at the position where the cursor has moved by using both the first neural network model (620) and the second neural network model (640).

[0255] Below, the operation of the display device (100) training the neural network model (620) and the second neural network model (640) will be described.

[0256] FIG. 7 is a drawing for illustrating aspects of a first neural network model according to one or more embodiments of the present disclosure.

[0257] According to FIG. 7, the display device (100) can capture the initial position of the fingertip (711) and the fingertip after movement (721) as the user of the display device (100) moves the fingertip. Here, the fingertip may correspond to the fingertip of a finger wearing an external device (10).

[0258] The display device (100) can capture the movement of the fingertip to obtain image data (740). Here, the image data may include multiple images.

[0259] Here, multiple images may correspond to multiple images acquired by capturing at specific intervals according to finger movement. The multiple images acquired here are continuous still images (frames) and may correspond to an image data set that reflects changes over time.

[0260] For example, when the display device (100) recognizes the fingertip (711) at the initial position through the camera, it may display a first UI marked with '1' on the fingertip at the initial position. When the first UI is displayed, the display device (100) may display a second UI marked with '2' to the left of the first UI at a certain interval from the first UI, along with a left arrow. Along with this, the display device (100) may display a third UI marked with 'LEFT'. The third UI may correspond to a UI that instructs the user to move the fingertip to the left. The second UI may indicate the position to which the fingertip moves.

[0261] The user can move the fingertip to the left as the first UI to the third UI is displayed, and the display device (100) can capture the fingertip at a set time interval to obtain multiple images. Here, the multiple images can be implemented as multiple frames included in a video.

[0262] According to FIG. 7, the case where the user's fingertip moves to the left has been described as an example, but it is not necessarily limited to this, and the display device (100) can repeat the same process even when the fingertip moves in multiple other pre-set directions (e.g., up, down, right, etc.).

[0263] The display device (100) can obtain the direction of movement of the fingertip by analyzing a plurality of images (frames) included in the image data. Specifically, the display device (100) can calculate the change in coordinates of the fingertip through feature point tracking between consecutive images (frames) (e.g., optical flow, CNN-based keypoint detection), estimate the direction of movement, and output it in the form of a vector.

[0264] For example, the display device (100) can recognize that the direction of movement of the fingertip is to the left from the image data obtained as the fingertip moves to the left. The display device (100) can repeat the same process even when the fingertip moves in other preset directions (e.g., up, down, right, etc.).

[0265] The display device (100) can acquire first prior sensing data (731), second prior sensing data (732), and third prior sensing data generated by the motion of the external device (10).

[0266] Here, the first prior sensing data (731) and the second prior sensing data (732) have been specifically described in FIG. 2, so a redundant description will be omitted.

[0267] The third prior sensing data may correspond to the sensing data previously acquired by the display device (100) through the UWB sensor.

[0268] For example, when the external device (10) moves according to the motion of the user's fingertip, the display device (100) can acquire third prior sensing data (733) through the UWB sensor.

[0269] The display device (100) may include a plurality of distances obtained at predetermined intervals according to the movement of the external device (10) based on the third prior sensing data (733). That is, the third prior sensing data (733) may include information on how far the external device (10) has moved from the display device (100).

[0270] Meanwhile, the first neural network model (750) can be trained to acquire bio-motion information based on the first previous sensing data (731), the second previous sensing data (732), the third previous sensing data (733), and image data.

[0271] In this case, the display device (100) can label at least one of the first prior sensing data (731), the second prior sensing data (732), and the third prior sensing data (733) using a specific direction identified from the image data. Since the labeling has been described in detail above, a redundant description will be omitted.

[0272] FIG. 8 is a flowchart for illustrating aspects of a first neural network model according to one or more embodiments of the present disclosure.

[0273] The display device (100) can capture a fingertip (S810). For example, the display device (100) can capture a fingertip located at an initial position.

[0274] Next, the display device (100) can indicate that the user should move the fingertip in a circular and up, down, left, and right direction through the UI (S820). For example, the display device (100) can display a UI (e.g., a first UI to a third UI) that instructs the user to move the fingertip in a specific direction, and when the user moves in a specific direction, it can display a UI corresponding to a different direction to repeat the same process.

[0275] The display device (100) can acquire image data by capturing the moving fingertip at predetermined time intervals when the user moves the fingertip according to the UI (e.g., first UI to third UI) displayed on the display device (100).

[0276] Next, the display device (100) acquires the first prior sensing data, the second prior sensing data, and the third prior sensing data, and can label based on the image data (S830). For example, the display device (100) can label the first prior sensing data, the second prior sensing data, and the third prior sensing data with a specific direction (up, down, left, right, etc.) acquired based on the image data.

[0277] Next, the display device (100) can train a first neural network model to output fingertip movement using the first previous sensing data, the second previous sensing data, and the third previous sensing data as input data (S840). Here, fingertip movement may refer to the aforementioned bio-motion information.

[0278] Next, the display device (100) can identify whether the fingertip movement predicted through the first neural network model matches the image data (S850).

[0279] Here, the fingertip movement predicted through the first neural network model may correspond to the result output by the first neural network model when current sensing data (e.g., first sensing data, second sensing data, and third sensing data, etc.) is input into the first neural network model trained as test input data.

[0280] Here, the fact that the predicted fingertip movement matches the image data may mean that the resulting fingertip movement (direction) matches the labeled direction. In other words, the fact that the predicted fingertip movement matches the image data may mean that the first neural network model has been trained to accurately predict the direction of the fingertip.

[0281] If the predicted fingertip movement matches the image data, the display device (100) can complete the process of training the first neural network model.

[0282] On the other hand, if the predicted fingertip movement does not match the image data, the first previous sensing data, the second previous sensing data, and the third previous sensing data can be acquired and labeled based on the image data (S830), and the subsequent processes described above can be repeated.

[0283] That is, if the predicted fingertip movement does not match the image data, the display device (100) can additionally acquire the first previous sensing data, the second previous sensing data, and the third previous sensing data, and train the first neural network model using the increased training data.

[0284] FIG. 9 is a drawing for illustrating aspects of a second neural network model according to one or more embodiments of the present disclosure.

[0285] According to FIG. 9, a process is illustrated in which a display device (100) trains a second neural network model based on the movement of a test cursor at a first cursor position (912). Here, the first cursor position (912) may correspond to the position where the test cursor is activated and first displayed.

[0286] In the initial state, the display device (100) can display a sensitivity slider (911) and a test cursor at a first cursor position (912).

[0287] At this time, the display device (100) may be held by a finger wearing an external device. Here, the external device may correspond to a device worn at a first device position (914). At this time, the fingertip of the finger wearing the external device may be located at a first biological position (913).

[0288] Meanwhile, the display device (100) can display a screen in which the sensitivity slider (911) is set to level 2. Here, the sensitivity slider may correspond to a UI that allows the user to adjust the sensitivity. Here, sensitivity may refer to the ratio of the degree of movement of the fingertip to the degree of movement of the cursor. Here, the ratio may be expressed as (distance of movement of the cursor) / (distance of movement of the fingertip), etc.

[0289] Meanwhile, the display device (100) can display a screen in which the cursor has moved from the first cursor position (912) to the second cursor position (912'). Here, the screen may correspond to a screen that instructs the user to move the fingertip by looking at the cursor moved to the second cursor position (912').

[0290] The user can move the fingertip from the first bio-position (913) to the second bio-position (913') by looking at the screen above. Accordingly, the external device can move from the first device position (914) to the second device position (914').

[0291] At this time, the user can observe the degree to which the cursor has moved from the first cursor position (912) to the second cursor position (912') and move the fingertip as much as is appropriate for the degree of movement. For example, the user can move the fingertip so that the cursor moves according to the fingertip located on the back of the display device (100).

[0292] The display device (100) can acquire previous sensing data (e.g., first previous sensing data to third sensing data) as the external device moves from the first device location (914) to the second device location (914').

[0293] The display device (100) can train a second neural network model based on previously acquired sensing data. That is, the display device (100) can train the second neural network model to output cursor movement information.

[0294] FIG. 10 is a drawing for illustrating aspects of a second neural network model according to one or more embodiments of the present disclosure.

[0295] According to FIG. 10, a first prior sensing data (1011), a second prior sensing data (1012), a third prior sensing data (1013), sensitivity (1020), cursor movement information (1030), and a second neural network model (1040) are shown.

[0296] Here, the display device (100) can train a second neural network model (1040) using the first previous sensing data (1011), the second previous sensing data (1012), the third previous sensing data (1013), sensitivity (1020), and cursor movement information (1030) as training data.

[0297] Here, the first prior sensing data (1011), the second prior sensing data (1012), and the third prior sensing data (1013) may correspond to sensing data acquired according to the movement of the external device when the external device moves as the fingertip moves. Specific details regarding each of the other first prior sensing data (1011), second prior sensing data (1012), and third prior sensing data (1013) have been described previously, so a redundant description will be omitted.

[0298] Sensitivity (1020) may refer to the sensitivity set by the user while the test cursor is displayed. According to the example in FIG. 9, the sensitivity may be set to two levels.

[0299] For example, if a user moves a fingertip in accordance with the movement of a test cursor, the sensitivity can be obtained as a result of dividing the movement distance of the test cursor by the movement distance of the fingertip. Here, the movement distance of the fingertip can be obtained by the first prior sensing data.

[0300] The cursor movement information (1030) may include the direction and distance the test cursor has moved. According to the example of FIG. 9, the cursor movement information (1030) may include information that the cursor has moved a specific distance in the upper left direction.

[0301] The second neural network model (1040) can be trained to output cursor movement information corresponding to at least one of the first previous sensing data (1011), the second previous sensing data (1012), the third previous sensing data (1013), and the sensitivity (1020).

[0302] For example, if the second neural network model (1040) is trained to output cursor movement information corresponding to a sensitivity set to level 2, the second neural network model (1040) can obtain cursor movement information suitable for a sensitivity of level 2 while the current sensitivity is set to level 2.

[0303] However, it is not limited to this, and even if the current sensitivity is set to other stages (stages 1, 3, 4, etc.), the second neural network model (1040) can obtain cursor movement information suitable for each of the other stages' sensitivities.

[0304] For example, as the sensitivity level decreases, the cursor movement distance per unit movement of the fingertip may appear shorter. On the other hand, as the sensitivity level increases, the cursor movement distance per unit movement of the fingertip may appear longer.

[0305] In this way, the second neural network model (1040) can be trained using various training data, and specific operations related thereto will be described in the following section.

[0306] FIG. 11 is a flowchart illustrating aspects of a second neural network model according to one or more embodiments of the present disclosure.

[0307] The display device (100) can display a cursor in the center of the screen and instruct the user to tap the center of the back of the display device (100) (S1110). The display device (100) can display a UI on the screen that instructs the user to tap with a finger wearing an external device. For example, the display device (100) can display text on the screen such as 'Tap the back with a finger wearing a smart ring.'

[0308] Next, the display device (100) can obtain the initial position of the external device when it recognizes a tap gesture (S1120). For example, the display device (100) can display the initial position of the external device when it recognizes that the cursor is active. Here, the meaning of the cursor being active has been explained in detail above, so a redundant explanation will be omitted.

[0309] Here, the initial position of the external device can be obtained by a sensor (e.g., UWB sensor, motion sensor and image sensor (camera) etc.) equipped in the display device (100).

[0310] Next, the display device (100) can display a new direction of the cursor for each direction (up, down, left, right, etc.) (S1130). Here, each direction may correspond to a direction included in a plurality of pre-set directions (up, down, left, right, etc.). For example, the display device (100) can display the direction of the cursor through an arrow-shaped UI.

[0311] Subsequently, the fingertip moves in a new direction, and the display device (100) can detect a change in the position of the external device (S1140). For example, the user can move the fingertip by looking at the displayed new direction, and the display device (100) can detect a change in the position of the external device in accordance with the movement of the fingertip. For example, the display device (100) can acquire sensing data (first previous sensing data and second previous sensing data, etc.) regarding a change in the distance from the external device and a change in the acceleration of the external device in accordance with the movement of the fingertip.

[0312] Next, the display device (100) can recognize a tap gesture at a new location (S1150). Here, the gesture may correspond to an action for the user to complete the movement of the fingertip. For example, the user may move the fingertip by looking at the arrow and the cursor at the target location (second cursor position) displayed on the screen, and then tap the fingertip on the back of the display device (100) at the moved location.

[0313] Next, the display device (100) can identify whether the external device has moved in the proposed direction (S1160). Here, the proposed direction may refer to the direction indicated by an arrow on the screen.

[0314] Here, the display device (100) can identify whether an external device has moved in the proposed direction through a sensor provided in the display device (100).

[0315] For example, the display device (100) can identify the location of an external device at the location where a tap gesture is recognized from sensing data (distance data, image data, motion data (acceleration, speed, angular velocity, etc.) obtained through a sensor. The display device (100) can identify whether the location of the external device has changed and whether the external device has moved in the proposed direction.

[0316] If it is identified that the external device has not moved in the proposed direction, the fingertip moves again in a new direction, and the display device (100) can detect the change in the position of the external device (S1140). That is, if the external device does not move in the target direction, the user can move the fingertip from the initial position to the target position to move the cursor from the initial position to the target position. Accordingly, the display device (100) can recognize the tap gesture again at the new position and repeat the same process.

[0317] On the other hand, if it is identified that the external device has moved in the proposed direction, the display device (100) can acquire acceleration data of the external device and the display device (100) (S1170).

[0318] For example, if the display device (100) identifies that the external device has moved in the target direction, it can recognize that the fingertip has moved in the correct direction. In this case, the display device (100) can acquire acceleration data of the external device.

[0319] Specifically, the display device (100) can receive acceleration data acquired by an external device. Here, the acceleration data acquired by the external device may correspond to data acquired by the external device sensing the motion of the external device.

[0320] Next, the display device (100) can train a second neural network model using data on the position change of an external device (S1180).

[0321] Here, data regarding the change in position may include the first to third prior sensing data described above. For example, data regarding the change in position may include the distance from an external device acquired by the display device (100) through a sensor (e.g., a UWB sensor).

[0322] The display device (100) can train a second neural network model to output cursor movement information using such data. Here, the operation of training the second neural network model has been described in detail above, so a redundant description will be omitted.

[0323] Next, the display device (100) can identify whether learning has been completed for all directions (S1190). Here, all directions may refer to multiple pre-set directions (e.g., up, down, left, right, etc.).

[0324] The fact that training is complete for all directions does not necessarily mean that training data must be acquired once for each direction to train the second neural network model.

[0325] For example, the display device (100) can acquire training data multiple times for one direction and train a second neural network model for that direction. Here, acquiring training data may mean the process of the display device (100) sensing the movement of an external device to acquire data regarding the change in position of the external device.

[0326] If it is identified that learning has not yet been completed for all directions, the display device (100) can again display a new direction of the cursor for each direction (up, down, left, right, etc.) (S1130). Here, each direction may refer to the direction among up, down, left, and right for which learning has not been completed.

[0327] The fingertip can move in a new direction and the display device (100) can repeat the subsequent processes described above, such as detecting a change in the position of an external device.

[0328] If it is identified that learning is complete for all directions, the display device (100) can complete the process of training the second neural network model. Accordingly, the second neural network model can be trained to acquire cursor movement information corresponding to the movement of the fingertip.

[0329] FIG. 12 is a drawing for explaining the operation of activating a cursor according to one or more embodiments of the present disclosure.

[0330] According to FIG. 12, the process of activating the cursor (1231) of the display device (100) is illustrated.

[0331] The display device (100) may first perform the process of activating the cursor (1231) before acquiring sensing data corresponding to the user's bio-motion (e.g., motion of the fingertip). Here, the display device (100) may activate the cursor (1231) when at least one predetermined condition is satisfied. When the cursor (1231) is activated, the display device (100) may display the cursor (1231) on the screen.

[0332] Herein, at least one condition may include a first condition and a second condition. Examples of the first condition and the second condition will be described below.

[0333] The display device (100) can identify the gap (1211) between the display device (100) and the external device (10). For example, the display device (100) can acquire first initial sensing data (e.g., ToF) through a UWB sensor and identify the gap (1211) based on the acquired sensing data.

[0334] The display device (100) can identify that the first condition is satisfied if it identifies that the gap (1211) is less than a predetermined value. If the gap (1211) is less than a predetermined value, the display device (100) can recognize that the fingertip has come into contact with the display device (100) (or the rear surface of the display device (100)) or is in close proximity, such as when the gap is less than a preset range. The preset range may consider value-to-value analysis per unit of time, similarity of multiple values ​​over a certain period of time, or similar factors.

[0335] Meanwhile, the display device (100) can identify that the second condition is satisfied if the second initial sensing data (1221) and the third initial sensing data (1222) are identified as matching a predetermined gesture pattern.

[0336] For example, the display device (100) can analyze the acceleration included in each of the second initial sensing data (1221) and the third initial sensing data (1222) to identify whether each of the second initial sensing data (1221) and the third initial sensing data (1222) matches a predetermined gesture pattern. Here, acceleration may refer to a change in acceleration over time.

[0337] Specifically, the display device (100) can analyze the pattern of change in acceleration at a specific point in time (1223). That is, the display device (100) can compare the pattern of change in acceleration over time of each of the second initial sensing data (1221) and the third initial sensing data (1222) at a specific point in time (1223). Here, the specific point in time may correspond to the time when the user performs a gesture to activate the cursor.

[0338] Here, the second initial sensing data (1221) may correspond to data obtained by an external device (10) sensing a user's gesture. The third initial sensing data (1222) may correspond to data obtained by a display device (100) sensing the aforementioned user's gesture. Here, the user's gesture may correspond to a hand movement (e.g., tapping, double-tapping, etc.) taken by the user toward the display device (100) to activate the cursor.

[0339] Here, the process of the display device (100) comparing initial sensing data with a predetermined gesture pattern to identify whether it matches has been specifically explained in FIG. 3, so a redundant explanation will be omitted.

[0340] When the initial sensing data is identified as matching a gesture pattern, the display device (100) may be recognized as being in contact with the display device (100) (or the rear of the display device (100)) or as being in close proximity, such as when the range is less than a preset range. The preset range may consider value-to-value analysis per unit of time, similarity of multiple values ​​over a certain period of time, or similar factors.

[0341] When the display device (100) identifies that both the first condition and the second condition are satisfied, it can activate the cursor (1231) and display it on the screen. In this case, the display device (100) can display the activated cursor (1231) at an initial position. Here, the initial position may correspond to the aforementioned first position.

[0342] According to FIG. 12, the activated cursor (1231) is shown as being displayed in a first position located at the top left, but this is merely an example, and the activated cursor (1231) may be displayed at other positions on the screen of the display device (100). Here, the activated cursor (1231) may be displayed at a position that can be pre-set by the user.

[0343] Accordingly, the display device (100) can identify whether the state of the display device (100) satisfies a predetermined condition and perform a pre-activation process to move the cursor (1231). Through this, the display device (100) can prevent malfunction of the display device (100) caused by unintentional movements of the user when the external device (10) is not in a normal state, such as when it is too far from the display device (100).

[0344] FIG. 13 is a flowchart for explaining a control method of a display device according to one or more embodiments of the present disclosure.

[0345] The display device (100) can acquire bio-motion information based on the first sensing data (S1310).

[0346] According to one or more embodiments, the display device (100) may receive first sensing data generated from an external device. Here, the first sensing data may correspond to data generated by a second motion of the external device according to a first motion of the living organism.

[0347] And the display device (100) can obtain bio-motion information corresponding to the first motion based on the first sensing data.

[0348] Next, the display device (100) can acquire cursor movement information corresponding to the bio-motion information (S1320).

[0349] Next, the display device (100) can display a screen in which the cursor position has been moved from a first position to a second position based on cursor movement information.

[0350] According to one or more embodiments, the display device (100) may display a screen in which the position of the cursor is moved from a first position to a second position based on cursor movement information. Here, the first position may correspond to the position where the cursor is activated and first displayed.

[0351] Next, the display device (100) can display a screen in which the cursor has moved from a first position to a second position based on cursor movement information (S1330).

[0352] As described above, the display device (100) can change the position of the cursor based on the movement of the living body wearing the external device. Even if the external device is not worn directly on the part of the living body (e.g., fingertip) that the user actually wants to move, the display device (100) can predict the movement of the part of the living body from the movement of the external device.

[0353] Since the display device (100) can move a cursor to correspond to the movement of a predicted biological part, the user can be provided with a user experience of controlling the display device (100) through a virtual trackpad even if the display device (100) is not equipped with a configuration such as a trackpad (touchpad).

[0354] Additionally, there may be cases where the external device and the display device (100) are far apart, such as when a user wearing the external device is not holding the display device (100). In this case, the display device (100) may perform a process of activating the cursor in advance to prevent malfunction caused by the user's movement.

[0355] And, even after the cursor is activated, the display device (100) can be separated from the external device and the display device (100) by means of the user dropping the display device (100).

[0356] In this case as well, to prevent malfunction of the display device (100) caused by the user's movement, it is possible to determine whether the display device (100) and the external device are located close to each other even before the cursor moves, and to determine whether to proceed with subsequent operations.

[0357] Meanwhile, in FIG. 13, the order of all steps has been mapped for convenience of explanation, but it goes without saying that the order of steps that are not related to the order or can be performed in parallel is not necessarily limited to that order.

[0358] Meanwhile, methods according to at least some of the various embodiments of the present disclosure described above can be implemented in the form of an application that can be installed on an existing display device.

[0359] In addition, methods according to at least some of the various embodiments of the present disclosure described above can be implemented by software upgrades or hardware upgrades alone for existing display devices.

[0360] In addition, methods according to at least some of the various embodiments of the present disclosure described above may also be performed through an embedded server equipped in a display device, or through at least one external server among the display devices.

[0361] Meanwhile, according to one or more embodiments of the present disclosure, the various embodiments described above may be implemented as software containing instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include a display device (e.g., a display device) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions. When an instruction is executed by a processor, the processor may perform a function corresponding to the instruction directly or using other components under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory storage medium" simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium. For example, A 'non-transient storage medium' may include a buffer in which data is temporarily stored. According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones).In the case of online distribution, at least a portion of a computer program product (e.g., a downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0362] Various embodiments of the present disclosure may be implemented as software including instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include a display device (e.g., a display device (100)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions.

[0363] When the above-described instruction is executed by a processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter.

[0364] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In a display device, Memory for storing instructions; Communication circuit; A display for displaying a cursor; and at least one processor including processing circuitry; and When the above instructions are executed individually or collectively by the at least one processor, the display device, Receive first sensing data generated by a second motion of the external device corresponding to a first motion of the biological body from an external device through the communication circuit, and Based on the first sensing data above, bio-motion information corresponding to the first motion is obtained, and Obtain cursor movement information corresponding to the above bio-motion information, and A display device that controls the display to display a screen in which the cursor is moved from a first position to a second position based on the cursor movement information.

2. In Paragraph 1, It further includes a UWB sensor, When the above instructions are executed individually or collectively by the at least one processor, the display device, The distance between the external device and the display device is obtained from the above UWB sensor, and A display device that receives the first sensing data through the communication circuit when it is determined that the above distance is less than a predetermined distance.

3. In Paragraph 1, It further includes at least one motion sensor that senses a third motion of the display device; When the above instructions are executed individually or collectively by the at least one processor, the display device, A display device that acquires the bio-motion information when the second sensing data acquired through the at least one motion sensor is identified as matching the first sensing data.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the display device, A display device for obtaining biomotion information from a first neural network model trained to output biomotion information according to previous sensing data and a first previous motion of the bio.

5. In Paragraph 4, The above first neural network model is, Based on the above previous sensing data, it is trained to output bio-motion information according to the second previous motion and the above previous sensing data, and The above previous sensing data is, A display device comprising first prior sensing data obtained by sensing the second prior motion of the external device and second prior sensing data obtained by sensing the first prior motion.

6. In Paragraph 5, The above second prior sensing data includes image data obtained by capturing the above first prior motion multiple times in multiple directions, and A display device wherein the first neural network model is trained to output bio-motion information including a direction corresponding to the second previous motion among the plurality of directions.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the display device, A display device that obtains cursor movement information from a second neural network model trained to output cursor movement information according to previous sensing data and a first previous motion of the biological body.

8. In Paragraph 7, The above second neural network model is, A display device trained to output cursor movement information according to the first previous motion of the test cursor and the biological body, based on previous sensing data obtained by sensing a second previous motion of the external device corresponding to the movement of the test cursor displayed on the display.

9. In Paragraph 2, It further includes a UWB sensor, When the above instructions are executed individually or collectively by the at least one processor, the display device, Based on the first initial sensing data obtained through the above UWB sensor, the distance between the external device and the display device is obtained, and Identify whether the cursor is activated based on whether the above distance is less than a predetermined distance, and A display device that receives the first sensing data when the cursor is identified as active.

10. In Paragraph 9, It further includes at least one motion sensor that senses a third motion of the display device; When the above instructions are executed individually or collectively by the at least one processor, the display device, If it is determined that the above distance is less than a predetermined distance, second initial sensing data generated by a gesture for activating the cursor from the external device is received through the communication circuit, and If the third initial sensing data and the second initial sensing data obtained by sensing a motion corresponding to the gesture through the at least one motion sensor are identified as matching a predetermined gesture pattern, the cursor is identified as being activated. A display device that controls the display to display the cursor.

11. In a method for controlling a display device, A step of receiving first sensing data generated by a second motion of the external device corresponding to a first motion of the biological body from an external device; A step of obtaining bio-motion information corresponding to the first motion based on the first sensing data; A step of obtaining cursor movement information corresponding to the above bio-motion information; and A control method comprising the step of displaying a screen in which the cursor is moved from a first position to a second position based on the cursor movement information.

12. In Paragraph 11, The step of receiving the first sensing data is The distance between the external device and the display device is obtained from the UWB sensor of the display device, and A control method for receiving the first sensing data when it is determined that the above distance is less than a predetermined distance.

13. In Paragraph 11, The step of acquiring the above bio-motion information is, A control method for acquiring bio-motion information when second sensing data acquired through at least one motion sensor sensing a third motion of the display device is identified as matching the first sensing data.

14. In Paragraph 11, The step of acquiring the above bio-motion information is, A control method for obtaining biomotion information from a first neural network model trained to output biomotion information according to previous sensing data and a first previous motion of the bio.

15. A non-transient computer-readable recording medium storing computer instructions that cause the display device to perform an operation when executed by a processor of the display device, wherein the operation is, A step of receiving first sensing data generated by a second motion of the external device corresponding to a first motion of the biological body from an external device; A step of obtaining bio-motion information corresponding to the first motion based on the first sensing data; A step of obtaining cursor movement information corresponding to the above bio-motion information; and A non-transient computer-readable recording medium comprising: a step of displaying a screen in which the cursor is moved from a first position to a second position based on the cursor movement information.

Images