Wearable device providing user assistance function and operation method thereof

Abstract

A wearable device, according to various embodiments, comprises: a pair of earcups configured to output sound; a rotating frame coupled to the earcups so as to be changeable in shape into a first state or a second state; at least one sensor acquiring sensing data including at least one of state data of the rotating frame, image data, or audio data; a display coupled to the rotating frame; at least one processor; and a memory storing instructions, wherein the instructions are individually or collectively executed by the at least one processor such that the wearable device may: acquire language service information including language conversion information for sensing data on the basis of processing the sensing data acquired via the at least one sensor; and determine at least one of the earcups and the display on the basis of a state of the rotating frame and output the language service information.

Description

Wearable device providing user assistance functions and method of operation thereof

[0001] The various embodiments disclosed in this document relate to a wearable device that provides user assistance functions, a method of operating the same, and a recording medium for performing the method. More specifically, it relates to a wearable device that outputs language service information, a method of operating the same, and a recording medium for performing the method.

[0002] Recently, various studies on the form and structure of wearable devices to improve user convenience and accessibility have been actively underway, and related technologies are also rapidly advancing accordingly. In particular, there are ongoing attempts to provide new forms of user experiences by enabling real-time information provision, user assistance functions, and enhanced interaction through wearable devices during the user's daily activities.

[0003] These wearable devices can be implemented through various hardware platforms, such as head-mounted devices worn on the head, glasses-type devices, and hearing aids in the form of earphones or headsets. In particular, with the recent advancement of eXtended Reality (XR) technology, devices utilizing Augmented Reality (AR) and Virtual Reality (VR) are becoming increasingly commercialized, and these technologies are combined with wearable platforms to enhance user immersion and interactive capabilities.

[0004] In addition, wearable devices are equipped with one or more speakers, enabling them to provide users with stereo sound or spatial sound. Spatial sound technology is designed to go beyond simple left-right sound separation and allow users to perceive the location, direction, and depth of sound sources, thereby providing an immersive auditory experience. While this technology has traditionally been implemented in fixed environments, research is recently expanding toward enabling it to be provided in dynamic and fluid environments through wearable devices.

[0005] However, existing wearable devices have limitations in responding appropriately to user needs in actual usage situations due to a lack of adaptive control functions based on the user's state or the device's physical posture. In particular, there have been many restrictions on recognizing non-verbal expressions such as sign language or providing real-time interpretation or translation services in multilingual environments. Context-aware operations, such as automatically determining the means of information output based on the user's current field of vision or wearing style, or distinguishing between one's own and others' gestures or voice signals to set different output devices, have not been sufficiently implemented in existing technologies.

[0006] Furthermore, from the perspective of assistive technology, there is an increasing demand for sign language interpretation services and voice recognition-based translation services for users with hearing or language limitations, but the technology to realize these on wearable devices is still in its early stages. In particular, advanced interface control functions are required, such as automatically switching the display direction according to the wearer's field of vision or distinguishing between the received and transmitted languages ​​through speaker identification.

[0007] Accordingly, there is a demand for technology that goes beyond simple sound output for wearable devices to dynamically adjust the method of information representation based on user state and input signals, and to process sign language or language information in real time to output it in auditory and visual ways. Such technology can be particularly useful for sign language users, users in multicultural environments, or environments requiring real-time interpretation.

[0008] The foregoing description is provided as related technology to aid in understanding the present invention and does not contain any claim or suggestion that it should be applied as prior art to deny the novelty or inventive step of the present invention.

[0009] The problem to be solved in the present disclosure may be to provide a wearable device capable of interpreting information obtained from a user or another person's gestures or voice in real time and appropriately outputting corresponding sign language information or language service information to the user.

[0010] The problem to be solved in the present disclosure may be to provide a wearable device that improves user convenience and transmission accuracy by automatically selecting an output means (display, ear cup, external speaker, etc.) according to the state (shape) of the rotating frame and providing sign language information or language conversion information through said means.

[0011] The problem to be solved in the present disclosure may be to provide a wearable device that automatically recognizes sign language information or language conversion requests made by a user through gestures or voice, distinguishes between the user and an external speaker, and determines an output path suitable for the situation.

[0012] The problem to be solved in the present disclosure may be to implement a wearable device that selectively provides sign language or interpretation information according to the user's visual or auditory conditions through a display or sound output device of the wearable device, and provides an assistive system that can adapt to various usage environments.

[0013] The problem to be solved in the present disclosure may be to provide a wearable device that acquires and outputs language conversion information in real time based on pre-set language information or user input, thereby supporting smooth communication between users in a multicultural and multilingual environment.

[0014] According to various embodiments, a wearable device comprises a pair of earcups configured to output sound, a rotating frame coupled to the earcups to enable shape change to a first state or a second state, at least one sensor for acquiring sensing data including at least one of state data of the rotating frame, image data or acoustic data, a display coupled to the rotating frame, at least one processor, and a memory for storing instructions, wherein the instructions are executed individually or collectively by the at least one processor so that the wearable device: acquires language service information including language conversion information for the sensing data based on processing the sensing data acquired through the at least one sensor, and determines at least one of the earcups or the display based on the state of the rotating frame to output the language service information.

[0015] According to various embodiments, a method for outputting language service information may include the step of acquiring sensing data including at least one of state data of the rotating frame, image data, or sound data, performed by a wearable device comprising: a pair of earcups configured to output sound; a rotating frame coupled to the earcups so as to be changeable to a first state or a second state; at least one sensor acquiring sensing data including at least one of state data of the rotating frame, image data, or sound data; a display coupled to the rotating frame; at least one processor; and a memory storing instructions.

[0016] A wearable device according to various embodiments disclosed in this document can interpret a user's gesture or voice information in real time and provide sign language information or language service information in an appropriate form.

[0017] A wearable device according to various embodiments disclosed in this document can efficiently deliver sign language information or language service information to a user by selectively utilizing various output means, such as a display, ear cup, or external speaker, depending on the rotational frame state of the wearable device.

[0018] A wearable device according to various embodiments disclosed in this document can provide a natural and immersive user assistance experience even in a multilingual environment by recognizing the identity of the speaker and distinguishing whether the user is the speaker or an outsider, and controlling the output means of language conversion information and the output language differently accordingly.

[0019] A wearable device according to various embodiments disclosed in this document can automatically execute language service functions and adjust output targets based on various sensing data, such as gesture recognition, voice recognition, and user input triggers, thereby providing easy operability and high practicality.

[0020] A wearable device according to various embodiments disclosed in this document can adjust the language of output sign language information or language conversion information to suit the user's environment by utilizing preset language information or user-defined settings, and thus has universality that can be used in various countries and cultural regions.

[0021] In addition to this, various effects that can be identified directly or indirectly through this document may be provided.

[0022] FIG. 1 is a diagram illustrating the usage environment of a wearable device according to various embodiments.

[0023] FIG. 2 illustrates a wearable device according to various embodiments worn on at least a part of a user's body.

[0024] FIG. 3 illustrates a wearable device according to various embodiments worn on at least a part of the body.

[0025] FIG. 4 is a block diagram of a wearable device according to various embodiments.

[0026] FIG. 5 is a diagram illustrating the usage environment of a wearable device according to various embodiments.

[0027] FIG. 6 is a flowchart illustrating the operation of a wearable device outputting literacy information according to various embodiments.

[0028] FIG. 7 is a diagram illustrating a usage environment in a first state of a wearable device according to various embodiments.

[0029] FIG. 8 is a diagram illustrating a second state usage environment of a wearable device according to various embodiments.

[0030] FIG. 9 is a flowchart illustrating the operation of a wearable device outputting cipher information according to various embodiments.

[0031] FIG. 10 is a drawing for illustrating an embodiment of a second state of a wearable device according to various embodiments.

[0032] FIG. 11 is a drawing for illustrating an embodiment of a wearable device changing state from a first state to a second state according to various embodiments.

[0033] FIG. 12 is a diagram illustrating the usage environment of a wearable device according to various embodiments.

[0034] FIG. 13 is a drawing for explaining an embodiment of a second state of a wearable device according to various embodiments in the usage environment of FIG. 12.

[0035] FIG. 14 is a diagram illustrating the usage environment of a wearable device according to various embodiments.

[0036] FIG. 15 is a flowchart illustrating the operation of a wearable device outputting language service information according to various embodiments.

[0037] FIG. 16 is a drawing for illustrating an embodiment of a second state of a wearable device according to various embodiments.

[0038] FIG. 17 is a diagram illustrating a multiple speaker usage environment of a wearable device according to various embodiments.

[0039] FIG. 18 is a diagram illustrating a multiple speaker usage environment of a wearable device according to various embodiments.

[0040] FIG. 19 is a flowchart illustrating the operation of a wearable device according to various embodiments outputting language service information when the speaker is the user.

[0041] FIG. 20 is a flowchart illustrating the operation of a wearable device according to various embodiments outputting language service information when the speaker is not the person themselves.

[0042] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0043] Specific structural or functional descriptions regarding various embodiments are illustrative for the purpose of explaining various embodiments, and various embodiments may be implemented in various forms and should not be interpreted as being limited to the embodiments described in this specification or application.

[0044] Since various embodiments may be subject to various modifications and may take various forms, various embodiments are illustrated in the drawings and described in detail in this specification or application. However, the details disclosed in the drawings are not intended to specify or limit the various embodiments, and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the various embodiments.

[0045] Terms such as "first" and / or "second" may be used to describe various components, but said components shall not be limited by said terms. For the sole purpose of distinguishing one component from another, for example, without departing from the scope of rights according to the concept of the present disclosure, the first component may be named the second component, and similarly, the second component may be named the first component.

[0046] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. Conversely, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions describing the relationship between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0047] The terms used herein are used merely to describe specific embodiments and are not intended to limit various embodiments. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0048] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this disclosure pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0049] The present disclosure will be described in detail below by explaining preferred embodiments of the present disclosure with reference to the attached drawings. Identical reference numerals in each drawing indicate identical components.

[0050]

[0051] FIG. 1 is a diagram illustrating the usage environment of a wearable device according to various embodiments.

[0052] Referring to FIG. 1, a user environment (e.g., system) of a wearable device according to one embodiment of the present invention may include a wearable device (1), a network (2), and an external device (3). According to one embodiment, the external device (3) may be integrated into the wearable device (1) or interconnected via the network (2) as an independent device. In this case, the wearable device (1) and the external device (3) may be connected via any network (2). According to one embodiment, the wearable device (1) may be connected to another wearable device (1') via the network (2). However, the user environment may be implemented to include more configurations or / or fewer configurations, not limited to the illustrated and / or described examples.

[0053] Additionally, the wearable device (1) can be configured to be connected to another wearable device (1') to exchange gesture information, language service information, or status data between users in real time, thereby supporting two-way communication or performing cooperative tasks.

[0054] According to various embodiments, the wearable device (1) and / or external device (3) may be various types of devices. For example, it may include a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, a headset, headphones, AR glasses, an HMD device, or a consumer electronics device.

[0055] At this time, the external device (3) can be embodied as a display device, voice output device, control input device, etc., depending on the function, and each device can perform a role of assisting or expanding the function of the wearable device (1).

[0056] According to various embodiments, the wearable device (1) may be implemented to provide various content to the user. For example, the wearable device (1) may be implemented to establish a connection with an external device (3) through a network (2) so that content available through the external device (3) is also provided through the wearable device (1). At this time, the network according to one embodiment of the present invention may be configured regardless of the communication type, such as wired and wireless, and may be configured as various networks such as a Personal Area Network (PAN), a Local Area Network (LAN), and a Wide Area Network (WAN). In addition, the network may be the known World Wide Web (WWW), and may utilize wireless transmission technology used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. At this time, the type of network is not limited to the above-described embodiment, and various communication systems may be included in the network.

[0057] In addition, the wearable device (1) may internally operate by electrically connecting a processor, display, memory, communication circuit, sensor, speaker, etc. through a system bus or an integrated circuit board.

[0058] According to various embodiments, the wearable device (1) may be implemented to provide services to the user in various ways using an external device (3) (e.g., a speaker (not shown), a display (not shown), etc.).

[0059] In one embodiment, the wearable device (1) may be implemented as a system type. The system type may be defined as a type that provides services in cooperation with other external devices. For example, the wearable device (1) may be implemented to provide services based on exchanging data (or information) with a server, etc., via an external device (3) (e.g., transmission and / or reception). The wearable device (1) may establish a communication connection with the external device (3), and the external device (3) may establish a communication connection with a server, etc. Accordingly, the wearable device (1) and the server may transmit or / or receive data (or information) through the external device (3). For another example, the wearable device (1) may be implemented to provide services based on exchanging data (or information) with a server (e.g., transmission and / or reception). The wearable device (1) can establish a direct communication connection with the server and transmit data (or information) to the server or receive data (or information) from the server (not shown).

[0060] In another embodiment, the wearable device (1) may be implemented as an on-device type. For example, the wearable device (1) may be implemented to provide services independently without exchanging data (or information) with a server or an external device (3).

[0061] According to various embodiments, the wearable device (1) may acquire media data from an external device (3) through the network and provide it to the user through a component of the wearable device (1) (e.g., a display, a speaker, etc.). In another embodiment, the wearable device (1) may provide media data to the user through a component of the wearable device (1) itself. In another embodiment, the wearable device (1) may determine the output mode of the media data and control the speaker to output the media data according to the output mode.

[0062] In addition, according to various embodiments, the wearable device (1) can enhance personal information protection and communication security by performing a user authentication procedure or encrypting transmitted data when transmitting and receiving data through a network.

[0063] In the following, a method is described in which a wearable device (1) operates in conjunction with an external device (3) or independently to provide various output information, including sign language information or language service information, to a user. At this time, the output information may be provided through a speaker, a display, or an output means included in the external device (3) that is included in the wearable device (1).

[0064] FIG. 2 illustrates a wearable device according to various embodiments worn on at least a part of a user's body.

[0065] FIG. 3 illustrates a wearable device according to various embodiments worn on at least a part of the body.

[0066] Hereinafter, the coordinate axes illustrated in FIGS. 2 and FIGS. 3 illustrate the left (L), right (R), upper (U), lower (D), front (F), and rear (B) sides defined relative to the user. The coordinate axes may be understood as exemplary coordinate axes for describing a wearable device (1) according to one embodiment of the present disclosure.

[0067] The configurations of FIGS. 2 and FIGS. 3 may be referenced by configurations of other drawings to the extent that they are not disposed of from each other. The same terms and / or the same reference numerals have been used for configurations that are identical or substantially identical to configurations of other drawings.

[0068] Referring to FIGS. 2 and FIGS. 3, a wearable device (1) according to one embodiment of the present disclosure may include a display device (DR, DL). The display device (DR, DL) may be configured to provide visual information (e.g., an image or video) to a user. As an example, the display device (DR, DL) may be connected to the outside through separate wiring (not shown).

[0069] According to one embodiment of the present disclosure, a display device (DR, DL) may include a right display (DR) and a left display (DL). The right display (DR) may correspond to the user's right eye, and the left display (DL) may correspond to the user's left eye. The right display (DR) may provide visual information to the user's right eye, and the left display (DL) may provide visual information to the user's left eye. The display device (DR, DL) may be configured to provide visual information associated with sound output from a pair of earcups (101, 102) to the user.

[0070] According to one embodiment, the display device (DR, DL) may include at least one lens portion and a window. The at least one lens portion may include a first lens portion disposed on the right display (DR) and a second lens portion disposed on the left display (DL). According to one embodiment, the lens portion may be implemented to receive image light output from the display (DR, DL) and provide it to the user's pupil. Additionally, the lens portion may be implemented to provide light provided from the outside to the user's pupil of the wearable device (1), while receiving image light output from the display (DR, DL) and providing it to the user's pupil.

[0071] According to one embodiment, a wearable device (1) may include a pair of earcups (101, 102) configured to output sound. At least one of the pair of earcups (101, 102) may have a speaker (not shown) built in it configured to generate sound. Auditory information may be provided to a user through the pair of earcups (101, 102).

[0072] For convenience of explanation, a pair of ear cups (101, 102) may be described as being divided into a right ear cup (101) and a left ear cup (102) based on the user. As an example, the right ear cup (101) and the left ear cup (102) may have a structure that is symmetrical to the left and right based on the user, and unless specifically mentioned otherwise, the description of the right ear cup (101) described below may be applied substantially the same to the left ear cup (102) to the extent that they are not positioned relative to each other.

[0073] According to one embodiment of the present disclosure, a wearable device (1) may include a right pivot member (201) and a left pivot member (202) coupled to a pair of ear cups (101, 102) such that at least a portion thereof may be rotatable with respect to a pair of ear cups. The right pivot member (201) and the left pivot member (202) may be configured to rotate with respect to the ear cups (101, 102) around a left-right axis. The right pivot member (201) may be coupled to the right ear cup (101), and the left pivot member (202) may be coupled to the left ear cup (102). Here, it can be understood that the rotating part (201, 202) and the ear cup (101, 102) are coupled, for example, by extending from the outside of the housing of the ear cup (101, 102) and being rotatably coupled, and that one side of the ear cup (101, 102) is open so that the rotating part (201, 202) is provided inside the housing of the ear cup (101, 102), and that the position of the connecting part (301, 302) to which the rotating part (201, 202) is coupled can be changed by the rotating part (201, 202) rotating through the open side of the housing.

[0074] For example, the right ear cup (101) and the left ear cup (102) may have a structure that is symmetrical to the left and right relative to the user, and unless specifically mentioned otherwise, the description of the right pivot part (201) described below may be applied substantially the same to the left pivot part (202) to the extent that they are not arranged relative to each other.

[0075] According to one embodiment of the present disclosure, a wearable device (1) may include a pivot frame (M) configured to rotate (see FIG. 3) and / or move with respect to a pair of ear cups (101, 102). The pivot frame (M) may be connected to a pair of ear cups (101, 102) through pivot portions (e.g., a right pivot portion (201) and a left pivot portion (202)). The pivot frame (M) may include a display device (DR, DL) and a connection portion (301). The display device (D) may be rotated and / or moved with respect to a pair of ear cups (101, 102) to facilitate providing visual information to a user. The pivot frame (M) may include a right connection portion and a left connection portion as described below, provided, but not limited thereto.

[0076] According to one embodiment of the present disclosure, a wearable device (1) may include a right connecting part (301) connecting the display device (DR, DL) and the right rotating part (201). The wearable device (1) may include a left connecting part (not shown) connecting the display device (DR, DL) and the left rotating part (202). As an example, the right connecting part (301) and the left connecting part (not shown) may have a structure that is symmetrical to the left and right with respect to the user, and unless specifically mentioned otherwise, the description of the right connecting part (301) described below may be applied substantially the same to the left connecting part (not shown) to the extent that they are not arranged relative to each other.

[0077] Referring to FIGS. 2 and 3, according to one embodiment of the present disclosure, a rotating frame (M) of a wearable device (1) can be rotated relative to a pair of earcups (101, 102) so that a display device (DR, DL) is positioned in front of a user. FIG. 3 illustrates a second state in which the display device (DR, DL) is positioned in front of (or within the user's field of vision). FIG. 2 illustrates a first state in which the display device (DR, DL) is not positioned in front of the user (or is positioned outside the user's field of vision), for example, in which the display device (DR, DL) is positioned above the user's head. In another description, FIG. 2 illustrates a first state in which the display device (DR, DL) is rotated from the second state so that it is positioned above the user's head. FIG. 3 illustrates the second state in which the display device (DR, DL) is rotated from the first state so that it is positioned in front of the user.

[0078] According to various embodiments, the pivot frame (M) may be located in various positions, not limited to the illustrated examples. Accordingly, a state other than the second state in which the pivot frame (M) is located in front of the user may be referred to as the first state. For example, the pivot frame (M) of the wearable device (1) may be formed in a structure that is connected to / disconnected from the mounting frame (H) of the wearable device (1). Accordingly, the wearable device (1) may include a second state in which the pivot frame (M) is directly or indirectly connected to the mounting frame (H) and located in front of the user, and a first state in which the pivot frame (M) is directly or indirectly separated from the mounting frame (H) and located in a position other than in front of the user.

[0079] Referring to FIGS. 2 and 3, a wearable device (1) according to one embodiment of the present disclosure can provide visual information to a user through a display device (DR, DL) by changing from the first state to the second state, and can provide auditory information to a user through a pair of earcups (101, 102).

[0080] According to one embodiment of the present disclosure, a wearable device (1) may include a mounting frame (H) that connects a pair of ear cups (101, 102) to each other. When a user wears the wearable device (1), the mounting frame (H) may be mounted on the user's head. The mounting frame (H) may be named a connecting member or a head band. Referring to FIG. 2, a right connecting portion (301) and a left connecting portion (not shown) may extend along the mounting frame (H), and, for example, the right connecting portion (301) and the left connecting portion (302) may extend along the outer side of the mounting frame (H).

[0081] The description below regarding the right ear cup (101), right pivot part (201), right connecting part (301), and the connection relationship between them can be applied substantially the same to the left ear cup (102), left pivot part (202), left connecting part (not shown), and the connection relationship between them, to the extent that they do not conflict with each other.

[0082] Referring to FIGS. 2 and 3, at least one sensor according to one embodiment (e.g., at least one sensor (460) of FIG. 4) may be placed on a rotating frame (M). Additionally, for example, at least one sensor may be placed on at least one of a right rotating part (201) or a left rotating part (202). However, it is not limited thereto. In one example, at least one sensor may be placed on the rotating frame (M) and an ear cup (e.g., at least one of a right ear cup (101) or a left ear cup (102)).

[0083] According to one embodiment, a wearable device (1) can determine the state of a rotating frame (M) through at least one sensor (not shown). For example, the wearable device (1) can sense through at least one sensor (not shown) whether the rotating frame (M) is in a second state located at a second position adjacent to the user's eyes, or in a first state located at a first position not adjacent to the user's eyes (e.g., above the head).

[0084] Referring to FIGS. 2 and 3, at least one sensor according to one embodiment (e.g., at least one sensor (460) of FIG. 4) may include at least one image sensor (461). The image sensor (461) according to one embodiment may be placed at various locations on the wearable device (1), but is not limited thereto. For example, at least one image sensor (460) may be placed to acquire image data (or video data) in the same direction as the user's field of view (FOV) when the rotating frame (M) is in a first state where it is not adjacent to the user's eye. Additionally, at least one image sensor (460) may be placed to acquire image data (or video data) in the same direction as the user's field of view (FOV) when the rotating frame (M) is in a second state where it is adjacent to the user's eye.

[0085] That is, a wearable device (1) according to one embodiment may include at least two image sensors (461), and, referring to FIG. 2, the image sensors (461) may be positioned such that when the rotating frame (M) is in a first position, they are oriented in the same direction as the user's field of view (FOV), and, referring to FIG. 3, when the rotating frame (M) is in a second position, they are oriented in the same direction as the user's field of view (FOV). However, this is not limited thereto, and the image sensors (461) may be positioned to obtain various directions and / or multiple image data (or video data) according to the user's convenience. That is, it can be understood that at least one image sensor (461) may be positioned in the wearable device (1) to obtain image data (or video data) regarding the external environment (e.g., front, side, rear).

[0086] According to various embodiments, the image sensor (461) has a viewing direction that changes depending on the state of the rotating frame (M), and accordingly, the control mode or information acquisition position can be automatically switched.

[0087] FIG. 4 is a block diagram of a wearable device according to various embodiments.

[0088] The configuration of FIG. 4 may be referenced by the configuration of other drawings to the extent that they are not disposed of from each other. The same terms and / or the same reference numerals have been used for configurations that are identical or substantially identical to the configuration of other drawings.

[0089] Referring to FIG. 4, a wearable device (1) may include a processor (410), at least one display (420) (hereinafter referred to as display (420) for convenience), memory (430), a communication circuit (440), at least one speaker (450) (hereinafter referred to as speaker (450) for convenience), and at least one sensor (460) (hereinafter referred to as sensor (460) for convenience). The listed components may be operatively or electrically connected to each other. As an example, some of the components of the wearable device (1) illustrated in FIG. 4 may be modified, deleted, or added.

[0090] According to various embodiments, the wearable device (1) may include a processor (410). In various embodiments, the processor (410) may execute software (e.g., a program) to control at least one other component (e.g., a hardware or software component) of the wearable device (1) connected to the processor (410) and may perform various data processing or operations. According to various embodiments, as at least part of the data processing or operations, the processor (410) may store commands or data received from another component (e.g., a communication circuit (440)) in volatile memory, process the commands or data stored in volatile memory, and store the resulting data in non-volatile memory. According to various embodiments, the processor (410) may include a main processor (e.g., a central processing unit) or an auxiliary processor (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with it. For example, if the wearable device (1) includes a main processor and an auxiliary processor, the auxiliary processor may be configured to use less power than the main processor or to be specialized for a specified function. The auxiliary processor may be implemented separately from the main processor or as part of it.

[0091] An auxiliary processor may control at least some of the functions or states associated with at least one component of the wearable device (1) (e.g., display (420), or communication circuit (440)) on behalf of the main processor while the main processor is in an inactive (e.g., sleep) state, or together with the main processor while the main processor is in an active (e.g., application execution) state. According to various embodiments, the auxiliary processor (e.g., communication processor) may be implemented as part of another functionally related component (e.g., communication circuit (440)). According to various embodiments, the auxiliary processor (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the wearable device (1) itself where the artificial intelligence is performed, or through a separate server.

[0092] According to various embodiments, the processor (410) can execute operations or data processing regarding the control and / or communication of at least one other component of the wearable device (1) using instructions stored in memory (430). According to one embodiment, the processor (410) may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a micro controller unit (MCU), a sensor hub, a supplementary processor, a communication processor, an application processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a Neural Processing Unit (NPU), and may have multiple cores.

[0093] According to various embodiments, the wearable device (1) may include a display (420). The display (420) may visually provide information to the outside of the wearable device (1) (e.g., a user). According to various embodiments, the display (420) may display various content (e.g., text, images, videos, icons, and / or symbols). According to various embodiments, the display (420) may include a liquid crystal display (LCD), a light-emitting diode (LED) display, or an organic light-emitting diode (OLED) display. According to various embodiments, the display (420) may be composed of various displays (420) that allow image light to be emitted into the user's pupil through a lens portion. For example, the display (420) may include various displays such as a laser display, an LCOS display, an LED display, etc. The structure of the lens portion may be changed depending on the type of display (420) of the wearable device (1).

[0094] According to various embodiments, the wearable device (1) may include a memory (430). According to various embodiments, the memory (430) may store various data used by at least one component of the wearable device (1) (e.g., processor (410)). The data may include, for example, input data or output data for software (e.g., a program) and related commands. The memory (430) may include volatile memory or non-volatile memory.

[0095] According to various embodiments, the program may be stored in memory (430) as software and may include, for example, an operating system, middleware, or an application. According to various embodiments, memory (430) may store instructions that allow the processor (410) to process data or control components of the wearable device (1) to perform operations of the wearable device (1) at runtime. The instructions may include code generated by a compiler or code that can be executed by an interpreter.

[0096] According to various embodiments, the memory (430) can store various information obtained through the processor (410). For example, the memory (430) can store configuration information for controlling the components of the wearable device (1). Accordingly, the processor (410) can control the components of the wearable device (1) so that the wearable device (1) can operate based on the configuration information stored in the memory (430).

[0097] According to various embodiments, the wearable device (1) may include a communication circuit (440). The communication circuit (440) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the wearable device (1) and an external device (3), a speaker (3), or a server (not shown), and the performance of communication through the established communication channel. The communication circuit (440) may include one or more communication processors that operate independently of the processor (410) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication circuit (130) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (e.g., a LAN (local area network) communication module, or a power line communication module). Among these communication modules, the corresponding communication modules can communicate with external electronic devices through a first network (e.g., a short-range communication network such as Bluetooth, WiFi Direct (wireless fidelity direct), or IrDA (infrared data association)) or a second network (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a long-range communication network such as a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0098] According to various embodiments, the wearable device (1) may include a speaker (450). For example, the wearable device (1) may be configured such that the speaker (450) is embedded in a pair of earcups (101, 102) configured to output sound.

[0099] According to one embodiment, auditory information may be provided to the user through a pair of earcups (101, 102). According to various embodiments, the wearable device (1) may control the output mode of the speaker (450). For example, when outputting media data, the wearable device (1) may control the speaker (450) to output sound according to various sound modes such as mono sound, stereo sound, spatial sound, surround sound, 3D sound, stereo sound, live sound, dynamic sound, etc.

[0100] According to one embodiment, the speaker (450) may include a speaker built into a pair of earcups (101, 102) and an external speaker configured to output sound. In the following description, the meaning that the wearable device (1) outputs information through the speaker (450) may include not only outputting a sound signal through a pair of earcups (101, 102) but also outputting a sound signal through an external speaker, and it can be understood that this may be appropriately considered for the convenience of the user. The external speaker is a configuration for providing sound to a conversational partner as well as to the user of the wearable device (1), and may be a configuration included in the wearable device (1), but may be a configuration not included in the wearable device (1) connected through a separate network (2).

[0101] According to various embodiments, the wearable device (1) may include a sensor (460). According to one embodiment, the wearable device (1) may obtain information regarding the rotation of the pivot frame (M) using the sensor (460). For example, the wearable device (1) may obtain information regarding the rotation of at least a portion of the pivot frame (M) relative to the ear cup using the sensor (460).

[0102] According to one embodiment, at least one sensor (460) may include a position sensor, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, a sound sensor, or an illuminance sensor. However, it is not limited thereto. Various other sensors may also be included and appropriately mounted / placed.

[0103] According to one embodiment, the position sensor may include at least one of a Hall sensor, a tunnel magnetoresistance (TMR) sensor, an anisotropic magneto-resistance (AMR) sensor, or a giant magneto-resistance (GMR) sensor. However, it is not limited thereto.

[0104] According to one embodiment, the image sensor (461) may include an image sensor (461) configured to capture an external environment to acquire a user's gesture or a person other than the user's gesture. According to various embodiments, the image sensor (461) may include components for capturing. For example, the image sensor (461) may include a lens assembly, an image sensor, a memory, and / or an image signal processor.

[0105] Meanwhile, the sensor (460) may include a sound sensor that acquires acoustic data by sensing external sounds (e.g., user speech, sounds of the surrounding environment).

[0106] According to various embodiments, the wearable device (1) may include various devices, not limited to the components described above. For example, the wearable device (1) may include an input device implemented to obtain certain information from outside the wearable device (1). For example, the input device may include a touch sensor and a physical key for receiving a user's physical input (e.g., touch) to the wearable device (1).

[0107] According to various embodiments, the wearable device (1) may include a connection terminal (not shown). The connection terminal may include a connector through which the wearable device (1) can be physically connected to an external electronic device (e.g., external device (3) of FIG. 1, speaker (3) of FIG. 1).

[0108] According to one embodiment, the wearable device (1) can enable the speaker (450) to output an acoustic signal obtained from a sound sensor as a spatial acoustic signal. For example, the wearable device (1) can obtain spatial information for a virtual space for generating spatial acoustics and output a screen for setting virtual speakers within the virtual space through a display (420). In one embodiment, the wearable device (1) can obtain acoustic setting information based on user input for the screen for setting virtual speakers and generate spatial acoustics based on the acoustic setting information. For example, the wearable device (1) can convert the obtained acoustic signal into a spatial acoustic signal based on the acoustic setting information and provide the spatial acoustics to the user.

[0109] According to various embodiments, the wearable device (1) can obtain sign language information and / or language service information based on processing sensing data obtained from at least one sensor (460) according to an auxiliary function required by the user, and output this to an ear cup (101, 102), display (420), speaker (450) or external device (3).

[0110] That is, the wearable device (1) can perform various operations to provide gesture-based sign language information and language conversion-based language service information to the user. According to various embodiments, the details of the wearable device (1) acquiring sensing data to provide user assistance functions and changing output information and output targets according to the state change of the wearable device (1) and / or the sensing data will be described in detail later with reference to FIGS. 5 to 20.

[0111] Each of the operations described below may be performed in combination with one another. Additionally, among the operations described below, an operation by the wearable device (1) may refer to an operation by the processor (410) of the wearable device (1).

[0112] In addition, the term “information” described below may be interpreted as having the meaning of “data” or “signal,” and the term “data” may be understood as a concept that includes both analog data and digital data.

[0113] According to various embodiments, the operations described below may be performed in various orders, not limited to the order shown. Additionally, according to various embodiments, more operations may be performed than those shown in the drawings, or at least one fewer operation may be performed.

[0114] Hereinafter, communication assistance functions that a wearable device (1) according to various embodiments can provide to a user will be described with reference to FIGS. 5 to 14. However, it should be understood that the embodiments are not limited to the contents of the drawings.

[0115] FIG. 5 is a diagram illustrating the usage environment of a wearable device according to various embodiments.

[0116] FIG. 6 is a flowchart illustrating the operation of a wearable device outputting literacy information according to various embodiments.

[0117] Referring to FIGS. 5 and 6, a wearable device (1) can be worn on a user's head to acquire image data and / or state data of a rotational frame (M) (601). More specifically, the wearable device (1) can acquire image data corresponding to the user's field of view (FOV) of the wearable device (1) through at least one image sensor (461). Additionally, the wearable device (1) can acquire state data of at least one of a first state and / or a second state of the rotational frame (M) based on at least one sensor (460).

[0118] Here, the video data may include, for example, gestures (e.g., gestures using hands, body movements including head movements, etc.) of a person (hereinafter, conversationalist) conversing with the user of the wearable device (1).

[0119] A wearable device (1) according to one embodiment can obtain sign language information corresponding to a gesture of a target (interlocutor) included in the image data based on processing image data obtained from at least one image sensor (461) (602). Here, the sign language information refers to information that identifies a linguistic meaning corresponding to the gesture of the other party, for example, and more specifically, the format of the sign language information may include text information corresponding to the linguistic meaning and voice information corresponding to the linguistic meaning.

[0120] For example, the output of sign language information by the wearable device (1) to the ear cups (101, 102) can be understood as outputting voice information corresponding to the linguistic meaning of the sign language information obtained based on the processing of image information by the wearable device (1). Likewise, the output of sign language information by the wearable device (1) to the display (420) can be understood as outputting text information corresponding to the linguistic meaning of the sign language information obtained based on the processing of image information by the wearable device (1), but is not limited thereto.

[0121] According to various embodiments, the wearable device (1) may output to an externally installed speaker (not shown) or an externally installed display (not shown) through network (2) communication with an external device (3).

[0122] Meanwhile, according to one embodiment, the wearable device (1) may perform a plurality of processing steps to obtain sign language information based on image data obtained through at least one image sensor (461). Below, an embodiment of obtaining sign language information based on image data will be described in detail.

[0123] As an example, a wearable device (1) can acquire gesture image data of a conversationalist located in front of or in the user's field of view (FOV) through at least one image sensor (461). The image data is sequence data consisting of multiple frames and may include the position, direction, speed, and motion pattern of a hand that changes over time.

[0124] According to various embodiments, the wearable device (1) can perform a preprocessing step to detect the position and shape of the hand from image data. For example, the processor (410) can perform image preprocessing operations such as background removal, contrast adjustment, and binarization on raw image data received from the image sensor (461), thereby extracting the hand region.

[0125] Next, the wearable device (1) can analyze the shape of the hand (e.g., extended, bent, or folded state of the fingers) and relative position information based on the extracted hand region. At this time, the processor (410) can recognize specific gesture movements in each frame or in a sequence of frames by utilizing a pre-trained deep learning-based gesture recognition model (e.g., CNN, RNN, or Transformer-based model, etc.).

[0126] For example, the processor (410) of the wearable device (1) can determine the motion pattern of a gesture performed during a specific time interval by estimating the shape and position of the hand in each frame and analyzing the movement between consecutive frames. This motion pattern can be mapped to a class corresponding to a specific sign language word or sentence through a classifier of a gesture recognition model.

[0127] According to one embodiment, the processor (410) can convert a linguistic meaning corresponding to a sign language recognition result into a form of sign language information including text information and / or voice information. In this process, a natural language processing (NLP) step may be included to convert the sign language word or sentence into a natural language sentence by considering the temporal order of each gesture sequence constituting the sign language word or sentence.

[0128] For example, a wearable device (1) can analyze a sequence of gestures that express everyday expressions such as "Have you eaten?" in sign language and convert them into text ("Have you eaten?") and / or voice ("Have you eaten?" voice output) corresponding to the meaning. Text information can be output through a display (420), and voice information can be output through earcups (101, 102) or an external speaker.

[0129] According to various embodiments, the wearable device (1) can be implemented to process gesture recognition and conversion processes at high speed so as to output sign language information in real time. At this time, by introducing a lightweight deep learning model or an NPU-based computational structure, it can be designed to enable real-time processing even in an on-device environment.

[0130] In addition, according to one embodiment, to improve the accuracy of sign language information, the wearable device (1) may provide an adaptive sign language recognition model for each user. For example, since the speed or range of sign language gestures may vary depending on the user, the wearable device (1) may improve recognition performance by providing a user-customized gesture recognition model learned based on previous usage history.

[0131] In addition, according to various embodiments, the wearable device (1) can perform a method of obtaining linguistic information by linking with an external device (3) in addition to an on-device processing method.

[0132] According to one embodiment, a wearable device (1) can transmit image data acquired through at least one image sensor (461) to an external device (3) via a network (2). At this time, the external device (3) can be configured to perform complex gesture recognition and natural language processing based on relatively high computational performance.

[0133] For example, the external device (3) may be a cloud server or a local high-performance device that utilizes a large-scale deep learning model to analyze video data received from the wearable device (1) and recognizes the gestures of the speaker to generate sign language information corresponding to the corresponding linguistic meaning. That is, the generated linguistic meaning may be converted into text information and / or voice information and then transmitted back to the wearable device (1), or provided directly through an output means (e.g., external display, external speaker) connected to the external device (3).

[0134] This method of linking with an external device (3) reduces the computational burden on the wearable device (1) and enables more precise and complex sign language recognition processing. In particular, by operating an artificial intelligence model that requires continuous updates based on a vast sign language dataset on the external device (3), there is an advantage of maintaining the latest recognition performance. Additionally, when processing acquired image data and sign language information, the wearable device (1) may adopt an encrypted data transmission and storage method to protect user personal information. For example, when communicating with the external device (3) via the network (2), a security protocol (TLS, SSL, etc.) may be applied to prevent data leakage.

[0135] Additionally, according to various embodiments, the wearable device (1) may selectively perform on-device processing and external device (3) interlocking processing depending on the state of the network (2), data throughput, user settings, etc. For example, simple or frequent gestures may be processed by the wearable device (1) itself, and complex or difficult-to-recognize gestures may be processed through the external device (3) in a hybrid manner.

[0136] Through such configuration and operation, the wearable device (1) can efficiently and accurately recognize the speaker's sign language gestures and convert them into linguistic meanings to output as text or voice, thereby supporting sign language-based two-way communication. In particular, the combined operation of the on-device and the external device (3) provides a processing method optimized for the usage environment, thereby satisfying real-time, accuracy, and energy efficiency.

[0137] Referring to FIG. 6, the wearable device (1) can identify the state of the rotating frame (M) in response to performing operation 602 (603). More specifically, the wearable device (1) can identify the current state of the rotating frame (M) using at least one sensor (460) (603). For example, the sensor (460) may include a gyroscope sensor, an accelerometer sensor, a position sensor, etc., to determine whether the rotating frame (M) is in a second state located in front of the user's field of vision, or in a first state located in a non-field of vision area such as above the user's head.

[0138] FIG. 7 is a drawing illustrating a first state usage environment of a wearable device (1) according to various embodiments.

[0139] FIG. 8 is a drawing illustrating a second state usage environment of a wearable device (1) according to various embodiments.

[0140] According to one embodiment, the wearable device (1) can determine an output means for outputting sign language information based on the state of the rotating frame (M) (604). For example, the processor (410) can be set to output text information through a display (420) when the rotating frame (M) is located in a second state (in front of the user's field of vision). On the other hand, when the rotating frame (M) is located in a first state (outside the user's field of vision), it can be set to output voice information through a speaker (450) built into the earcups (101, 102).

[0141] Referring to FIGS. 7 and 8, in the case where a conversationalist conversing with a user of a wearable device (1) according to one embodiment takes a gesture meaning "good," the wearable device (1) can obtain sign language information regarding the gesture-based linguistic meaning "good" through the above-described series of actions. Here, the sign language information may include text information of "good" regarding text and / or voice information of "good" regarding voice.

[0142] Accordingly, the wearable device (1) can identify the state of the rotating frame (M) and determine an output means for outputting sign language information corresponding to the state. For example, in response to acquiring sign language information, if the state of the rotating frame (M) is a first state, the wearable device (1) can output the sign language information to the ear cups (101, 102). For example, if sign language information having the meaning of 'good' is acquired, the wearable device (1) can output voice information (701) having the meaning of 'good' to the ear cups (101, 102).

[0143] According to various embodiments, the wearable device (1) may output the sign language information to the ear cups (101, 102) and / or the display (420) when the state of the rotating frame (M) is in a second state in response to the acquisition of sign language information. For example, when sign language information having the meaning of 'good' is acquired, the wearable device (1) may output voice information (801) having the meaning of 'good' to the ear cups (101, 102) and simultaneously output text information (802) having the meaning of 'good'. However, it is not limited thereto.

[0144] Additionally, according to various embodiments, the wearable device (1) can output sign language information based on preset language information stored in the memory (430). For example, if the user sets English, the sign language information can be output as English text and English voice, and if Korean is set, it can be output in Korean. Such multilingual support features enable efficient communication tailored to the user environment.

[0145] According to one embodiment, the wearable device (1) can automatically set the default language based on the user's location information or previous usage history, and the language can be changed via voice command or UI if necessary.

[0146] According to various embodiments, the wearable device (1) can flexibly change the output means depending on the user's situation or environment. For example, in an environment with loud ambient noise, a visual output (display) may be selected first, and when the user is walking or moving, an auditory output (speaker) may be selected. This method can be automatically determined by comprehensively considering the state data of the rotating frame (M) and additional environmental sensor data.

[0147] Next, the wearable device (1) can provide sign language information to the user through the output means determined in operation 604 (605). For example, the processor (410) can display the sign language information in the form of text on the display (420) or output it as voice through the ear cups (101, 102) or an external speaker.

[0148] According to one embodiment, the wearable device (1) can provide a user-customized interface depending on the output method. For example, when outputting via a display (420), in addition to text information, an icon or animation of the corresponding sign language gesture can be displayed together to help the user understand. On the other hand, when outputting voice, voice synthesis can be performed at a natural tone and speed of speech to provide an auditory experience similar to a real conversation.

[0149] That is, according to various embodiments, the wearable device (1) can provide a multifaceted interface (UI) to enhance user convenience. For example, when visual output is made through the display (420), the text size, font, color contrast, background theme (e.g., dark mode, high contrast mode), etc., can be configured so that the user can set them.

[0150] In addition, when providing auditory output, the speed, tone, and gender (male / female voice) of the voice can be selected, and a customized acoustic profile that emphasizes a specific frequency band can be provided by taking into account the user's hearing condition (e.g., a user with hearing loss).

[0151] Furthermore, according to various embodiments, the wearable device (1) can automatically optimize UI settings by learning the user's usage patterns. For example, it can automatically lower the display brightness at night and prioritize visual output over voice output in quiet environments.

[0152] Additionally, according to various embodiments, the wearable device (1) can automatically adjust the output mode based on user feedback or settings after outputting sign language information. For example, if the user frequently uses the display (420), a learning-based setting can be made to provide visual output preferentially when the rotating frame (M) is in a second state.

[0153] Meanwhile, when linked with an external device (3), the wearable device (1) may determine an external output means in operation 604. For example, in an environment where there is a large display connected to the external device (3), sign language information may be output to the display, or in a situation such as a conference room, voice information may be provided to multiple users through an external speaker.

[0154] According to another embodiment, the wearable device (1) may support manually switching output means based on user commands (e.g., voice commands, touch inputs, etc.) in addition to the state of the rotating frame (M). For example, when a user inputs a voice command such as "output as voice," the speaker output may be prioritized regardless of the current state of the rotating frame (M).

[0155] Through this series of operations (601 to 605), the wearable device (1) can provide sign language information in a manner optimized for the user's environment and state, thereby enabling more intuitive and efficient communication support. As a result, the wearable device (1) supports intuitive and efficient communication by providing sign language information optimized for the user's environment, state, and language settings, which can maximize usability in various situations.

[0156] FIG. 9 is a flowchart illustrating the operation of a wearable device outputting cipher information according to various embodiments.

[0157] FIG. 10 is a drawing for illustrating an embodiment of a second state of a wearable device according to various embodiments.

[0158] Referring to FIG. 9, the wearable device (1) can perform a series of operations to recognize a user's gesture and output sign language information.

[0159] More specifically, the wearable device (1) can acquire image data included within the user's field of view (FOV) through at least one image sensor (461) in operation 901.

[0160] Afterward, the wearable device (1) can process the image data obtained from motion 902 to obtain sign language information corresponding to the gesture. As previously described, this process is performed by a processor (410) using a deep learning-based gesture recognition model, and the recognized gesture sequence is converted into linguistic meaning through natural language processing (NLP).

[0161] Next, the wearable device (1) performs a step of identifying whether the gesture recognized in motion 903 is the gesture of the interlocutor or the gesture of the user. More specifically, the processor (410) can determine that it is the user's gesture by utilizing various parameters. For example, various parameters can distinguish whether the hand in the image is close (the user) or far (the interlocutor) through field of view analysis (when the position of the hand in the image data captured by the image sensor (461) is close to the center of the field of view (FOV)) or distance and size recognition (using information on the relative size and distance of the hand). To this end, a depth sensor or an image sensor (461) may be used as an auxiliary tool.

[0162] As another example, a method of recognizing a specific pattern (when the user performs a specific action that has been pre-set for recognizing their own gesture (e.g., a gesture of raising a hand at a specific angle), the processor (410) recognizes this as a trigger and confirms that it is the user's own gesture) may be utilized.

[0163] In addition, by utilizing sensor fusion data (e.g., using data from IMU sensors (gyroscope, accelerometer, etc.) to analyze the user's arm or wrist movements in conjunction with hand movements in video data), it is possible to accurately identify whether the subject of the gesture is the user themselves or the conversational partner.

[0164] Through such complex analysis, the wearable device (1) can accurately distinguish the subject of the gesture and prevent errors caused by incorrect recognition.

[0165] When the processor (410) of the wearable device (1) identifies that the gesture belongs to the user, it can output sign language information obtained from action 904 through an external speaker (not shown). Through this, the sign language gesture performed by the user is converted into voice information and transmitted to the other party or people around. For example, if the user performs a sign language gesture saying "Good," the wearable device (1) recognizes this and can output voice information saying "Good" through an external speaker.

[0166] Referring to FIG. 10, a case is illustrated in which a user of a wearable device (1) according to one embodiment directly performs a sign language gesture having the meaning of 'good'.

[0167] At this time, the wearable device (1) can recognize the gesture within the user's field of view (FOV) and process it to generate text information (1001) and voice information (1002). More specifically, the wearable device (1) can apply different output means depending on the state of the rotation frame (M).

[0168] When the rotating frame (M) is in a second state (when the rotating frame (M) is located in front of the user's field of view (FOV), the wearable device (1) can visually output text information (1001) that says 'Good' through the display (420) and simultaneously output voice information (1002) through an external speaker or ear cup. Through this, the user can intuitively see that their sign language gestures are converted in real time and transmitted to the other party.

[0169] According to various embodiments, the wearable device (1) can detect the noise level of the external environment through a sound sensor (e.g., a microphone) included in at least one sensor (460) and automatically adjust the output method and volume of the external speaker. For example, the wearable device (1) can monitor the ambient noise level in real time through a built-in microphone or noise sensor, and if the noise is higher than a certain threshold, it can automatically amplify the volume of the external speaker or set it to prioritize visual output over voice output. For example, in a relatively quiet environment, the volume of the voice output can be appropriately adjusted to prevent the generation of unnecessary noise.

[0170] In addition, according to various embodiments, the user can turn ON / OFF the wearable device (1) providing an automatic volume adjustment function according to the environment through user input, and can also adjust the noise threshold. Therefore, the wearable device (1) provides a user-customized environment response function, thereby supporting efficient communication in any situation.

[0171] Meanwhile, when the rotating frame (M) is in the first state (when the rotating frame (M) is located outside the user's field of vision), the use of the display (420) is restricted, so the wearable device (1) can output only voice information through an external speaker (not shown). For example, if the user performs a sign language gesture corresponding to "good" in the first state, the wearable device (1) can automatically select voice output and transmit the voice "good" to the surroundings.

[0172] According to various embodiments, the wearable device (1) can utilize a large display or an external speaker by linking with an external device (3) even in a first state according to the user's settings. For example, in a conference room environment, the wearable device (1) can be utilized in a way that automatically outputs sign language information through an external display and provides voice information to multiple people.

[0173] Additionally, if privacy protection is required, the user can set the display (420) of another wearable device (1') connected via the network (2) instead of voice output, and can selectively set whether to output only for specific gestures.

[0174] Through such configuration and operation, the wearable device (1) can efficiently transmit the user's voluntary sign language gestures through various output means and can provide a smart interface that supports user-centered active communication. In particular, gesture subject identification technology and situation-optimized output methods can provide high convenience and usability in actual usage environments.

[0175] Referring to FIGS. 10 and 11, a wearable device (1) according to various embodiments may include a function to verify the accuracy of a sign language gesture performed directly by a user in real time and to immediately stop external speaker output when an error occurs. More specifically, the wearable device (1) is configured to recognize a user's gesture in a second state and, when the generated sign language information is output through a display (420), allow the user to visually check the information.

[0176] According to one embodiment, the processor (410) of the wearable device (1) may delay external speaker output for a certain period of time while the user checks text information displayed on the display (420) after the user’s gesture has been converted into sign language information. The user may immediately review whether the text information displayed on the display (420) matches their intention, and if inaccurate sign language information is generated due to incorrect gesture recognition, the user may stop external speaker output and request re-recognition through user input (e.g., specific gesture, touch input, voice command, etc.).

[0177] Additionally, according to various embodiments, the wearable device (1) may be configured to immediately stop output by reflecting real-time feedback from the user even while voice output through an external speaker is in progress. For example, after the user performs a sign language gesture, if the user checks text information displayed on the display (420) and recognizes an error while voice information is being output from the external speaker, the remaining voice output can be blocked through immediate user input (e.g., paper feeding gesture, touch input, voice command "stop," etc.).

[0178] According to one embodiment, the processor (410) can constantly monitor user interaction with text information displayed on the display (420) even during voice output, and control the output of the external speaker to stop immediately when a specific input set by the user is detected. Such a configuration can be utilized to prevent incorrect information from being transmitted externally, particularly in public places or quiet environments.

[0179] Additionally, according to various embodiments, the wearable device (1) may include an automatic verification function. For example, after recognizing the user's own gesture, the processor (410) may automatically determine whether to output based on a pre-set reliability standard (e.g., a gesture recognition accuracy threshold). If the recognized sign language information is below the reliability standard, voice output through an external speaker is blocked, and a guidance message such as "re-recognition required" is displayed on the display (420), thereby preventing unnecessary mis-output. As a result, the user can perform more accurate and reliable communication and minimize unnecessary misunderstandings.

[0180] FIG. 11 is a drawing for illustrating an embodiment of a wearable device changing state from a first state to a second state according to various embodiments.

[0181] Referring to FIG. 11, the wearable device (1) can freely switch between a first state and a second state by changing the position of the rotating frame (M) according to the user's situation or conversation method. This state change directly or indirectly affects the method of outputting sign language information and is designed to support more efficient and intuitive communication.

[0182] For example, consider the case where a user switches the wearable device (1) from a first state to a second state. In this case, if the conversationalist communicates through sign language, initially, sign language information is provided only through voice output (earcups (101, 102)). However, when the rotating frame (M) switches to the second state, the wearable device (1) can activate the display (420) to additionally provide sign language information to the user in the form of text. Accordingly, the user can listen to the voice output while simultaneously visually checking the sign language information through the display (420), thereby enabling clearer and more intuitive communication.

[0183] Conversely, when the user switches the wearable device (1) from the second state to the first state, the use of the display (420) is restricted, so the wearable device (1) automatically stops text output and provides sign language information only as voice output.

[0184] Meanwhile, when a user takes a gesture related to sign language, and the wearable device (1) is switched from a first state to a second state, initially, sign language information is provided only through an external speaker and / or ear cup (101, 102). However, when the rotating frame (M) is switched to the second state, the wearable device (1) activates the display (420) so that the user can check sign language information corresponding to their sign language in text form. Through this, the user can increase the reliability of communication by confirming whether their sign language is being clearly transmitted to the other party.

[0185] FIG. 12 is a diagram illustrating the usage environment of a wearable device according to various embodiments.

[0186] FIG. 13 is a drawing for explaining an embodiment of a second state of a wearable device according to various embodiments in the usage environment of FIG. 12.

[0187] Referring to FIG. 12, a situation is illustrated in which an obstacle (11) is located in front of a user while the user is walking with the user wearing the wearable device (1). The wearable device (1) can detect the obstacle (11) located within the user's field of view (FOV) or on the path ahead in real time through at least one image sensor (461) and / or a distance sensor (not shown).

[0188] More specifically, the processor (410) of the wearable device (1) performs basic image preprocessing operations on the image data, such as noise removal, contrast adjustment, and edge detection, to clearly distinguish object candidate regions. In this process, the outline of an object that is distinct from the background is extracted, and a stationary object and a moving object can be distinguished. The preprocessed image data is analyzed through a pre-trained deep learning-based object recognition model (CNN, YOLO, SSD, etc.). Through this, the processor (410) can identify objects that can be judged as obstacles within the image (e.g., pillars, benches, walls, vehicles, etc.) and obtain location coordinates and size information of each object.

[0189] According to one embodiment, the processor (410) of the wearable device (1) can analyze image data to determine whether there is an obstacle (11) in front, its distance, size, or movement, and can operate to provide a warning message to the user when an obstacle is detected within a certain distance.

[0190] According to various embodiments, the wearable device (1) can more precisely determine the location, distance, and direction of movement of an obstacle by fusion analyzing data such as an image sensor (461), a distance sensor (not shown), and an IMU sensor.

[0191] Referring to FIG. 13, the wearable device (1) can provide a warning message (1301) through the earcups (101, 102) in response to a detected obstacle (11). For example, by outputting a guidance voice such as "There is an obstacle 2 meters ahead," the user can immediately recognize the situation and respond safely.

[0192] According to various embodiments, the wearable device (1) can flexibly adjust the content and method of the warning message to suit the situation.

[0193] According to one embodiment, the wearable device (1) can provide a distance-based warning based on processing video data. For example, the warning intensity can be adjusted in stages according to the distance from the obstacle. For example, a voice message reflecting the urgency can be output to the earcups (101, 102), such as "There is an obstacle ahead" when 5 meters ahead, and "Danger! Obstacle right ahead" when approaching within 2 meters.

[0194] Additionally, according to one embodiment, the wearable device (1) can identify the type of obstacle. For example, the processor (410) can utilize AI-based object recognition technology to analyze the type of obstacle (e.g., car, pillar, person, etc.) and provide specific information such as "there is a vehicle ahead" or "there is a pedestrian ahead".

[0195] In addition, according to one embodiment, the wearable device (1) can provide direction guidance based on processing video data. According to various embodiments, the wearable device (1) can provide guidance not only on simple warnings but also on avoidance paths. For example, it can support the safe movement of the user by additionally providing instructions such as "move 1 meter to the left."

[0196] Additionally, according to one embodiment, the wearable device (1) can provide vibration feedback based on processing image data. For example, in addition to voice guidance, the wearable device (1) can provide vibration signals through a vibration motor built into the wearable device (1) to effectively provide warnings in various environments (e.g., noisy outdoors). For example, intuitive warnings may be possible by providing different vibration patterns depending on the urgency of the obstacle.

[0197] In addition, according to one embodiment, the wearable device (1) can provide a warning based on user-customized settings through the processing of image data. For example, the user can select or combine the warning methods (voice, vibration, visual notification, etc.) described above according to their personal needs.

[0198] Meanwhile, a wearable device (1) according to one embodiment can provide a warning to a user by linking a map and GPS. For example, the wearable device (1) can link with an external device (3) to provide advance information about fixed obstacles (e.g., stairs, construction zones, etc.) based on GPS and map data, and can provide advance warnings about risk factors on the expected path.

[0199] Through these various embodiments, the wearable device (1) can provide a safer walking environment for users, including visually impaired people, and effectively support the user's daily life with real-time obstacle recognition and an intuitive warning system. In particular, the situation-specific customized warning function can be utilized as a smart mobility support solution by providing assistance functions optimized for the user's environment and preferences.

[0200] FIG. 14 is a diagram illustrating the usage environment of a wearable device according to various embodiments.

[0201] Referring to FIG. 14, the wearable device (1) can support safe driving for the hearing impaired by detecting major acoustic signals occurring in the surrounding environment in real time and providing them visually when the user is driving a vehicle. More specifically, the wearable device (1) continuously collects ambient sounds through at least one sound sensor (462), and the processor (410) analyzes them to identify major acoustic signals, such as specific event sounds, for example, the horn of another vehicle, the siren of an emergency vehicle, or a pedestrian warning sound.

[0202] According to one embodiment, the processor (410) can classify the type of sound based on collected acoustic data and analyze the direction of the sound's occurrence. To this end, the wearable device (1) utilizes a plurality of microphone arrays to identify the direction of the sound and evaluates the urgency by comprehensively analyzing the frequency, pattern, volume, etc. of the acoustic signal. For example, if a sudden, high-volume horn is detected, the processor (410) can immediately recognize this and provide a warning to the user.

[0203] As illustrated in FIG. 14, the wearable device (1) can visually display the detected acoustic signal through a display (420) within the user's field of view (FOV). For example, if a horn is sounded from the left rear, a warning icon or flashing indicator is provided at the bottom left (1401) of the display (420), and if a sound is detected from the right front, a corresponding indicator is provided at the top right (1402), thereby enabling the user to intuitively recognize the direction of the sound.

[0204] According to various embodiments, the wearable device (1) can provide different visual effects depending on the type of sound. For example, a regular vehicle horn can be displayed as a yellow warning icon, and an emergency vehicle siren can be displayed as a text notification saying "Emergency vehicle approaching" along with a red flashing effect. In addition, the user's attention can be effectively drawn by changing the icon size, flashing speed, and color on the display according to the urgency of the sound.

[0205] According to one embodiment, the wearable device (1) can provide summary information about the detected acoustic signal in real time and, if necessary, display a brief text-based guidance message on the display (420). For example, by providing specific information such as "right rear vehicle horn" or "left emergency vehicle approaching," the user can quickly recognize the situation and respond appropriately.

[0206] In addition, according to various embodiments, the wearable device (1) can automatically adjust the frequency and method of notifications by taking into account the driver's driving environment, speed, and ambient noise level. For example, when driving at high speeds, warnings can be provided with faster flashing speeds and conspicuous colors, and in congested areas, relatively less urgent warnings can be displayed in a reduced form.

[0207] Meanwhile, the user can customize the notification method (e.g., simple icon display, additional text display, etc.) to suit their personal preferences through the settings of the wearable device (1), and can disable notifications for specific sound signals or adjust the priority. For example, the notification can be set to be minimized for noise of a specific frequency that occurs repeatedly (e.g., construction noise).

[0208] Through such various configurations and operations, the wearable device (1) can support a safer and more efficient driving environment by visually providing key sound information that may occur while driving to drivers, including those with hearing impairments. In particular, real-time sound recognition and direction guidance functions enable a quick response to unexpected situations, thereby simultaneously preventing accidents and reducing driver stress.

[0209] According to various embodiments, the wearable device (1) can be used more broadly by visually providing various sound information that a hearing-impaired person may encounter in daily life, in addition to driving situations. For example, when a user is walking in the city or using public transportation, the wearable device (1) can detect horn sounds, sirens, announcements, or emergency alarm sounds occurring in the surroundings in real time and visually convey them through a display (420).

[0210] According to one embodiment, a wearable device (1) can automatically recognize an emergency sound signal having a specific frequency band or pattern and provide an immediate visual notification to the user's field of view (FOV). For example, in multi-use facilities such as subway stations or airports, a message saying "Emergency announcement has been detected" can be displayed on the display, or if a fire alarm is detected, a warning saying "Fire alarm has occurred" can be provided with a strong flashing effect.

[0211] In addition, according to various embodiments, the wearable device (1) can provide customized notification services based on the user's lifestyle patterns. For example, when the user is living at home, it can detect sound signals such as a doorbell, a phone ringtone, or a baby crying, and display the corresponding event on the display (420) in the form of text and icons. Through this, various sound information in daily life that is easily missed by the hearing impaired can be intuitively provided.

[0212] This configuration can be usefully utilized not only by the hearing impaired but also in situations where noise needs to be temporarily blocked (e.g., while wearing headphones, in noisy work environments, etc.), and the wearable device (1) can serve as a smart device that provides an optimized sound perception assistance function according to the user's environment changes.

[0213] According to various embodiments, the wearable device (1) can provide an efficient sound perception assistance function even in the daily life of a hearing-impaired person. More specifically, the wearable device (1) can collect ambient sounds in real time through at least one sound sensor (462), and the processor (410) can analyze this to filter only the sounds needed by the user and provide them visually on the display (420). For example, everyday background noise (e.g., street noise, wind sound, people's chatter, etc.) can be removed, and only important event sounds, such as doorbell sounds, warning sounds, and phone ringtones, can be selected and notified to the user.

[0214] According to one embodiment, the wearable device (1) may intuitively provide information related to the direction of sound generation by combining an image sensor (461) and a sound sensor (462). For example, when a specific sound is detected, the wearable device (1) may analyze the direction of the sound and output a direction indicator icon or a visual signal on the display (420) to help the user easily identify the location of the sound. For example, if someone calls from the rear right side of the user, the notification may be displayed at the bottom right of the display (420).

[0215] Additionally, according to various embodiments, the wearable device (1) may include a function to selectively recognize only the voice of the person being conversed with and to remove unnecessary ambient noise. For example, an image sensor (461) may recognize the face and mouth movements of the person being conversed with, and in conjunction with a sound sensor (462), may intensively analyze only the acoustic data generated from that direction, thereby automatically filtering out environmental noise other than conversation. Through this, the hearing impaired can receive clearer and more focused conversational support, and effective communication is possible even in complex environments.

[0216] Additionally, the wearable device (1) can provide information about a tourist destination based on the user's location information. When the user arrives at a tourist destination while wearing the wearable device (1), surrounding information is collected through at least one sensor (460) and a communication circuit (440), and guidance information about the area is output through a display (420) or a speaker (450).

[0217] In addition, by recognizing information or signs provided in a foreign language using an image sensor (461), translating them into the user's language, and then displaying them on a display within the field of view or providing voice guidance, the user can enjoy sightseeing without language barriers.

[0218] FIG. 15 is a flowchart illustrating the operation of a wearable device outputting language service information according to various embodiments.

[0219] FIG. 16 is a drawing for illustrating an embodiment of a second state of a wearable device according to various embodiments.

[0220] Referring to FIG. 15, the wearable device (1) can perform a series of operations to provide language service information based on various sensing data according to the user's environment and situation.

[0221] More specifically, the wearable device (1) can acquire sensing data through at least one sensor (460) in operation 1501. Here, the sensing data may include, for example, voice data (speech of an external speaker collected through a voice sensor (462), image data (visual information such as books, signs, screens, etc. collected through an image sensor (461)), and state data of a rotation frame (M) (frame position information recognized through a posture sensor (463)).

[0222] Subsequently, the wearable device (1) can process the sensing data obtained in operation 1502 to generate language service information including language conversion information. More specifically, the processor (410) can obtain language conversion information by applying speech-to-text (STT), optical character recognition (OCR), and translation algorithms to convert the detected external language (second language, hereinafter referred to as 'second language' for convenience) data into the user's preset language information (e.g., first language, hereinafter referred to as 'first language' for convenience). Accordingly, the wearable device (1) can obtain language service information including voice information and text information regarding the language conversion information.

[0223] For example, a wearable device (1) can recognize second language voice data of an external speaker to obtain language conversion information regarding the first language, and obtain language service information regarding text or voice based on the language conversion information, thereby providing the language service information to the user.

[0224] Next, the wearable device (1) can identify the current state of the rotation frame (M) in operation 1503. For example, the processor (410) can use the posture sensor (463) to determine whether the rotation frame (M) is in a first state (outside the user's field of vision) or a second state (in front of the user's field of vision).

[0225] In operation 1504, based on the state of the identified rotating frame (M), it is determined which of at least one means to output language service information, namely a speaker (450) included in the ear cup (101, 102) or a display (420), will be used. For example, if the rotating frame (M) is in a second state, visual output (text, translated document, etc.) through the display (420) may be prioritized, and if it is in a first state, auditory output (provision of voice information) may be selected.

[0226] Finally, the wearable device (1) can provide language service information to the user through the output means determined in operation 1505. For example, the processor (410) can display the translated language information in the form of text on the display (420) or output it as voice through the speakers of the earcups (101, 102).

[0227] According to various embodiments, the wearable device (1) can recognize the contents of books, signs, menus, etc., written in a second language through an image sensor (461), translate them into the user's set language (first language), and display them in real time on a display (420). For example, if a user sees a notice written in a second language while overseas, the text is automatically translated and provided on a display within the user's field of vision, thereby enabling intuitive information delivery.

[0228] Referring to FIG. 16, when a user recognizes a document written in a second language, such as a book or a guide, in a second state, the wearable device (1) can recognize the corresponding phrase (1601) in real time through an image sensor (461) and output language service information (1602) converted into the user's set language (first language) through a display (420). More specifically, the processor (410) obtains language conversion information by utilizing OCR technology to extract text within image data and then performing translation into the first language through a built-in or externally linked translation algorithm.

[0229] In addition, the wearable device (1) can process language conversion information and provide language service information regarding voice to the earcups (101, 102). For example, when a user looks at a specific sentence while reading a book, the sentence may be translated and displayed on the display (420) or output as voice information (1603) through the earcups (101, 102) and / or an external speaker. Through this, the user can intuitively receive translation information in a visual and auditory manner, making it easier to understand foreign language documents.

[0230] According to various embodiments, the wearable device (1) can first recognize a specific highlighted area (e.g., highlighter mark, box processing, etc.) within the user's field of view (FOV) and set only the part the user is focusing on as the translation target. This minimizes unnecessary information translation and enables the provision of user-customized information.

[0231] Additionally, the wearable device (1) can flexibly adjust the translation speed, output language format (written / spoken), text size, voice output, etc. according to the user's settings. For example, if the user disables voice output, it can be set to provide only visual output through the display (420), or conversely, to perform only voice output when driving or moving.

[0232] Through this configuration and operation, the wearable device (1) can provide real-time translation services in various situations such as overseas travel, foreign language learning, and work environments, thereby providing intuitive and efficient language support functions to the user.

[0233] Additionally, according to various embodiments, the wearable device (1) can support smooth communication even when the languages ​​of the conversationalist and the user are different. For example, the wearable device (1) can recognize the conversationalist's second language speech through a voice sensor (462), translate it into the first language in real time, provide it as text on the display (420), or output it as voice through a speaker (450). This function effectively supports communication in a multilingual environment, and an appropriate output method can be automatically selected depending on the state of the rotating frame (M) and the user's environment.

[0234] Through this series of operations, the wearable device (1) can provide optimal language service information by comprehensively analyzing the user's visual and auditory environment, thereby resolving various language barriers and providing an intuitive and efficient user experience. In particular, the real-time translation and automatic switching function of the output method maximizes user convenience and can be utilized in various situations such as business, travel, and education, as well as in daily life.

[0235] According to various embodiments, the wearable device (1) provides a function that can automatically identify whether the speaker is the user or an external conversationalist based on voice data. To this end, the wearable device (1) may apply an algorithm that distinguishes the identity of the speaker by fusing the characteristics of voice data collected from the voice sensor (462) with additional sensor data.

[0236] More specifically, the processor (410) can identify the speaker through at least one of the following methods.

[0237] According to one embodiment, the processor (410) may utilize a voice profiling technique. The wearable device (1) stores the user's unique voice characteristics (e.g., timbre, frequency band, speech pattern, etc.) in advance during the user registration process or initial setup. Subsequently, by comparing and analyzing whether the voice data received in real time matches the previously registered user voice profile, it can determine whether the speaker is the same person. This process can be performed with high accuracy by applying machine learning-based speaker recognition technology.

[0238] According to one embodiment, the processor (410) may perform Direction of Arrival (DoA) analysis of the voice. The wearable device (1) may utilize a plurality of microphone arrays to detect the direction from which the voice originated. For example, a voice received from a location close to the user's mouth may be determined as the user's own speech, while a voice received from an external front or side direction may be recognized as the speech of the speaker. Through such spatial analysis, the accuracy of speaker identification can be improved.

[0239] According to one embodiment, the processor (410) may utilize a vibration sensor or a bone conduction sensor as an auxiliary component. Fine vibrations generated when a user speaks can be detected through a sensor embedded in the wearable device (1), thereby allowing for immediate recognition that it is one's own speech. For example, a bone conduction sensor mounted on the earcups (101, 102) can detect the user's skull vibrations to accurately determine whether speech is occurring.

[0240] According to one embodiment, the processor (410) may apply an identification method based on the recognition of the user's mouth movements. By monitoring the user's face or the area around the mouth with an image sensor (461) and detecting lip movement during speech, it can be further confirmed that the speech belongs to the user.

[0241] By comprehensively analyzing such various sensor data, the wearable device (1) can quickly and accurately distinguish whether the speaker is the speaker or a conversational partner even in a complex environment. In particular, the hybrid method combining voice profiling, direction analysis, and body-based sensing technology has the advantage of significantly lowering the false positive rate compared to a single method.

[0242] According to various embodiments, the wearable device (1) may be designed to select only a specific method according to the user's settings, or to automatically apply an optimal speaker identification algorithm depending on the situation. For example, in a noisy environment, a vibration-based identification method may be applied first, and in a relatively quiet environment, voice profiling and directionality analysis may be performed in parallel.

[0243] A processor (410) of a wearable device (1) according to various embodiments may receive a file containing text or voice data from an external device (3). The received file may be provided from various types of external devices, such as a user terminal, a server, or a cloud, and communication with the external device (3) is performed through a communication circuit (440). The processor (410) automatically recognizes the language of the received file and performs translation into a set target language.

[0244] The translated information can be visually output through a display (420) and is displayed at an appropriate location by a rotating frame (M) according to the user's field of vision. Additionally, the processor (410) can convert the translated content into speech and provide it to the user through a speaker (450) or ear cups (101, 102). This allows the user to easily understand files in various languages ​​provided by an external device and to select the appropriate output between visual and auditory methods.

[0245] This configuration can be particularly useful when files such as meeting materials, guide documents, or emails need to be translated and checked in real time. Users can perform tasks efficiently without language barriers by receiving automatically translated information through a wearable device (1) without any separate translation work.

[0246] FIG. 17 is a diagram illustrating a multiple speaker usage environment of a wearable device according to various embodiments.

[0247] Referring to FIG. 17, when a speaker (1701) speaks in a second language, the wearable device (1) receives the speaker's voice data through a voice sensor (462). The received voice data (1701) in the second language is processed in real-time by a processor (410) using speech-to-text (STT) and converted into text data.

[0248] The converted second language text is then translated into the first language, which is the user's set language, through a translation module. This translation process can be performed by an on-device translation engine of the wearable device (1) or by a cloud-based translation system of an external device (3) connected via a network (2).

[0249] Language service information regarding the translated first language is provided in two forms. First, the processor (410) generates voice information (1702) corresponding to the first language through Text-to-Speech (TTS) and outputs it through the built-in speaker (450) of the earcup (101, 102). Through this, the user can listen to the speaker's speech in their own language in real time.

[0250] Simultaneously or optionally, when the rotating frame (M) is in a second state, the wearable device (1) can visually output text information (1703) translated into a first language to the user through the display (420). This allows the user to understand the content of the conversation more accurately through visual assistance in addition to auditory information.

[0251] Meanwhile, when the rotating frame (M) is in the first state, the wearable device (1) has limited visual output, so the output of text information of the first language is omitted, and only the voice information of the first language (1702) can be provided through the ear cup. This configuration can be designed to respond flexibly according to the user's field of vision and environmental conditions.

[0252] FIG. 18 is a diagram illustrating a multiple speaker usage environment of a wearable device according to various embodiments.

[0253] Referring to FIG. 18, the user speaks in a first language, and this spoken voice data (1801) is acquired in real time through a voice sensor (462) of a wearable device (1). The processor (410) converts the voice data into text data through speech recognition (STT), and then converts it into a second language, which is the language of the speaker, using a translation engine.

[0254] The converted second language is converted into voice data (1802) through a Text-to-Speech (TTS) engine and output through an external speaker. This allows the speaker to listen in real-time to the utterance spoken by the user in the first language in their own language (second language). This is one of the key functions for realizing two-way interpretation in an auditory-centered conversational environment.

[0255] Simultaneously or optionally, the wearable device (1) can provide text information (1803) of the first language in real time so that the user can check whether their speech content has been accurately recognized in the first language through the display (420). Through this, the user can immediately identify whether there is a recognition error or a contextual error, and if necessary, correct or re-speech can be performed.

[0256] Meanwhile, when the rotating frame (M) is in the first state, the wearable device (1) has limited display usage, so text information corresponding to 1803 is not output, and only the second language voice output (1802) through an external speaker is maintained. The wearable device (1) can provide text output separately by linking with an external device (3), such as a smartphone or another wearable device (1'), according to the user's settings.

[0257] Referring to FIGS. 17 and 18, it can be understood that a wearable device (1) according to various embodiments determines the voice information corresponding to the language service information including language conversion information differently in the earcups (101, 102) or external speaker (not shown) depending on the type of speaker.

[0258] More specifically, referring to FIG. 17, when the speaker is a conversationalist, that is, when an outsider speaks in a second language, the wearable device (1) recognizes the speech and generates language service information converted into the first language, which is the user's preset language. At this time, voice information (1702) regarding the generated first language is output through earcups (101, 102) so that the user can listen to it directly. This configuration allows the user to receive the speech of an external conversationalist in their own language in real time and understand it smoothly. Output through earcups ensures the user's personal listening environment and has the advantage of maintaining privacy because it is not exposed to the outside.

[0259] On the other hand, referring to FIG. 18, when the speaker is the user himself, that is, when the user speaks in the first language, the wearable device (1) converts this into the second language, which is the language of the conversationalist, to generate language service information. At this time, since the voice information (1802) regarding the generated second language needs to be delivered to the conversationalist, it is output through an external speaker (not shown) rather than an ear cup. Output through the external speaker allows the user's speech to be directly delivered to a nearby conversationalist in a translated form, thereby enabling real-time interpretation functions. Accordingly, the wearable device (1) according to various embodiments can provide an optimized communication environment for both the user and the conversationalist by distinguishing and applying a voice output means for language service information including language conversion information depending on the type of speaker, that is, whether the speaker is the user himself or an external conversationalist.

[0260] According to various embodiments, the wearable device (1) can individually control voice output and text output through voice commands or user settings, and may include a function to automatically switch output means according to changes in the conversational environment.

[0261] Referring to FIGS. 17 and 18, a situation can be considered in which the pivot frame (M) transitions from a second state to a first state while the user and the interlocutor are having a conversation using different languages. For example, this applies to the case where the user tilts the pivot frame (M) upward to change to the first state due to a change in the external environment (e.g., it becoming dark).

[0262] At this time, the display (420) is out of the user's field of vision, so the means of visual output is limited. Accordingly, the wearable device (1) immediately stops the display output and switches the language service information previously provided through the display to an auditory output method and outputs it as voice through the speaker (450) of the ear cup (101, 102).

[0263] For example, in the case of FIG. 17, while information spoken by a speaker in a second language is simultaneously provided to the user as text (1703) and voice (1702), when the rotation frame (M) changes to a first state, the text output is stopped and only the voice output is maintained. Likewise, in the case of FIG. 18, while the first language spoken by the user is provided as real-time text (1803) on the display (420), when the rotation frame (M) changes to a first state, only the second language voice output (1802) through an external speaker is maintained and visual feedback is omitted.

[0264] This switching process is performed in real time by the processor (410) of the wearable device (1) continuously monitoring the state data of the rotating frame (based on the posture sensor (463)). Depending on the settings, the user can control this switching method (visual -> auditory output switching) manually or automatically. The opposite situation (first state <- second state) can also be understood.

[0265] According to various embodiments, the wearable device (1) may include a function for managing the user's communication history. More specifically, the wearable device (1) may automatically store a conversation history including sign language information and language service information in memory (430). For example, real-time sign language recognition results, translated language information, voice output content, etc., may be recorded in chronological order to support the user in recalling or reviewing the conversation content later.

[0266] According to one embodiment, the wearable device (1) may include a function for summarizing stored conversation history. The processor (410) can utilize a natural language processing (NLP)-based summarization algorithm to automatically organize long conversation content around key keywords and main sentences. For example, in a meeting situation or during a business discussion, the wearable device (1) can provide summary information to the display (420) in the form of "Meeting main points: project schedule coordination, budget review, discussion of next meeting schedule."

[0267] In addition, according to various embodiments, the wearable device (1) can manually activate the history saving and summary functions through user commands (e.g., "save conversation," "view summary"), and can also be set to operate automatically when specific conditions occur (e.g., conversation lasting for a certain period of time, detection of specific keywords, etc.). The saved conversation history is encrypted for security, and the user can back up or share data by linking with an external device (3) when necessary. Through this, the wearable device (1) can provide smart functions that support not only simple real-time communication assistance but also conversation history management and information organization.

[0268] FIG. 19 is a flowchart illustrating the operation of a wearable device according to various embodiments outputting language service information when the speaker is the user.

[0269] Referring to FIG. 19, the wearable device (1) can recognize the user's own speech to acquire (generate) language service information in real time and provide it in an appropriate output manner according to the state of the rotation frame (M).

[0270] More specifically, the voice sensor (462) of the wearable device (1) can acquire the user's own speech content as voice data, and the processor (410) converts the acquired voice data into text information through STT (Speech-To-Text) technology. Subsequently, the text information is compared with pre-set language information and, if necessary, translated into a second language, and the translated information is converted into voice through TTS (Text-to-Speech).

[0271] Subsequently, the processor (410) of the wearable device (1) distinguishes whether the speaker is the user or a conversationalist based on the processing of the sensing data obtained in operation 1901. If it is voice spoken directly by the user, the wearable device (1) identifies the state of the rotation frame (M) (operation 1902) and determines whether the state is the first state (operation 1903).

[0272] When the rotating frame (M) is in the first state (i.e., when the display is located outside the user's field of vision), the processor (410) can be configured to output the generated language service information through an external speaker (operation 1904). At this time, the user's voice is converted into a second language and then transmitted to the interlocutor through the external speaker.

[0273] On the other hand, when the rotating frame (M) is in a second state (i.e., when the display is located in front of the user's field of vision), the wearable device (1) can output language service information in text form through the display (420) and at the same time provide voice information through an external speaker (operations 1905, 1906). The user can check in real time through the display whether the content they spoke is accurately recognized and translated, which enables more accurate communication.

[0274] This flow enables active communication by generating language service information based on the user's utterance and selecting an appropriate output channel (visual or auditory) according to the physical state of the rotation frame.

[0275] FIG. 20 is a flowchart illustrating the operation of a wearable device according to various embodiments outputting language service information when the speaker is not the person themselves.

[0276] Referring to FIG. 20, the wearable device (1) can recognize the speech of a conversationalist, generate language service information in real time, and provide it to the user in an optimized output manner according to the state of the rotation frame (M).

[0277] More specifically, the voice sensor (462) of the wearable device (1) acquires the utterance of an external conversationalist as voice data, and the processor (410) processes the sensing data in operation 2001 to identify that the speaker is an external conversationalist rather than the user (2001). This process can be performed through voice profiling, voice reception direction analysis (Direction of Arrival), or distance-based signal analysis.

[0278] Subsequently, the processor (410) converts the acquired speech data of the speaker into text information using STT (Speech-To-Text) technology, and then compares it with the user's pre-set language information (first language). If translation is required, it generates language service information converted into the first language through a translation module. The generated language service information may consist of text information and speech information via TTS (Text-to-Speech).

[0279] The processor (410) identifies the current state of the rotation frame (M) in operation 2002 and determines whether the rotation frame (M) is in the first state in operation 2003.

[0280] When the rotating frame (M) is in a first state (where the display is located outside the user's field of vision), the wearable device (1) can be configured to output voice information through the speaker (450) of the ear cup (101, 102) (2004). In this case, the user can listen to the speaker's second language speech in their own first language in real time, enabling natural auditory-based communication.

[0281] On the other hand, when the rotating frame (M) is in a second state (the display is positioned in front of the user's field of vision), the processor (410) outputs language service information in text form through the display (420) (Operation 2005). Additionally, voice information may also be provided through an ear cup or external speaker as needed (Operation 2006). Through this, the user can utilize visual and auditory information simultaneously to understand the speaker's speech more clearly.

[0282] According to various embodiments, the wearable device (1) can automatically adjust the output method according to the user's environment (e.g., ambient noise, movement) and settings, and, for example, prioritize text output in a noisy environment, or perform only voice output according to user settings.

[0283] Through such configuration and operation, the wearable device (1) can efficiently recognize and translate the speech of the speaker and provide intuitive and flexible language services to the user, thereby effectively supporting the user's communication in various multilingual communication environments.

[0284] A wearable device (1) according to various embodiments can be utilized in a multi-party conference environment where three or more speakers each use different languages. For example, a situation can be assumed where the first speaker participating in the conference uses Korean, the second speaker uses English, and the third speaker uses Chinese. In such an environment, the wearable device (1) can detect each speaker's language data in real time and generate interpretation information based on it, and provide it in the form of voice and visual output.

[0285] According to one embodiment, a voice sensor (462) of at least one sensor (460) collects the spoken voice of each speaker individually, and a processor (410) analyzes the voice data to identify each language. For example, when a first speaker speaks in Korean, the language is immediately recognized by the processor (410) of the wearable device (1), and language service information translated into English and Chinese is generated based on this. The generated interpretation information is provided to other speakers through an external display or an external speaker. In this case, it can be understood that language service information corresponding to the first speaker's language for feedback is output through the display (420).

[0286] According to the present embodiment, the wearable device (1) can automatically switch the output method depending on whether the speaker is the speaker themselves or another person. When the first speaker speaks, the first speaker's wearable device (1) translates the content of the speech into English and Chinese and transmits it to the other speakers' wearable devices (1') and external device (3) (e.g., external speaker), and at the same time, language service information corresponding to Korean can be transmitted to the first speaker's wearable device (1) for the speech of the other speakers. Therefore, smooth communication is possible even if the speakers use different languages.

[0287] Additionally, the processor (410) tracks the order and content of each speaker's speech, allowing for effective management of language data even in complex situations where multiple speakers speak simultaneously. For example, even if a second speaker and a third speaker speak in English and Chinese, respectively, almost simultaneously, the wearable device (1) can separate and translate the speech content individually and then clearly distinguish and output each speaker's speech on the display (420) in real time. This helps the user effectively grasp information without confusion in multi-speaker conversations.

[0288] According to various embodiments, the processor (410) can refer to language preference information stored in memory (430) to provide interpretation information optimized for the language desired by each user. For example, if a second speaker prefers additional translation into French rather than English, the processor (410) can generate and output additional French interpretation information based on these settings stored in memory (430). Accordingly, the present invention can be utilized more effectively as a user-customized multilingual service.

[0289] Through such embodiments, the present invention can provide high flexibility and convenience in various multilateral communication environments, particularly international conferences, internal meetings of multinational corporations, or workshops involving users of various language groups.

[0290] In the present disclosure, each of the phrases such as “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0291] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in other aspects (e.g., importance or order).

[0292] Terms such as ‘part’, ‘module’ as used in various embodiments of the present disclosure may include units implemented in hardware, software, or firmware. For example, they may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. As used in various embodiments of the present disclosure, ‘part’, ‘module’ may be implemented by various programs that are stored in an addressable storage medium and can be executed by a processor.

[0293] Various embodiments of the present disclosure may be implemented as software (e.g., a program) comprising one or more instructions stored in memory (430) (e.g., internal memory or external memory) that can be read by a device (e.g., a wearable device (1)). The memory (430) may be represented as a storage medium.

[0294] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., CD-ROM (compact disc read-only memory)), or distributed online (e.g., download or upload) through an application store or directly between two user devices.

[0295] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Additionally or substantially, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to the integration.

[0296] According to various embodiments, operations performed by a module, program or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0297] Methods according to the claims or embodiments described in the specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0298] When implemented in software, a computer-readable storage medium may be provided for storing one or more programs (software modules). One or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. One or more programs include instructions that cause the electronic device to execute methods according to the claims of the present disclosure or the embodiments described in the specification.

Claims

1. In the case of wearable devices, A pair of earcups configured to output sound; A rotating frame coupled to the above ear cup so as to enable shape change to a first state or a second state; At least one sensor that acquires sensing data including at least one of state data, image data, or acoustic data of a rotational frame; A display coupled to the above-mentioned rotating frame; At least one processor; and It includes memory for storing instructions, The above instructions are executed individually or collectively by the at least one processor so that the wearable device: Based on processing the sensing data obtained through the above-mentioned at least one sensor, language service information including language conversion information for the sensing data is obtained, and A wearable device that determines at least one of the earcup or the display based on the state of the above-mentioned rotating frame to output the language service information.

2. In Paragraph 1, The above instructions are executed individually or collectively by the at least one processor so that the wearable device: Acquiring a user input signal regarding the start of a language service through the above at least one sensor, and A wearable device that acquires the language service information based on the above user input signal.

3. In Paragraph 1, The first state above includes a fixed state in which the display is positioned outside the user's field of view, and The above second state is a wearable device comprising a state in which the display is fixed in a form corresponding to the user's field of vision.

4. In Paragraph 3, The above-mentioned wearable device further includes an external speaker, and The above instructions are executed individually or collectively by the at least one processor so that the wearable device: Based on the processing of the above sensing data, distinguish the speaker, and In response to the fact that the above speaker is the self, if the above rotation frame is in a second state, Language service information including language conversion information corresponding to a first language is output to the display, and A wearable device that outputs language service information, including language conversion information corresponding to a second language different from the first language, to the external speaker.

5. In Paragraph 3, The above-mentioned wearable device further includes an external speaker, and The above instructions are executed individually or collectively by the at least one processor so that the wearable device: Based on the processing of the above sensing data, distinguish the speaker, and In response to the fact that the above speaker is the self, if the above rotation frame is in the first state, A wearable device that outputs language service information, including preset language information and language conversion information corresponding to another language, to the external speaker.

6. In Paragraph 3, The above instructions are executed individually or collectively by the at least one processor so that the wearable device: Based on the processing of the above sensing data, distinguish the speaker, and In response to the fact that the above speaker is not the person in question, if the above rotation frame is in the first state, A wearable device that outputs language service information, including language conversion information corresponding to preset language information, to the earcup.

7. In Paragraph 3, The above instructions are executed individually or collectively by the at least one processor so that the wearable device: Based on the processing of the above sensing data, distinguish the speaker, and In response to the fact that the above speaker is the self, if the above rotation frame is in a second state, A wearable device that simultaneously outputs language service information, including language conversion information corresponding to a first language, to the display and the earcup.

8. In Paragraph 1, The above memory stores preset language information, and The above instructions are executed individually or collectively by the at least one processor so that the wearable device: A wearable device for obtaining language service information including language conversion information corresponding to the language information.

9. Regarding the method of outputting language service information, A wearable device comprising: a pair of earcups configured to output sound; a rotating frame coupled to the earcups so as to be shape-changeable to a first state or a second state; at least one sensor acquiring sensing data including at least one of state data of the rotating frame, image data, or acoustic data; a display coupled to the rotating frame; at least one processor; and a memory storing instructions, wherein (a) acquiring sensing data including at least one of state data, image data, or sound data of the above-mentioned rotational frame; (b) a step of processing the above sensing data to obtain language service information including language conversion information; and (c) a step of determining at least one of the ear cup or the display based on the state of the above-mentioned rotating frame and outputting the language service information, method.

10. In Paragraph 9, The step of processing the above sensing data is, A method further comprising a step of triggering the start of a language service through a user's input signal.

11. In Paragraph 9, The above-mentioned wearable device further includes an external speaker, and A step of distinguishing the speaker based on the processing of the above sensing data; and In response to the fact that the above speaker is the self, if the above rotation frame is in the first state, A method for outputting language service information, including preset language information and language conversion information corresponding to another language, to the external speaker.

12. In Paragraph 9, The above-mentioned wearable device further includes an external speaker, and A step of distinguishing the speaker based on the processing of the above sensing data; and A method comprising the step of: responding that the above speaker is the person in question, and when the above turning frame is in a second state, outputting language service information including language conversion information corresponding to a first language to the display, and outputting language service information including language conversion information corresponding to a second language different from the first language to the external speaker.

13. In Paragraph 9, A step of distinguishing the speaker based on the processing of the above sensing data; and A method comprising the step of outputting language service information, including language conversion information corresponding to preset language information, to the earcup when the above speaker responds that the above speaker is not the person in the first state, and the above turning frame is in the first state.

14. In Paragraph 9, The first state above includes a state in which the display is positioned outside the user's field of vision, and The above second state includes a state in which the display corresponds to the user's field of vision.

15. A computer-readable recording medium storing one or more programs comprising instructions for performing the method of any one of claims 9 to 14.

Images