Wearable device for implementing binaural hearing and binocular stereopsis
The wearable device integrates wireless earphones, cameras, and microphones with an analysis device to provide three-dimensional spatial recognition, addressing the limitations of conventional devices by utilizing binocular vision and binaural hearing for enhanced environmental perception.
Patent Information
- Application Number
- PCT/KR2025/014548
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-16
- Filing Date
- 2025-09-18
- Publication Date
- 2026-11-19
AI Technical Summary
Conventional wearable devices with cameras and bone conduction earphones do not effectively implement human stereoscopic vision and binaural hearing, failing to consider the visual and auditory structural differences between the human head and the device placement, and lack technology for perceiving objects in three dimensions.
A wearable device equipped with wireless earphones, cameras, microphones, and an analysis device that processes image and sound data to recognize objects in three dimensions using binocular vision and binaural hearing, incorporating extensions to mitigate blind spots and employing machine learning models for object detection and inference.
Enables real-time recognition and feedback of three-dimensional spatial information, assisting users in navigating their environment by providing additional information through augmented reality and aiding individuals with disabilities.
Smart Images

Figure KR2025014548_19112026_PF_FP_ABST
Abstract
Description
Wearable device for realizing bilateral hearing and stereoscopic vision
[0001] The technology described below is for a wearable device for implementing biaural hearing and binocular stereopsis.
[0002] With the recent rapid advancement of deep learning technology, technological development in the field of artificial intelligence (AI) is actively underway. This AI technology is evolving beyond simple computational capabilities to mimic human cognitive functions. For example, various application technologies are being developed, such as devices that automatically open and close doors in specific situations by using AI to analyze video data and automatically identify objects appearing in the video.
[0003] Such advanced artificial intelligence technology is gradually permeating not only the entire industry but also our daily lives. In particular, recently, it has become possible to provide personalized services by analyzing various biosignals and environmental data collected from portable devices such as smartphones and smart tablets, as well as wearable devices such as wireless earphones, smartwatches, and smart glasses, using AI technology. Through this, innovation is taking place in various areas of daily life, including health management, behavior monitoring, and personalized feedback.
[0004] One of the primary goals of artificial intelligence technology is to enable humans to interact with AI systems more naturally and intuitively. To this end, active research is underway to understand human cognitive and sensory systems and to implement them in a similar manner.
[0005] Humans perceive the surrounding world in three dimensions through stereoscopic vision (binocular vision). Binocular vision is a function that uses both eyes simultaneously to receive visual information from different angles, which the brain then integrates to form a single three-dimensional visual image. This visual mechanism plays a crucial role in enabling humans to accurately perceive spatial depth, distance, and the relative positions of objects, as well as the direction and speed of their dynamic movements. Additionally, humans can perceive the direction and distance of sound through binaural hearing. Binaural hearing is a function that determines the location of a sound source based on the time difference and intensity difference between sounds entering each ear, and it is essential for understanding the surrounding environment in three dimensions. Therefore, for artificial intelligence to perceive the world in a manner similar to humans, it needs to possess not only the ability to recognize three-dimensional objects based on visual information from various angles, but also the ability to perceive spatial acoustics based on binaural hearing.
[0006] Conventionally, there has been technology for connecting a camera to a wearable device. For example, Korean Registered Patent Publication No. 10-2382985 discloses a configuration in which a camera for photographing the front or rear of a user is connected to one side of a bone conduction earphone.
[0007] Human eyes are located at the front of the head. However, a camera connected to an earphone is located on the side of the head. This is because the earphone is positioned at the ear. While human eyes are located at the front of the head, the camera connected to the earphone is located on the side of the head, that is, at the ear. Conventional technology discloses only a camera attached to a bone conduction earphone without considering these visual structural differences. Furthermore, conventional bone conduction earphones merely capture images and do not include technology for the process of a person perceiving objects placed in space in three dimensions.
[0008] The technology described below aims to disclose a wearable device for solving these problems.
[0009] The technology described below aims to disclose a wearable device equipped with a camera.
[0010] In one embodiment, the wearable device includes a wireless earphone, a camera, a microphone, and an analysis device. The wireless earphone is attached to each of the user's ears to play sound, the camera is connected to each of the wireless earphones attached to the user's ears to acquire image data through shooting, the microphone is connected to each of the wireless earphones attached to the user's ears to collect sounds around the user to acquire sound data, and the analysis device can recognize objects around the user based on the image data and the sound data.
[0011] In one embodiment, the wearable device further includes an extension, and the extension connects the wireless earphones and the camera so that the camera can be moved away from both ears of the user to whom the wireless earphones are attached by a preset distance.
[0012] In one embodiment, an object recognition method using a wearable device comprising a wireless earphone, a camera, a microphone, and an analysis device comprises: a step in which the analysis device acquires image data collected by the camera and sound data collected by the microphone; a step in which the analysis device detects an object in the image data; a step in which the analysis device calculates an image disparity of each image data based on the object detection result; a step in which the analysis device detects an object sound in the sound data; a step in which the analysis device calculates a difference in each sound data based on the result of detecting the object sound; and a step in which the analysis device recognizes an object based on the calculated image disparity of each image data, the difference in each sound data, the position information of the camera, the angle information of the camera, and the position information of the microphone.
[0013] In one embodiment, the object recognition method using a wearable device comprises: a step in which the analysis device acquires an image based on time-series data captured by the camera; a step in which the analysis device generates an inferred image from the image using an inferred model; and a step in which the analysis device performs the object recognition method using the wearable device based on the inferred image and sound data to recognize an object located in a blind spot.
[0014] By utilizing the technology described below, a wearable device that implements human stereoscopic vision can be used. By utilizing the technology described below, the positional relationship of a sound-emitting object can be recognized using binaural hearing. By utilizing the technology described below, it is possible to assist in rapidly providing feedback on three-dimensional spatial information (front, back, left, right, perspective, speed and direction of dynamic movement, etc.) of a user or object at the current location. By utilizing the stereoscopic vision or binaural hearing technology described below, additional information can be generated using artificial intelligence, etc., based on the spatial information of the object obtained through this method, and provided to the user in real time.
[0015] FIGS. 1 to 3 show one embodiment of a wearable device. FIG. 1 shows a wearable device implemented in the form of an earbud (100). FIG. 2 shows a wearable device implemented in the form of a bone conduction device (200). FIG. 3 shows a wearable device implemented in the form of headphones (300).
[0016] FIG. 4 is one of the embodiments (400) in which an analysis device performs an object recognition method.
[0017] Figure 5 shows the difference in image data according to the position of the camera (500).
[0018] Figure 6 shows the difference in image data according to the movement of the object (600).
[0019] FIG. 7 is one of the other embodiments (700) in which an analysis device performs an object recognition method.
[0020] FIG. 8 is one of other embodiments (800) in which an analysis device performs an object recognition method.
[0021] FIGS. 9 to 13 show one embodiment of a wearable device further including extension parts (950a, 950b) made for the purpose of reducing blind spots in images obtained from left and right cameras. FIG. 9 is a perspective view (900) of a wearable device further including extension parts (950a, 950b). FIG. 10 is a top view of a wearable device further including extension parts (950a, 950b). FIG. 10 is a left side view of a wearable device further including extension parts (950a, 950b). FIG. 10 is a rear view of a wearable device further including extension parts (950a, 950b). FIG. 10 is a front view of a wearable device further including extension parts (950a, 950b). FIG. 10 is a front view of a wearable device further including extension parts (950a, 950b).
[0022] FIG. 14 shows the results of the camera's forward field of view (1400) depending on the presence or absence of an extension. FIG. 14(a) shows the case without an extension. FIG. 14(b) shows the case with an extension.
[0023] FIG. 15 shows one of the problems (1500) that cannot be solved even by employing an extension part.
[0024] FIG. 16 is one of the embodiments (1600) of an object recognition method for solving the problem of FIG. 15.
[0025] FIGS. 17 and FIGS. 18 are examples (1700, 1800) for illustrating the method of the embodiment of FIG. 10.
[0026] The technology described below may be subject to various modifications and may have various embodiments. Specific embodiments of the technology described below may be described in the drawings of the specification. However, this is for the purpose of explaining the technology described below and is not intended to limit the technology described below to specific embodiments. Accordingly, it should be understood that all modifications, equivalents, and substitutions that fall within the spirit and scope of the technology described below are included in the technology described below.
[0027] Terms such as first, second, A, B, etc., may be used to describe various components. However, the above terms are used merely to distinguish one component from other components and are not intended to limit the components to the said terms. For example, without departing from the scope of rights of the technology described below, the first component may be named the second component, and similarly, the second component may be named the first component. The term “and / or” includes a combination of multiple related described items or any of the multiple related described items.
[0028] In the terms used below, singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as "includes" should be understood to mean that the described features, number, steps, actions, components, parts, or combinations thereof exist, and not to exclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0029] Before providing a detailed description of the drawings, it is to clarify that the classification of components in this specification is merely based on the primary function each component is responsible for. That is, two or more components described below may be combined into a single component, or a single component may be divided into two or more components based on more subdivided functions. Furthermore, each component described below may additionally perform some or all of the functions of other components in addition to its own primary function, and it is obvious that some of the primary functions of each component may be exclusively performed by other components.
[0030] Furthermore, in performing the method or operation method, each process constituting the method may occur differently from the specified order unless a specific order is clearly indicated in the context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.
[0031] In the technology described below, a wearable device may refer to a device designed to be attached to or worn on a user's body (e.g., head, wrist, chest, etc.).
[0032] In the technology described below, a wearable device may refer to a device having at least one assembly unit formed by combining hardware such as wireless earphones, a camera, and a microphone.
[0033] In the technology described below, the wearable device can operate continuously even while the wearer is moving and may include sensors, cameras, communication modules, etc.
[0034] In the technology described below, wearable devices may include devices such as earphones, headphones, smartphone smart glasses, smart watches, and chest-mounted body cams.
[0035] In the technology described below, the wearable device may include devices such as a communication device and a user interface.
[0036] In the technology described below, wearable devices can be utilized by various people and are not limited to specific uses. For example, in the technology described below, wearable devices can be widely used not only by people without disabilities but also by people with disabilities.
[0037] In one embodiment, the wearable device may include a wireless earphone, a camera, a microphone, and an analysis device. In one embodiment, the wearable device may further include a fixed part.
[0038] FIGS. 1 to 3 illustrate one embodiment of a wearable device. FIG. 1 shows a wearable device implemented in the form of an earbud (100). FIG. 2 shows a wearable device implemented in the form of a bone conduction device (200). FIG. 3 shows a wearable device implemented in the form of headphones (300). The following description will focus on FIG. 2.
[0039] Wireless earphones (210a, 210b) can be attached to each of a person's two ears to play sound.
[0040] The wireless earphones (210a, 210b) may be earphones without wires. The wireless earphones (210a, 210b) may include a device that connects to another device without a cable to play sound (e.g., music). The wireless earphones (210a, 210b) may receive data for playing sound, etc. via wireless communication. For example, the wireless earphones (210a, 210b) may receive data for playing sound, etc. via a method such as Bluetooth.
[0041] Wireless earphones (210a, 210b) can have various forms. In one embodiment, wireless earphones (210a, 210b) can have various forms such as earbuds, headsets, neckbands, bone conduction earphones, and clip-on earphones.
[0042] Wireless earphones (210a, 210b) can be attached to each person's ear.
[0043] The camera (220a, 220b) may be a device for capturing the surroundings. Image data can be acquired through the camera (220a, 220b).
[0044] The camera (220a, 220b) may include various types of cameras. For example, the camera (220a, 220b) may include an ultraviolet camera, an infrared camera, a visible light camera, a thermal imaging camera, a multispectral camera, a high-speed camera, a zoom camera, etc.
[0045] Cameras (220a, 220b) can be positioned close to each of the wireless earphones (210a, 210b) to enable binocular vision. Since each of the multiple cameras (220a, 220b) is located at a different position, each of the multiple cameras (220a, 220b) can acquire different image data. That is, even if the image data captures the same object, there is a difference between the camera's shooting angles, so image disparity exists, and stereoscopic information can be obtained by utilizing this.
[0046] The camera (220a, 220b) can be connected to wireless earphones (210a, 210b) attached to each of the person's ears.
[0047] The cameras (220a, 220b) can capture images of the surroundings of a person (e.g., front or / and rear). The cameras (220a, 220b) can be designed to have a field of view of the user's front, rear, and side. Since both cameras (220a, 220b) are operated simultaneously, functions similar to human eyes, such as detecting distance, detecting subtle movements of the user, and acquiring images focused on the direction from which sound is heard, can be implemented.
[0048] Each camera (220a, 220b) can look in a different direction. This is because the angle at which the wireless earphones (210a, 210b) are worn can vary from ear to ear.
[0049] The angles at which each camera (220a, 220b) looks can be fixed differently from each other. The fixing part can fix the angles at which each camera (220a, 220b) looks.
[0050] The microphone (230a, 230b) may be a device for acquiring ambient sound. Sound data can be acquired through the microphone (230a, 230b).
[0051] Microphones (230a, 230b) may be provided for each of the wireless earphones (210a, 210b) so that the concept of bilateral hearing can be applied. Since each of the multiple microphones (230a, 230b) is located at a different position, each of the multiple microphones (230a, 230b) can acquire different data from each other.
[0052] Microphones (230a, 230b) may be equipped with wireless earphones (210a, 210b) attached to each of a person's ears. Since microphones (230a, 230b) are provided for each of the wireless earphones (210a, 210b), each of the multiple microphones (230a, 230b) may be located at a different position. Similar to cameras (220a, 220b), the position of each of the multiple microphones (230a, 230b) may vary depending on the shape of the person's head.
[0053] The microphone (230a, 230b) can acquire human voice data. The human voice data may include commands that the person intends to transmit to the wearable device via voice. For example, the person may transmit a pre-set detection rule to the analysis device (240) via voice.
[0054] The analysis device (240) may be a device for data processing. The analysis device (240) may be a device for data processing to recognize objects around the user.
[0055] The analysis device (240) may be a device provided inside a wearable device. For example, the analysis device (240) may include a device such as a chip or processor existing inside the wearable device. Alternatively, the analysis device (240) may be a separate device separated from the wearable device that exchanges data via wired or wireless communication. For example, the analysis device (240) may be a device such as a computer, smartphone, portable tablet, portable laptop, and server.
[0056] The analysis device (240) can acquire and process various data. The analysis device (240) can provide static or dynamic stereoscopic information obtained by acquiring and analyzing image data and sound data. Based on the stereoscopic information, it can be processed into additional information using artificial intelligence, etc., and provided to the user.
[0057] The analysis device (240) can recognize objects around the user based on acquired data. The analysis device (240) can recognize objects around the user based on image data. The analysis device (240) can recognize objects around the user by further utilizing sound data along with image data. The analysis device (240) can perform tasks according to the user's commands.
[0058] Recognizing objects around a user may include recognizing objects around a user in a three-dimensional manner as objects in three-dimensional space. Recognizing objects around a user may include identifying static stereoscopic information regarding the objects' front, back, up, down, left, and right. Recognizing objects around a user may include identifying dynamic stereoscopic information regarding the objects. For example, recognizing objects around a user may include identifying not only the relative position of objects around a user but also the direction and velocity of the objects.
[0059] Recognizing objects around a user may include recognizing the type and location of objects around a user wearing a wearable device. For example, recognizing objects around a user may include recognizing that an object around a user wearing a wearable device is an “apple” and recognizing where the location of the “apple” is located in three dimensions relative to the user. Specific details will be described later.
[0060] The analysis device (240) can transmit the analysis results to wireless earphones (210a, 210b). The wireless earphones (210a, 210b) can notify the user of the received results through sound. The analysis device (240) can also notify the user of the analysis results in the form of a video.
[0061] The fixing part (250) serves to fix the camera (220a, 220b) so that it can have a predetermined angle. The fixing part (250) is connected to wireless earphones (210a, 210b) attached to both ears, and by fixing the angle of the wireless earphones (210a, 210b), the camera (220a, 220b) connected to the wireless earphones (210a, 210b) can have a predetermined angle. In one embodiment, as shown in FIG. 2, the fixing part (250) may have a neckband shape that allows the wireless earphones (210a, 210b) to be worn around the neck for use. Alternatively, as shown in FIG. 3, the fixing part (350) may have a headphone shape that allows the wireless earphones (310a, 320b) to be worn on the head for use.
[0062] FIG. 4 is one of the embodiments (400) in which an analysis device performs an object recognition method.
[0063] The analysis device can acquire image data and sound data (410).
[0064] Video data and sound data may be data acquired from a camera and microphone included in a wearable device.
[0065] Video data may include data acquired from different locations. Video data may include data acquired by shooting from different angles. This is because, as previously mentioned, the camera is connected to wireless earphones attached to each of a person's ears. Therefore, video data may include data acquired from the positions of a person's left and right ears. The same applies to sound data.
[0066] Video data and sound data may be time-series data. Video data and sound data may be time-series data acquired over a preset period of time.
[0067] The analysis device can detect objects in the image data (420).
[0068] An image object detection model can be used to detect objects in an image. An image object detection model may be a model that detects objects appearing in an image. An image object detection model may be a model trained to detect objects in an image based on training data. An image object detection model may be a machine learning (ML)-based model. An image object detection model may be an artificial neural network (ANN)-based model. For example, an image object detection model may be a convolutional neural network (CNN)-based model.
[0069] The analysis device can detect objects in an image according to detection rules pre-set by the user. The detection rules pre-set by the user may include a determination of which object to detect among the various objects appearing in the image. For example, the detection rules pre-set by the user may include an instruction to detect an "apple" appearing in the image. Alternatively, the rules pre-set by the user may include an instruction to detect a "friend" appearing in the image.
[0070] The analysis device can calculate the image disparity of each image data based on the object detection results (430)
[0071] Calculating the deviation of image data may include calculating the difference between objects. The difference between objects may include the difference in the position of the object represented in the image data, the size of the object represented in the image, and the degree of rotation of the object represented in the image.
[0072] As mentioned above, cameras may be located at different positions. Consequently, when each camera captures a single object, the appearance of the object in the image may differ. For example, a case like that shown in Fig. 5 may occur. As can be seen in Fig. 5, there is a difference in the relative positions of the apple captured by both cameras. In particular, this difference may become larger as the object is closer to the person, and conversely, smaller as the object is farther away from the person. Furthermore, this difference may increase as the distance between the cameras increases, and conversely, decrease as the distance between the cameras decreases. In other words, the difference in image data captured at the positions of the two ears may be related to the position of the object and the position of the cameras.
[0073] Furthermore, the size of an object represented in the image data may vary depending on the distance between the person and the object. Additionally, the image data may be time-series data. When a single object is moving, its appearance in the image may differ depending on its position. For example, a case like that shown in Fig. 6 may occur. As can be seen in Fig. 6, even when an object moves at a constant speed, the appearance of the object represented in the time-series data may differ depending on the distance between the object and the person. In other words, the difference in the object represented in the time-series image data may be related to the object's position.
[0074] The analysis device can detect object sounds from sound data (440).
[0075] It can include sounds originating from the object to be recognized.
[0076] A sound object detection model can be used to detect sounds originating from objects. The sound object detection model may be a model that detects data originating from objects among the sound data. For example, the sound object detection model may be a model that detects data regarding the sound of an apple falling among noise, the sound of a car starting, and the sound of an apple falling included in the sound data. Similar to the aforementioned image object detection model, the sound object detection model may be a machine learning (ML)-based model.
[0077] To detect sound emanating from an object, the result of detecting an object in image data can be utilized. Detecting object sound may include the operation of detecting sound emanating from an object based on the result of detecting an object in the image data. That is, the analysis device first detects an object named "apple" in the image data, and then uses the information that the detected object is an "apple" to detect the object sound of "apple" in the sound data.
[0078] The analysis device can calculate the difference between each sound data based on the object sound detection results. (450)
[0079] Calculating the difference between each sound data may include calculating the difference between object sounds. As previously mentioned, microphones may be located at different positions. The sound data may be collected from different locations on the left and right. Therefore, the data of the object sounds represented in each sound data may differ. For example, a case like that shown in Fig. 5 may occur. As can be seen in Fig. 5, the time at which the object sounds were acquired in the sound data obtained from both microphones may differ. Specifically, since the apple is located to the right relative to the person, the sound produced when the apple falls on the table may reach the microphone attached to the person's right ear first and be input as a sound of greater intensity. Based on this, the analysis device can identify the spatial positional relationship—static or dynamic—of the sound-producing object even in the camera's blind spot.
[0080] The difference between object sounds represented in the sound data may include differences in the time and intensity of the sounds from the objects reaching the microphone.
[0081] The analysis device can recognize objects based on the deviation of each image data, the difference of each sound data, the position information of the camera, the angle information of the camera, and the position information of the microphone (460).
[0082] The object recognition result may include the result of recognizing objects around the user in three-dimensional space. The object recognition result may include the result of recognizing the three-dimensional location of objects around the user. The object recognition result may include the result of recognizing the direction of movement of objects around the user in three dimensions. The object recognition result may include the result of identifying the type of objects around the user. The object recognition result may include the result regarding the time at which objects around the user were recognized. The object recognition result may be stored in a database.
[0083] Furthermore, the analysis device can output object recognition results to the user. For example, the analysis device can output object recognition results to the user as voice through wireless earphones. Alternatively, the analysis device can output object recognition results to the user as video through the user's display.
[0084] Object recognition results can be utilized in various tasks. For example, object recognition results can be analyzed through an AI model for the visually impaired to guide them on what objects are in their surroundings or what objects they need to be careful of. Alternatively, object recognition results can be analyzed through an AI model for safety to guide workers on what dangerous objects are in their vicinity and to advise them to avoid them. Furthermore, object recognition results can be used to create augmented reality (AR) content, providing users with real-time interactive content by overlaying virtual information onto the locations of objects. Finally, object recognition results can be provided to surgeons wearing wearable devices to assist them during surgery.
[0085] FIG. 7 is one of the other embodiments (700) in which an analysis device performs an object recognition method.
[0086] As shown in FIG. 7, three objects (an apple, a shoe, and a camera) may be positioned in the direction in front of a person. As previously described, the analysis device can recognize the three objects through an object recognition method. That is, based on image data and sound data, the analysis device can output an object recognition result that includes information that the types of objects located in front of a person's eyes are an apple, a shoe, and a camera, as well as information regarding the three-dimensional positions of each object. The object recognition result can be stored and converted into a database.
[0087] As shown in FIG. 7, a microphone included in a wearable device can acquire human voice data. The human voice data may include commands that the person wishes to convey to the wearable device. That is, in order to know information about an object visible to the left front, the person may transmit voice data such as “Can you tell me the information about the object visible at the very front on the left?” to the wearable device through the microphone. However, this is merely one example, and the user may send various voice commands to the analysis device through the microphone of the wearable device.
[0088] As shown in FIG. 7, the analysis device analyzes image and sound information obtained through multiple cameras and microphones. Based on the stereoscopic information and binaural information of the object obtained through this process, it can perform tasks based on voice commands. First, the analysis device can analyze what the user's command is based on the voice data. To analyze what the user's command is, the analysis device may use a Natural Language Processing Model. For example, the analysis device may generate a command vector by outputting only the features from the user's command. Alternatively, the analysis device may classify the user's command into one of the pre-set classification results.
[0089] As shown in FIG. 7, the analysis device can output an analysis result based on the result of analyzing what the user's command is and the result of object recognition. The analysis device can also generate an output voice (“The object visible on the left is a patterned apple”) based on the analysis result. The output voice generated by the analysis device can be transmitted to a wireless earphone included in a wearable device. The wireless earphone can play sound according to the received output voice. The analysis result can be stored and converted into a database.
[0090] FIG. 8 is one of other embodiments (800) in which an analysis device performs an object recognition method.
[0091] As shown in FIG. 8, an object (apple) may be positioned in the direction in front of a person. As previously mentioned, the analysis device can recognize the apple through an object recognition method. The analysis device can also receive voice data containing the command, “I think I saw the apple on the right last time?” The analysis device can receive the object recognition result and data stored in the database. The analysis device can generate an analysis result based on the voice data, the object recognition result, and the data stored in the database. Based on the analysis result, the analysis device can generate an output voice saying, “The apple on the right is the apple seen at 19:00 on the 25th.” In other words, the data stored in the database can be utilized through this process.
[0092] In one embodiment, the wearable device may further include extension portions (950a, 950b).
[0093] FIGS. 9 to 13 show one embodiment of a wearable device further including extension portions (950a, 950b). FIG. 9 is a perspective view (900) of a wearable device further including extension portions (950a, 950b). FIG. 10 is a top view of a wearable device further including extension portions (950a, 950b). FIG. 10 is a left side view of a wearable device further including extension portions (950a, 950b). FIG. 10 is a rear view of a wearable device further including extension portions (950a, 950b). FIG. 10 is a front view of a wearable device further including extension portions (950a, 950b). FIG. 10 is a front view of a wearable device further including extension portions (950a, 950b).
[0094] The extension part (950a, 950b) can connect wireless earphones and a camera.
[0095] The extension part (950a, 950b) can cause the camera to be located at a distance greater than a preset distance from the ear of a person to whom the wireless earphone is attached. The extension part (950a, 950b) may exist to solve the problem of a portion of the camera's front view being obscured by a person's face when connecting the camera to the wireless earphone. The preset distance may have various distances. For example, the preset distance may have various ranges between 10 cm. Alternatively, the preset distance may be 0.5 cm to 1.5 cm, 1 cm to 3 cm, 1 cm to 5 cm, 2 cm to 3 cm, 2 cm to 6 cm, 3 cm to 7 cm, 5 cm to 10 cm, etc.
[0096] FIG. 14 shows the results of the camera's forward field of view (1400) depending on the presence or absence of an extension. FIG. 14(a) shows the case without an extension. FIG. 14(b) shows the case with an extension.
[0097] While a person's eyes are located at the front of the head, the camera equipped in the wireless earphone is located at the side of the head near the ear. Due to this positional characteristic, as shown in FIG. 14(a), a problem may occur where a portion of the image captured by the camera connected to the earphone is obscured by the head. On the other hand, as shown in FIG. 14(b), when the wireless earphone and the camera are connected via extensions (950a, 950b) and the camera is placed a certain distance away from the earphone, it can be seen that the relative field of view is widened. In this case, the problem of obscuration by the head can be resolved. Through this, the camera can be made to view objects as if with human binocular vision.
[0098] FIG. 15 shows one of the problems (1500) that cannot be solved even by employing an extension part.
[0099] Even with an extension, problems such as blind spots may arise. For example, consider Fig. 15. Fig. 15 depicts a case where an apple is placed directly in front of a person's head. As shown in Fig. 15, despite the presence of the extension, the camera cannot capture the apple located directly in front of the person's head. One method to solve this problem is to extend the length of the extension. However, extending the length of the extension may cause inconvenience in actual daily use. Furthermore, since wearable devices equipped with cameras are treated as accessories and utilized as a form of fashion, the length of the extension cannot be extended indefinitely due to design considerations. Therefore, a method is required to detect objects located in blind spots that may occur despite the presence of the extension.
[0100] FIG. 16 is one embodiment (1600) of an object recognition method for solving the problem of FIG. 15. FIG. 17 and FIG. 18 are examples (1700, 1800) for showing the method of the embodiment of FIG. 10.
[0101] The analysis device can acquire image data captured by the camera (1610).
[0102] The video captured by the camera may be time-series data. For example, as shown in FIG. 17, if it takes 4 seconds for an apple to reach a person's eyes, the video captured by the camera may be a video composed of 4 time-series data at t=-3, t=-2, t=-1, and t=0.
[0103] The analysis device can generate an inferred image from an image captured by a camera using an inferred model (1620).
[0104] The inference model may be a model that generates an inferred image based on video captured by a camera. The inference model may be an image representing an object placed in a blind spot.
[0105] It is unlikely that a non-existent object will suddenly appear in a blind spot. Objects in a blind spot are more likely to have been within the camera's field of view before moving into the blind spot. Considering this, the inference model can generate an inference image by inferring the objects present in the blind spot.
[0106] The inference model may be a model that processes time-series data. In one embodiment, the inference model may be a model that generates an inferred image based on the sequence of each image and the differences between images according to the sequence in a video captured by a camera composed of time-series data. Alternatively, the inference model may be a model that generates an inferred image based on the results obtained after determining the speed of an object based on the time difference interval of the video captured by the camera composed of time-series data and the size of the object expressed in the video. If the location of an object at a specific point in time and the speed of the object at a specific point in time can be determined, the location of the object in a blind spot can be determined.
[0107] Like the aforementioned detection model, the inference model can be a learning model, a machine learning model, or an artificial neural network model.
[0108] The analysis device can recognize objects located in blind spots based on the inference image (1630)
[0109] The analysis device can determine the 3D location of an object by performing the object recognition method shown in Fig. 4 based on the inference image. Alternatively, the analysis device can determine the type of object located in a blind spot based on the inference image.
[0110] The embodiments and drawings attached to this specification merely clearly illustrate a part of the technical ideas included in the aforementioned technology, and it is self-evident that variations and specific embodiments that can be easily inferred by a person skilled in the art within the scope of the technical ideas included in the specification and drawings of the aforementioned technology are all included within the scope of the rights of the aforementioned technology.
Claims
1. A wearable device comprising wireless earphones, a camera, a microphone, and an analysis device, The above wireless earphones are attached to each of the user's ears to play sound, and The above camera is connected to each of the wireless earphones attached to both ears of the user, and acquires video data through shooting, and The above microphone is connected to each of the wireless earphones attached to both ears of the user, collects sounds around the user to acquire sound data, and The above analysis device recognizes objects around the user based on the above image data and the above sound data, Wearable device.
2. In Paragraph 1, The above-mentioned wearable device further includes a fixing part, and The above fixing part is connected to wireless earphones attached to both ears and fixes the angle of the wireless earphones, thereby enabling the camera connected to the wireless earphones to have a preset angle. Wearable device.
3. In Paragraph 1, The above-mentioned wearable device further includes an extension part, and The above extension unit connects the wireless earphones and the camera, so that the camera is positioned at a distance greater than a preset distance from both ears of the user to whom the wireless earphones are attached. Wearable device.
4. In Paragraph 1, The above analysis device recognizes objects around the user through the following operations (a) to (f). Wearable device. (a) The operation of the analysis device acquiring image data collected by the camera and sound data collected by the microphone; (b) The operation of the analysis device detecting an object in the image data; (c) The operation of the analysis device to calculate the image disparity of each image data based on the object detection result; (d) The operation of the analysis device detecting object sounds from the sound data; (e) an operation of calculating the difference between each sound data based on the result of the analysis device detecting the object sound; (f) The operation of the analysis device to recognize an object based on the deviation of each calculated image data, the difference of each sound data, the position information of the camera, the angle information of the camera, and the position information of the microphone; 5. In Paragraph 4, The operation of detecting objects in the above image data includes using a detection model, and The above detection model is a model trained to detect objects in an image based on training data, Wearable device.
6. In Paragraph 4, The operation of detecting objects in the above image data includes detecting objects according to a pre-set detection rule, and The above detection rule includes a rule that the user predetermines which object to detect among the objects represented in the image data. Wearable device.
7. In Paragraph 4, The deviation of each of the above image data includes differences in the position of the object represented in the image data, the size of the object represented in the image, and the degree of rotation of the object represented in the image. Wearable device.
8. In Paragraph 4, The operation of detecting the above object sound includes using a sound object detection model, and The above sound object detection model is a model trained to detect object sounds, which are sounds originating from objects, in sound data based on training data, Wearable device.
9. In Paragraph 4, The operation of detecting the object sound includes the operation of detecting the sound from the detected object in the sound data according to the result of detecting the object in the image data. Wearable device.
10. In Paragraph 4, The difference in the above sound data includes the difference in the time and intensity of the sound emitted from the object reaching the microphone. Wearable device.
11. In Paragraph 4, The above analysis device recognizes an object located in a blind spot through the following operations (g) to (i). Wearable device. (g) The operation of the analysis device acquiring image data based on time-series data captured by the camera; (h) The operation of the analysis device generating an inferred image from the image data using an inferred model; and (i) The analysis device performs the operations (a) to (f) based on the inferred image and the sound data to recognize an object located in a blind spot; 12. In Paragraph 11, The above inference model is a model for processing time-series data, and is a model that generates an inferred image based on the order of each image data based on the time-series data and the difference between the image data according to the order. Wearable device.
13. In Paragraph 11, The above inference model is a model for processing time-series data, wherein the model determines the speed of an object based on the time difference of each image data based on the time-series data and the size of the object expressed in the image data, and then generates an inference image based on the determined result. Wearable device.
14. An object recognition method using a wearable device comprising a wireless earphone, a camera, a microphone, and an analysis device as described in claim 1, wherein The above analysis device acquires image data collected by the camera and sound data collected by the microphone; A step in which the analysis device detects an object in the image data; A step in which the analysis device calculates the image disparity of each image data based on the object detection result; A step in which the analysis device detects object sounds from the sound data; A step of calculating the difference between each sound data based on the result of the analysis device detecting the object sound; and The above analysis device recognizes an object based on the deviation of each calculated image data, the difference of each sound data, the position information of a camera, the angle information of a camera, and the position information of a microphone; comprising Object recognition method using a wearable device.
15. In Paragraph 14, The above-mentioned wearable device further includes an extension part, and The above extension unit connects the wireless earphones and the camera, so that the camera is positioned at a distance greater than a preset distance from both ears of the user to whom the wireless earphones are attached. Object recognition method using a wearable device.
16. In Paragraph 14, The above analysis device acquires image data based on time-series data captured by the camera; The step of the analysis device generating an inferred image from the image data using an inference model; and The above analysis device performs an object recognition method using the wearable device based on the inferred image and sound data to recognize an object located in a blind spot; comprising the step of Object recognition method using a wearable device.
