Electronic device for generating sound and operation method thereof

Abstract

The present disclosure provides a method of a HMD device. The method of a HMD device may comprise the operations of: obtaining structure information of a space in which a user of the HMD device is located, on the basis of image data acquired through a camera of the HMD device; obtaining material information of at least one object located in the space, on the basis of the image data and by using a trained first model; and generating sound on the basis of the structure information, the material information, and speaker arrangement setting information. The material information may include sound characteristics information corresponding to a material of the corresponding object.

Description

Electronic device that generates sound and method of operation thereof

[0001] The present disclosure relates to an electronic device for generating sound and a method of operation.

[0002] Virtual reality (VR) and augmented reality (AR) systems utilize various technologies to enable users to experience immersion. In particular, it is important to provide visual elements through a head-mounted display (HMD) and apply sound effects within the space to make users experience being in a real space.

[0003] Surround sound technology typically outputs sound through a fixed array of speakers or provides a sense of spatiality through simple left-right stereo sound. However, for users wearing an HMD, their position and orientation change in real time; accordingly, the direction and distance of the sound source must also be dynamically adjusted to match the user's location. This allows users to perceive the location of the sound source more accurately, thereby further enhancing immersion and realism.

[0004] Therefore, technology is required to provide optimized surround sound by considering the real-time location of the user wearing the HMD. To enhance the user experience, this technology enables customized sound output that reflects the relative position and direction of the user and the sound source.

[0005] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0006] According to one embodiment, an HMD device may be provided. The HMD device may include a camera, at least one processor including a processing circuit, and a memory including at least one storage medium for storing instructions. When the instructions are executed individually or collectively by the at least one processor, they may cause the HMD device to perform at least one operation. The at least one operation may include an operation of obtaining structural information about the space where the user of the HMD device is located based on image data obtained through the camera of the HMD device. The at least one operation may include an operation of obtaining material information about at least one object located in the space based on the image data using a learned first model. The at least one operation may include an operation of generating sound based on the structural information, the material information, and virtual speaker placement setting information. The material information may include sound characteristic information corresponding to the material of the object.

[0007] According to one embodiment, a method of operation of an HMD device may be provided. The method of operation of the HMD device may include at least one operation. The at least one operation may include an operation of acquiring structural information about a space where a user of the HMD device is located based on image data acquired through a camera of the HMD device. The at least one operation may include an operation of acquiring material information about at least one object located in the space based on the image data using a learned first model. The at least one operation may include an operation of generating sound based on the structural information, the material information, and virtual speaker placement setting information. The material information may include sound characteristic information corresponding to the material of the object.

[0008] According to one embodiment, a storage medium may be provided for storing at least one instruction readable by a computer. When executed by at least a part of at least one processor of the HMD device, the at least one instruction may cause the HMD device to perform at least one operation. The at least one operation may include an operation of obtaining structural information about the space where the user of the HMD device is located based on image data obtained through the camera of the HMD device. The at least one operation may include an operation of obtaining material information about at least one object located in the space based on the image data using a learned first model. The at least one operation may include an operation of generating sound based on the structural information, the material information, and virtual speaker placement setting information. The material information may include sound characteristic information corresponding to the material of the object.

[0009] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0010] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0011] FIG. 2 is a drawing showing the configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0012] FIGS. 3a to 3c are drawings showing the front and rear views of a wearable electronic device according to one embodiment of the present disclosure.

[0013] FIGS. 4a and 4b are drawings illustrating an operation in which an HMD device, according to one embodiment of the present disclosure, generates sound to be provided to a user wearing the HMD device using a first sound generation method.

[0014] FIG. 5 is a drawing illustrating the configuration of an HMD device according to one embodiment of the present disclosure.

[0015] FIG. 6 is a flowchart illustrating the operation of an HMD device generating sound according to one embodiment of the present disclosure.

[0016] FIG. 7 is a flowchart illustrating the operation of an HMD device outputting sound according to one embodiment of the present disclosure.

[0017] FIG. 8 is a flowchart illustrating the operation of an HMD device outputting sound according to one embodiment of the present disclosure.

[0018] FIG. 9 is a drawing illustrating a first model for material recognition of an object according to one embodiment of the present disclosure.

[0019] FIG. 10 is a drawing illustrating a second model for predicting a user's location according to one embodiment of the present disclosure.

[0020] FIG. 11 is a diagram illustrating an operation in which an HMD device performs spatial structure mapping based on image data according to one embodiment of the present disclosure.

[0021] FIG. 12 is a diagram illustrating an operation in which an HMD device, according to one embodiment of the present disclosure, generates sound to be provided to a user wearing the HMD device using a second sound generation method.

[0022] FIG. 13 is a diagram illustrating the operation of an HMD device changing a virtual speaker placement setting according to one embodiment of the present disclosure.

[0023] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0024] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0025] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0026] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0027] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0028] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0029] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0030] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0031] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0032] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0033] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0034] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0035] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0036] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0037] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0038] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0039] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0040] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0041] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a communication module (192) (e.g., cellular communication module, short-range communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0042] The communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0043] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0044] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0045] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface) and exchange signals (e.g., commands or data) with each other.

[0046] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0047] The number of processors (120) may be one or more. For example, the processor (120) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0048] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in memory (130). For example, the processor (120) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0049] FIG. 2 is a drawing showing the configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0050] Referring to FIG. 2, according to one embodiment, a wearable electronic device (200) (e.g., the electronic device (101) of FIG. 1) may include a light output module (211), a display member (201), a camera module (250), and / or a speaker (261).

[0051] According to one embodiment, a light output module (211) (e.g., a display module (160) of FIG. 1) may include a light source capable of outputting an image and a lens that guides the image to a display member (201). The light output module (211) may include, for example, a liquid crystal display, a digital mirror device, a liquid crystal on silicon display, an organic light emitting diode and / or a micro light emitting diode (micro LED).

[0052] According to one embodiment, a display member (201) (e.g., a display module (160) of FIG. 1) may include an optical waveguide (e.g., a waveguide). According to one embodiment, an output image of an optical output module (211) incident on one end of the optical waveguide may propagate within the optical waveguide and be provided to a user. According to one embodiment, the optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and / or a reflective element (e.g., a reflective mirror). For example, the optical waveguide may guide the output image of the optical output module (211) to the user's eye using at least one diffractive element or reflective element.

[0053] According to one embodiment, a camera module (250) (e.g., camera module (180) of FIG. 1) can capture images (e.g., still images and / or video). According to one embodiment, the camera module (250) may be placed within a lens frame and around a display member (201). In the present disclosure, images may be interpreted to include video as well as still images.

[0054] According to one embodiment, the first camera module (251) can capture and / or recognize the trajectory of the user's eye (e.g., pupil, iris) or gaze. According to one embodiment, the first camera module (251) can periodically or non-periodically transmit information related to the trajectory of the user's eye or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1).

[0055] According to one embodiment, the second camera module (253) can capture an external image. For example, the second camera module (253) can capture an image of the external environment in the front direction of the wearable electronic device (200).

[0056] According to one embodiment, the third camera module (255) may be used for hand detection and tracking and user gesture (e.g., hand movements) recognition. According to one embodiment, the third camera module (255) may be used for 3 degrees of freedom (3DoF) and 6DoF head tracking, location (space, environment) recognition, and / or movement recognition. According to one embodiment, the second camera module (253) may be used for hand detection and tracking and user gesture recognition. According to one embodiment, at least one of the first camera module (251) to the third camera module (255) may be replaced with a sensor module (e.g., LiDAR sensor). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), a diode, an infrared sensor, an infrared diode, and / or a photodiode.

[0057] According to one embodiment, a speaker (261) (e.g., the acoustic output module (155) of FIG. 1) can output an acoustic signal (e.g., sound and / or virtual vibration sound). Although the speaker (261) is described as being configured in a member that is mounted on the user's ear when the wearable electronic device (200) is worn in FIG. 2, it is not limited thereto and may be configured in other locations depending on the implementation of the wearable electronic device (200).

[0058] FIGS. 3a to 3c are drawings showing the front and rear views of a wearable electronic device according to one embodiment of the present disclosure.

[0059] Referring to FIGS. 3a through 3c, according to one embodiment, at least one first camera module (311, 312) and at least one second camera module (313, 314, 315, 316), a depth sensor (317), and / or a second display (350) may be disposed on a first surface (310) of a housing for acquiring information related to the surrounding environment of a wearable electronic device (300) (e.g., electronic device (101) of FIG. 1).

[0060] According to one embodiment, at least one first camera module (311, 312) can capture an image of the outside of the wearable electronic device (300). For example, the first camera module (311, 312) can capture an image of the external environment in the front direction of the wearable electronic device (300).

[0061] According to one embodiment, at least one second camera module (313, 314, 315, 316) can acquire images while the wearable electronic device (300) is worn by a user. The second camera module (313, 314, 315, 316) may be used for hand detection, tracking, and user gesture (e.g., hand movements) recognition. The second camera module (313, 314, 315, 316) may be used for 3DoF, 6DoF head tracking, location (space, environment) recognition, and / or movement recognition. According to one embodiment, a first camera module (311, 312) may be used for hand detection and tracking and user gestures.

[0062] According to one embodiment, the depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Alternatively, or additionally, a second camera module (313, 314, 315, 316) may determine the distance to an object.

[0063] According to one embodiment, the second display (350) (and / or lens) may be placed on the first surface (310) of the wearable electronic device (300). According to one embodiment, the second display (350) may provide visual information to the outside of the wearable electronic device (300). For example, the second display (350) may be used to provide an alternative notification indicating the operating status of the first camera module (311, 312) in place of the light emitter (340).

[0064] According to one embodiment, camera modules (325, 326) for face recognition and / or a first display (321) (and / or a lens) may be disposed on the second surface (320) of the housing.

[0065] According to one embodiment, camera modules (325, 326) for face recognition adjacent to the first display (321) may be used to recognize the user's face or to recognize and / or track both of the user's eyes.

[0066] According to one embodiment, the first display (321) (and / or lens) may be disposed on the second surface (320) of the wearable electronic device (300). According to one embodiment, the wearable electronic device (300) may not include camera modules (315, 316) among a plurality of second camera modules (313, 314, 315, 316). Although not illustrated in FIG. 3a and 3b, the wearable electronic device (300) may further include at least one of the configurations illustrated in FIG. 2.

[0067] Referring to FIG. 3c, according to one embodiment, the wearable electronic device (300) may have a form factor (e.g., a head-mounted display (HMD)) for being worn on a user's head. The wearable electronic device (300) may further include a strap and / or a wearing member for being secured on a part of the user's body. The wearable electronic device (300) may include a volume button (331), a vent (333), a status indicator (335), and a power button (e.g., including a fingerprint recognition sensor) (337), and such configurations may be identically included in the wearable electronic device (300) illustrated in FIG. 3a and FIG. 3b. When worn on a user's head, it may provide a user experience based on augmented reality, virtual reality, and / or extended reality (or mixed reality). The wearable electronic device (300) configured in the form of an HMD may include configurations identical or similar to the components of FIG. 3a and FIG. 3b described above.

[0068] According to one embodiment, a speaker (318) (e.g., the acoustic output module (155) of FIG. 1 or the speaker (261) of FIG. 2) may output an acoustic signal (e.g., sound and / or virtual vibration sound). Although the speaker (318) has been described as being configured in a location adjacent to the vent (333) in FIG. 3a through 3c as an example, it is not limited thereto and may be configured in other locations depending on the implementation of the wearable electronic device (200).

[0069] In the following embodiments, for convenience of explanation, the device performing at least one operation according to various embodiments of the present disclosure (e.g., embodiments of FIG. 4a to 13) is described as an HMD device worn on a user's head (e.g., electronic device (101) of FIG. 1, wearable electronic device (200) of FIG. 2, or wearable electronic device (300) of FIG. 3a to 3c). The HMD device may be, for example, a VR (virtual reality) device, a VST (video see-through) device, or an AR (augmented reality) device. A VR device (e.g., wearable electronic device (300) of FIG. 3a to 3c) may be a device designed to allow the user to be fully immersed in a virtual environment. A VST device (e.g., wearable electronic device (300) of FIG. 3a to 3c) may be a device that uses a camera to combine and display the external environment with an application, as an example of an HMD device. An AR device (e.g., the wearable electronic device (200) of FIG. 2) may show a virtual speaker to the user through a mini display or an image projected onto ordinary glasses, or may realistically play only sound without an image. However, the embodiments are not limited thereto, and the embodiments of the present disclosure may be applied to various types of electronic devices (e.g., the electronic device (101) of FIG. 1) or various other types of wearable electronic devices.

[0070] According to one embodiment of the present disclosure, an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) may provide sound to a user wearing the HMD device by using a specified audio technology or sound processing technology. For example, the HMD device may provide surround sound (e.g., 3D sound) to a user wearing the HMD device by using a specified spatial audio technology or three-dimensional (3D) audio technology. The specified spatial audio technology or 3D audio technology may be provided, for example, through a game engine or a dedicated audio engine.

[0071] According to one embodiment, an HMD device may utilize at least one virtual speaker to provide sound (e.g., surround sound). For example, the HMD device may place a plurality of virtual speakers in a space (e.g., an indoor space) based on virtual speaker placement setting information, and provide surround sound through the plurality of virtual speakers placed in the space. The virtual speaker placement setting information may be manually set by a user based on user input, for example, or may be automatically set and / or activated when an application for sound playback is executed. Placing virtual speakers in a space may include placing virtual speakers in a virtual space corresponding to the space.

[0072] According to one embodiment, virtual speaker placement setting information may include at least one setting information related to the placement of virtual speakers for providing sound (e.g., surround sound). For example, the virtual speaker placement setting information may include setting information for the number of virtual speakers, setting information for the placement location of the virtual speakers, and / or setting information for the placement direction of the virtual speakers.

[0073] According to one embodiment, an HMD device may provide sound (e.g., surround sound) through at least one virtual speaker using a first sound generation method or a second sound generation method. The first sound generation method may be a method of generating sound using only spatial position information, for example, which provides information about the user's location and / or direction. The second sound generation method may be a method of generating sound using the user's spatial position information, structural information of the space where the user is located, and material information of at least one object placed in the space. According to one embodiment, at least one object may include one or more objects that affect the provision of sound in the space. For example, at least one object may include a wall, column, ceiling, and / or floor placed in the space, but is not limited thereto. The material information may include sound characteristic information corresponding to the material of the object. The material may be, for example, concrete, wallpaper, wood, metal, glass, or vinyl, but is not limited thereto. The sound characteristic information may include at least one characteristic information that affects the sound of the object. For example, sound characteristic information may include characteristic information related to sound characteristics at the surface or boundary of the object. The sound characteristic information may include information indicating characteristics such as reflection, absorption, transmission, scattering, and attenuation of sound at the surface or boundary of the object. Such sound characteristic information may be associated with the material of the object or the boundary of the object. In the present disclosure, sound characteristic information may also be referred to as sound boundary characteristic information.

[0074] FIGS. 4a and 4b are drawings illustrating an operation in which an HMD device, according to one embodiment of the present disclosure, generates sound to be provided to a user wearing the HMD device using a first sound generation method.

[0075] As described above, when using the first sound generation method, the HMD device (400) (e.g., the wearable electronic device (200) of FIG. 2 or the wearable electronic device (300) of FIG. 3a and 3b) can generate sound (e.g., surround sound) to be provided to a user wearing the HMD device (400) by using only the user's spatial location information without using spatial structural information and material information.

[0076] Referring to FIG. 4a, the HMD device (400) acquires spatial location information (P1, P2) that provides information about the location and / or direction of a user located in a first space (SP1) (e.g., a living room), and can generate sound to be provided through at least one virtual speaker (VS1, VS2, VS3 and / or VS4) based on the spatial location information (P1, P2). For example, the HMD device (400) can generate a first sound for a user located in a first spatial location (P1) corresponding to a first location and a first direction. Subsequently, the HMD device (400) can generate a second sound for a user located in a second spatial location (P2) based on identifying that the user's location has changed from a first spatial location (P1) in the first space (SP) to a second spatial location (P2) in the same first space (SP1) corresponding to a second location and a second direction. The first sound may differ from the second sound. For example, the first sound may be a surround sound optimized for a user at a first spatial location (P1), and the second sound may be a surround sound optimized for a user at a second spatial location (P2). Through the generation of sound based on the user's location and / or direction, sound that changes according to the user's movement and / or change of direction can be provided to the user. Through this, realistic sound, such as hearing in a real environment, can be provided to the user.

[0077] Referring to FIG. 4b, a user wearing an HMD device (400) may move to a second space (SP2) (e.g., a room) that is different from a first space (SP1) (e.g., a living room) in which at least one virtual speaker (VS1, VS2, VS3 and / or VS4) is placed. In other words, the environment in which the user is located may change. Even in this case, the HMD device (400) using the first sound generation method may generate a third sound using only the user's changed third spatial location information (P3) without detecting the changed environment. Since the third sound generated in this way corresponds to a sound generated by considering only the user's spatial location without considering the changed environment, it is difficult to provide a sound optimized for the user located in the changed environment or space. For example, the third sound is a sound provided to a user located in a space (SP2) different from the space (SP1) in which at least one virtual speaker (VS1, VS2, VS3, and / or VS4) is placed, and it corresponds to a sound that considers only the user's relative position to the at least one virtual speaker (VS1, VS2, VS3, and / or VS4) and does not consider changes in sound caused by the wall between the two spaces and / or the sound characteristics (e.g., sound boundary characteristics) of the changed space. Therefore, the third sound cannot provide the user with a realistic sound similar to what is heard in a real environment.

[0078] As such, when using the first sound generation method, the HMD device (400) provides sound suitable for a fixed environment such as FIG. 4a, but cannot provide sound adapted to a changing environment such as FIG. 4b. Therefore, it is necessary to provide a sound generation method such as the second sound generation method to provide realistic sound even in a changing environment. Hereinafter, various embodiments for generating and providing sound using the second sound generation method will be described with reference to each drawing.

[0079] FIG. 5 is a drawing illustrating the configuration of an HMD device according to one embodiment of the present disclosure.

[0080] Referring to FIG. 5, the HMD device (500) (e.g., electronic device (101) of FIG. 1, wearable electronic device (200) of FIG. 2, wearable electronic device (300) of FIG. 3a to 3c, or HMD device (400) of FIG. 4) may include at least one camera (510), at least one sensor (520), a processor (530), a memory (540), at least one speaker (550) and / or a communication circuit (560). Depending on the embodiment, at least one of the components of the HMD device (500) may be omitted, or additional components (e.g., a display) may be further included.

[0081] According to one embodiment, a camera (510) (e.g., camera module (180) of FIG. 1) can acquire image data by capturing an image of the surrounding environment of the HMD device (500). For example, the camera (510) can acquire image data by capturing an image of the front direction of the HMD device (500). The image data acquired through the camera (510) may include an image of at least one object (e.g., wall, ceiling, floor) placed in the space (e.g., space (SP1, SP2) of FIG. 4a and 4b) where the HMD device (500) (or a user wearing the HMD device (500)) is located. The image data may be used to acquire structural information of the space, material information of at least one object placed in the space, and / or spatial location information of the HMD device (500) (or a user wearing the HMD device (300)).

[0082] According to one embodiment, a sensor (520) (e.g., sensor module (176) of FIG. 1) can sense the surrounding environment of an HMD device (500) to obtain sensing data. The sensing data can be used to obtain spatial location information and / or movement information of the HMD device (500) (or a user wearing the HMD device (500)). The movement information may include, for example, information corresponding to the movement of the HMD device (500), the movement of the user wearing the HMD device (500), or the movement of the head of the user wearing the HMD device (500).

[0083] According to one embodiment, the processor (530) (e.g., the processor (120) of FIG. 1) may include a map generation module (531), a material recognition module (532), a user movement tracking module (533), and / or a sound generation module (534). For example, when instructions stored in memory (540) (e.g., the memory (130) of FIG. 1) are executed by the processor (530), the HMD device (500) may be caused to perform at least one operation of the map generation module (531), the material recognition module (532), the user movement tracking module (533), and / or the sound generation module (534).

[0084] According to one embodiment, the map generation module (531) can generate a map representing the structure of a space (e.g., an indoor space). For example, the map generation module (531) can identify at least one object (e.g., a wall, a floor, a ceiling) placed in the space based on image data acquired through a camera (510), and generate a two-dimensional (2D) or three-dimensional (3D) map representing the structure of the space based on the identified at least one object. The identification of at least one object may include identifying information such as the placement, location (e.g., relative location), shape, and area of ​​at least one object.

[0085] According to one embodiment, the material recognition module (532) can recognize the material of at least one object placed in space. For example, the material recognition module (532) can identify the material of at least one object placed in space based on image data acquired through the camera (510) to acquire material-related data. The material-related data may include material classification information and / or material information. Since the material of an object affects the characteristics of sound (e.g., boundary characteristics) in the space (or environment), accurately identifying or classifying the material is important for providing realistic sound.

[0086] According to one embodiment, material information may include sound characteristic information corresponding to the material of the object. The material may be, for example, concrete, wallpaper, wood, metal, glass, or vinyl, but is not limited thereto. The sound characteristic information may include at least one characteristic information that affects the sound of the object. For example, the sound characteristic information may include characteristic information related to sound characteristics at the surface or boundary of the object. The sound characteristic information may include information indicating characteristics such as reflection, absorption, transmission, scattering, and attenuation of sound at the surface or boundary of the object.

[0087] According to one embodiment, the material recognition module (532) can obtain sound characteristic information for at least one object placed in space based on image data. For example, the material recognition module (532) can obtain material classification information for at least one object using image data, and can obtain sound boundary characteristic information including information on the reflection coefficient of sound by calculating the reflection coefficient of sound using the spatial location information of the user on the map and / or the material classification information of at least one object. For example, the material recognition module (532) can obtain sound boundary characteristic information for at least one object based on image data using a learned first model (e.g., the first model (900) of FIG. 9).

[0088] According to one embodiment, the first model may be trained to receive image data for at least one object as input data and output material classification information of at least one object as output data. The material classification information may include information indicating the material of the object. In this case, the material recognition module (532) may obtain sound boundary characteristic information corresponding to the material of the object identified based on the material classification information. For example, the material recognition module (532) may obtain sound characteristic information (e.g., reflection coefficient) corresponding to the material of the object identified based on the material classification information by using a preset material sound characteristic mapping table (e.g., the mapping table of Table 1).

[0089] According to one embodiment, the first model may be trained to receive image data for at least one object as input data and output sound characteristic information (e.g., reflection coefficient) for at least one object as output data. In this case, the material recognition module (532) may directly obtain sound characteristic information for at least one object through the first model without obtaining material classification information.

[0090] According to one embodiment, the user movement tracking module (533) can track the user's movement or motion (e.g., the user's head movement or the movement of the HMD device (500)) in real time and update the user's location on the map. For example, the user movement tracking module (533) can detect the user's movement (e.g., location and / or direction) in real time using image data obtained through the camera (510) and / or sensing data obtained through the sensor (520), and obtain (e.g., estimate) a new location of the user based on the amount of change in movement.

[0091] According to one embodiment, the sound generation module (534) can generate sound (e.g., surround sound) using structural information of a space, material information for at least one object placed in the space, and / or virtual speaker placement setting information. To generate surround sound, the sound generation module (334) can synthesize a plurality of sounds using a configuration such as a game engine or a digital signal processing library.

[0092] According to one embodiment, the sound generation module (534) can provide the generated sound to the user. For example, the sound generation module (534) can output surround sound through a plurality of speakers (550) of the HMD device (500). For example, the sound generation module (534) can transmit data of the generated sound to an external sound device (e.g., earbuds) through the communication circuit (560) of the HMD device (500). In this case, the sound device can output surround sound through a plurality of speakers of the sound device.

[0093] According to one embodiment, the sound generation module (534) can generate multichannel sound through a plurality of virtual speakers (e.g., virtual speakers of FIG. 4a and 4b (VS1, VS2, VS3 and / or VS4)). The multichannel sound may be, for example, 5.1 channel or 7.1 channel sound. Through this, the user can obtain a more vivid surround sound experience.

[0094] Meanwhile, according to the embodiment, some of the above-described configurations may be included in the HMD device (530), and other parts of the above-described configurations may be included in another electronic device (e.g., the electronic device (101) of FIG. 1). For example, an electronic device such as a smartphone or a server may be connected to the HMD device (530) to generate sound using image data acquired through the camera of the HMD device (530) and to transmit the generated sound to the HMD device (530). In this case, the map generation module (531), material recognition module (533), and / or sound generation module (534) of the processor (530) may be included in the electronic device that generates the sound. In this case, the electronic device may perform at least some of the operations of the HMD device described with reference to FIG. 6 through 13 to generate sound.

[0095] FIG. 6 is a flowchart illustrating the operation of an HMD device generating sound according to one embodiment of the present disclosure.

[0096] Referring to FIG. 6, in operation 610, an HMD device (e.g., the HMD device (500) of FIG. 5) can obtain structural information about the space where the user of the HMD device is located (e.g., the space (SP1, SP2) of FIG. 4a and 4b) based on image data obtained through the camera of the HMD device (e.g., the camera (510) of FIG. 5). The space where the user of the HMD device is located may correspond to the space where the HMD device worn by the user is located.

[0097] In operation 620, the HMD device can acquire material information for at least one object located in space based on image data using a learned first model (e.g., the first model (900) of FIG. 9). An example of the operation for acquiring material information is described below with reference to FIG. 7, 8 and 9.

[0098] According to one embodiment, at least one object may include, for example, at least one of a wall, ceiling, column, or floor located in the space. According to one embodiment, material information may include sound characteristic information corresponding to the material of the object. The sound characteristic information may include information regarding the reflection coefficient or sound absorption coefficient of sound corresponding to the material of the object.

[0099] According to one embodiment, the first model may be trained to receive image data for at least one object as input data and to output material classification information of at least one object as output data. The material classification information may include information indicating the material (or type of material) of the object. When material classification information is obtained, the HMD device may obtain sound characteristic information (e.g., reflection coefficient) corresponding to the material of the object identified based on the material classification information. For example, the HMD device may obtain sound characteristic information corresponding to the material of the object identified based on the material classification information by using a preset material sound characteristic mapping table (e.g., the mapping table of Table 1).

[0100] According to one embodiment, the first model may be trained to receive image data for at least one object as input data and to output sound characteristic information (e.g., reflection coefficient) for at least one object as output data. In this case, the HMD device may directly obtain sound characteristic information for at least one object from the image data through the first model. However, in this case, the size of the first model may be larger than when material classification information is obtained through the first model.

[0101] In operation 630, the HMD device can generate sound (e.g., surround sound) based on structural information, material information and / or virtual speaker placement setting information. For example, the HMD device can generate sound (e.g., surround sound) at the user's location based on structural information, material information and / or virtual speaker placement setting information.

[0102] According to one embodiment, the HMD device can acquire predicted location information of a user based on the user's location data using a learned second model. The user's location data may be acquired based on image data acquired through a camera or sensing data acquired through a sensor (e.g., sensor (520) of FIG. 5). According to one embodiment, when acquiring predicted location information, the HMD device can generate sound at the predicted location based on the predicted location information by using structural information, material information, and speaker placement setting information. An example of an operation for acquiring predicted location information is described below with reference to FIGS. 8 and 10.

[0103] According to one embodiment, the HMD device can generate or update a three-dimensional or two-dimensional map of space based on image data. The map can provide structural information about space and location information of the user on the map. The map thus generated may be, for example, the same as the maps illustrated in FIGS. 4a, 4b, 12, and 13.

[0104] According to one embodiment, a map may display the location, size, and / or shape of each space (e.g., room, living room). Through such a map, a user can determine their location and set at least one virtual speaker object at a desired location. In the present disclosure, the virtual speaker object may also be referred to as a virtual audio object.

[0105] According to one embodiment, the HMD device may set at least one virtual speaker object on a map. For example, as illustrated in FIGS. 4a, 4b, 12 and 13, the HMD device may set a plurality of virtual speaker objects (VR1 to VR4). The HMD device may provide sound corresponding to the user whenever the user moves in space through the set virtual speaker objects. For example, when the user moves through a room, the HMD device may provide surround sound based on the analysis results between the preset virtual speaker objects and the user's current location, taking into account factors such as the material of the walls and ceiling, and the location of the door between rooms. In this case, the HMD device may implement more realistic sound effects by taking into account sound characteristics (e.g., sound reflection coefficient) according to the material of the walls.

[0106] According to one embodiment, the HMD device may utilize a physical modeling or machine learning-based sound propagation algorithm to generate suitable sound. According to one embodiment, the HMD device may provide more realistic sound by considering sound reflections based on materials and structures in real time.

[0107] According to one embodiment, the HMD device can process the material analysis of an object recognized through a camera and / or the prediction of the user's current location using a learned model (e.g., an AI model and a machine learning model). In this way, by processing the object material analysis and user location prediction operations in real time through the learned model, the HMD device can output (or play) surround sound that is close to reality.

[0108] According to one embodiment, the HMD device can identify whether the user's current location or predicted location belongs to a second space different from a first space in which at least one virtual speaker is placed based on virtual speaker placement setting information. Based on the identification that the user's current location or predicted location belongs to the second space, the HMD device can change the virtual speaker placement setting information so that at least one virtual speaker is placed in the second space.

[0109] According to one embodiment, the HMD device may include a plurality of speakers (e.g., speakers (550) of FIG. 5). In this case, the HMD device may output surround sound through the plurality of speakers.

[0110] According to one embodiment, the HMD device includes a communication circuit (e.g., the communication circuit (360) of FIG. 5) and can transmit sound data generated through the communication circuit to a sound device including a plurality of speakers. In this case, the sound device can output surround sound through the plurality of speakers using the received sound data.

[0111] FIG. 7 is a flowchart illustrating the operation of an HMD device outputting sound according to one embodiment of the present disclosure.

[0112] According to one embodiment, in the embodiment of FIG. 7, the HMD device can obtain material information for at least one object placed in the space where the user is located by using a learned first model (e.g., the first model (900) of FIG. 9) and a preset material sound characteristic mapping table (e.g., the mapping table of Table 1).

[0113] Referring to FIG. 7, in operation 710, an HMD device (e.g., the HMD device (500) of FIG. 5) can acquire image data. For example, the HMD device can acquire image data through a plurality of cameras of the HMD device (e.g., the camera (510) of FIG. 5).

[0114] According to one embodiment, in operation 720, the HMD device may perform mapping of a spatial structure based on image data. For example, the HMD device may analyze image data to obtain spatial structure information and generate or update a map (e.g., a 3D map) by performing mapping of the spatial structure based on the structural information. The structural information may include, for example, the shape of the space and / or placement information regarding the arrangement of at least one object existing in the space. In the present disclosure, the structural information may be referred to as spatial structure information.

[0115] According to one embodiment, in operation 730, the HMD device can identify at least one object based on image data. In operation 740, the HMD device can analyze the material of the identified at least one object. Through this, material classification information can be obtained.

[0116] According to one embodiment, operation 730 and / or operation 740 may be performed using a learned first model. For example, an HMD device may obtain material classification information for at least one object based on image data using the learned first model. In this case, the first model may be used to identify an object from the image data and to classify the material of the identified object. The material classification information may include information indicating the material of the object.

[0117] According to one embodiment, in operation 750, the HMD device can obtain material information for at least one object by using material classification information for at least one object and a material sound characteristic mapping table. The material sound characteristic mapping table may be a mapping table that includes information on sound characteristics (e.g., reflection coefficients by material) of the object's material. In the present disclosure, the material sound characteristic mapping table may be abbreviated as a mapping table.

[0118] Table 1 below may be an example of a mapping table containing sound reflection coefficients by material of an object.

[0119] Material Reflectance Coefficient (R) Concrete 0.02~0.05 Wallpaper 0.02~0.04 Wood 0.1 Metal 0.5 Glass 0.03~0.05 Vinyl 0.02~0.04

[0120] Referring to Table 1, for example, if the material of the object is concrete, the HMD device can set the reflection coefficient of the object to one of 0.02 to 0.05.

[0121] According to one embodiment, the sound reflection coefficient may be a coefficient representing the ratio of sound (or sound waves) that is reflected when sound (or sound waves) strikes a specific boundary (or surface). Through this, it is possible to quantify how much acoustic energy is reflected, absorbed, or transmitted. This reflection coefficient can be used as information to analyze the sound characteristics of the physical surface of an object.

[0122] According to one embodiment, the reflection coefficient can be calculated by the following mathematical formula 1.

[0123] [Mathematical Formula 1]

[0124] R = Pr / Pi

[0125] Here, R is the reflection coefficient, Pr is the reflected acoustic energy, and Pi is the incident acoustic energy.

[0126] According to mathematical formula 1, the reflection coefficient ranges from 0 to 1, and it can be seen that as the value increases, sound reflection occurs more effectively. For example, if R=1, all sound is reflected, and if R=0, all sound is absorbed, so there may be no reflection.

[0127] According to one embodiment, a mapping table such as Table 1 can be generated in advance using a specified model (e.g., a machine learning model) and stored in memory (e.g., memory (540) of FIG. 5). For example, an electronic device other than the HMD device (e.g., the electronic device (101) or server (108) of FIG. 1) can generate a mapping table using a machine learning model and transmit the data of the generated mapping table to the HMD device. In this case, the HMD device can reduce the size of the entire model used to acquire material information by storing only the first model used to acquire material classification information, without needing to store the machine learning model for generating the mapping table. This prevents processor overload. Additionally, the HMD device can quickly generate sound boundary characteristic information corresponding to the material of an object acquired based on image data by using the pre-set and stored mapping table.

[0128] According to one embodiment, in operation 760, the HMD device can generate sound (e.g., surround sound) based on material information and spatial structure information. For example, the HMD device can generate surround sound based on material information, spatial structure information and virtual sound placement setting information.

[0129] According to one embodiment, an HMD device can generate surround sound by performing a simulation of a space (e.g., a room) using the Sabine formula or the Eyring formula. The Sabine formula may be a formula used to calculate reverberation time in indoor acoustics. Reverberation time may refer to the time until sound disappears. The Sabine formula may be as shown in Equation 2 below.

[0130] [Mathematical Formula 2]

[0131] T = (0.161 * V) / A

[0132] Here, T is the reverberation time (seconds) and V is the volume of space (m² 3 ) and A is the sound absorption area (m 2 It can be.

[0133] The sound absorption area A can be calculated by summing the products of the sound absorption coefficient and the area of ​​each surface in the space, as shown in the following mathematical formula 3.

[0134] [Mathematical Formula 3]

[0135]

[0136] Here, Si is the area of ​​each surface (m 2 ) and αi can be the sound absorption coefficient of each surface (a value between 0 and 1).

[0137] According to one embodiment, the sound absorption coefficient (α) may be related to the reflection coefficient (R). For example, the sound absorption coefficient (α) may have the relationship with the reflection coefficient (R) as shown in the following Equation 4.

[0138] [Mathematical Formula 4]

[0139] α + R = 1

[0140] According to one embodiment, in operation 770, the HMD device can output the generated sound. For example, the HMD device can output surround sound through a plurality of speakers (e.g., speakers (550) of FIG. 5).

[0141] FIG. 8 is a flowchart illustrating the operation of an HMD device outputting sound according to one embodiment of the present disclosure.

[0142] According to one embodiment, in the embodiment of FIG. 8, the HMD device can predict the user's location using a learned second model (e.g., the second model (1000) of FIG. 10) and generate a sound corresponding to the predicted location.

[0143] Referring to FIG. 8, in operation 810, an HMD device (e.g., the HMD device (500) of FIG. 5) can acquire image data. For example, the HMD device can acquire image data through a plurality of cameras of the HMD device (e.g., the camera (510) of FIG. 5).

[0144] According to one embodiment, in operation 820, the HMD device may perform mapping of a spatial structure based on image data. For example, the HMD device may analyze image data to obtain spatial structure information and generate or update a map (e.g., a 3D map) by performing mapping of the spatial structure based on the structural information. The structural information may include, for example, the shape of the space and / or placement information regarding the arrangement of at least one object existing in the space. In the present disclosure, the structural information may be referred to as spatial structure information.

[0145] According to one embodiment, in operation 830, the HMD device can identify at least one object based on image data and analyze the material of at least one identified object. Through this, material information can be obtained.

[0146] According to one embodiment, operation 830 may include operations 730, 740, and 750 of FIG. 7. In this case, operation 830 may be performed using a learned first model (e.g., the first model (900) of FIG. 9) and a preset material sound characteristic mapping table (e.g., the mapping table of Table 1).

[0147] According to one embodiment, unlike the embodiment of FIG. 7, in the embodiment of FIG. 8, the HMD device may acquire material information for at least one object placed in the space where the user is located by using only a learned first model without using a material sound characteristic mapping table. In this case, operation 830 may be performed using only a learned first model without using a preset material sound characteristic mapping table. For example, the HMD device may directly acquire material information for at least one object based on image data using the learned first model. In this case, the first model may be used to identify an object from the image data, classify the material of the identified object, and acquire material information corresponding to the classified material. The material information may include sound characteristic information (e.g., reflection coefficient) corresponding to the material of the object. When the HMD device directly acquires material information using only a learned first model without using a material sound characteristic mapping table, the performance and accuracy of the first model may increase with continuous learning or updating of the first model, thereby enabling the acquisition of increasingly accurate material information. However, in this case, the size of the first model may be larger than that of the embodiment of FIG. 7. As a result, the load on the processor for processing the first model may be larger.

[0148] According to one embodiment, in operation 840, the HMD device may obtain location prediction data by predicting the user's location based on the user's location data using a learned second model. The location prediction data may include predicted location information, and the predicted location information may include, for example, information about the predicted location (predicted location information) and / or information about the predicted movement path (predicted movement path information). The user's location data may be obtained based on image data obtained through a camera or sensing data obtained through a sensor (e.g., sensor (520) of FIG. 5). When obtaining the predicted location information, the HMD device may generate sound at the predicted location based on the predicted location information by using structural information, material information and / or speaker placement setting information. Through this, sound suitable for the user at the predicted location may be provided.

[0149] According to one embodiment, in operation 850, the HMD device can generate sound (e.g., surround sound) at a predicted location based on predicted location information, based on spatial structure information, material information and / or speaker placement setting information. If the user moves, and the sound is generated simply based on the user's current location, it is difficult to provide accurate sound corresponding to the user's actual location at the time the sound is output due to the delay in sound generation. Therefore, it is necessary to predict the user's movement and generate sound based on the user's predicted location. In this case, accurate sound corresponding to the user's actual location at the time the sound is output can be provided despite the delay in sound generation.

[0150] According to one embodiment, in operation 860, the HMD device can output the generated sound. For example, the HMD device can output surround sound through a plurality of speakers (e.g., speakers (550) of FIG. 5).

[0151] FIG. 9 is a drawing illustrating a first model for material recognition of an object according to one embodiment of the present disclosure.

[0152] Referring to FIG. 9, the first model (900) can receive image data (901) as input data and output material-related data (902) as output data. The material-related data (902) may include material classification information and / or material information. For example, the first model (900) can be trained to receive image data (901) for at least one object as input data and output material classification information of at least one object as output data. For example, the first model (900) can be trained to receive image data (901) for at least one object as input data and output material information of at least one object as output data.

[0153] According to one embodiment, the first model (900) may include a model using convolutional neural networks (CNNs). A CNN is a deep learning model designed to process image or video data and can be used to classify the type of material of an object or sound characteristics corresponding to the material based on image data (901).

[0154] According to one embodiment, the first model (900) can be trained using a transfer learning method. The transfer learning method is a training method for solving problems in a new domain using an already trained model (e.g., CNN), and can be used to convert a CNN used for existing image classification into a CNN used for classifying material types or sound characteristics corresponding to materials. In this case, the training time can be reduced compared to training a completely new CNN.

[0155] According to one embodiment, the first model (900) may include a model that utilizes support vector machines (SVMs). An SVM is a machine learning model based on feature vectors and can be used to classify the type of material of an object or sound characteristics corresponding to the material using feature data extracted from image data (901). The feature data input to the SVM may be, for example, feature data obtained through feature extraction via a CNN. In this case, the first model (900) may include both a CNN and an SVM, wherein the CNN extracts feature data from the image data (901), and the SVM uses the feature data extracted via the CNN to obtain (or classify) the type of material of an object or sound characteristics corresponding to the material.

[0156] According to one embodiment, the first model (900) may be a model including auto encoders. The auto encoder is a deep learning structure used for image compression and restoration, and may be used to classify the type of material of an object or sound characteristics corresponding to the material based on image data (901).

[0157] According to one embodiment, the first model (900) may be a model including GANs (generative adversarial networks). A GAN is a deep learning structure that generates virtual images that are close to reality and can be used to generate virtual images of the material of an object. The virtual images thus generated can be used to classify the type of material of an object or sound characteristics corresponding to the material.

[0158] According to one embodiment, the first model (900) may be a model that combines at least some of the above-described models (CNN, SVM, autoencoder, GAN).

[0159] FIG. 10 is a drawing illustrating a second model for predicting a user's location according to one embodiment of the present disclosure.

[0160] Referring to FIG. 10, the second model (1000) can receive location data (1001) as input data and output location prediction data (1002) as output data. The location prediction data (1002) may include information about the predicted location (predicted location information) and / or information about the predicted movement path (predicted movement path information). For example, the second model (1000) can be trained to receive location data (1001) as input data and output predicted location information as output data. For example, the second model (1000) can be trained to receive location data (1001) as input data and output predicted movement path information as output data.

[0161] According to one embodiment, location data (1001) may include values ​​of spatial locations (e.g., location and / or orientation) of a specified number of users over a specified period. For example, location data (1001) may include the spatial location value of the current user and the spatial location values ​​of the user over a previous specified period (e.g., 5 seconds).

[0162] According to one embodiment, the second model (1000) may be a model including an artificial neural network. For example, the second model (1000) may be a deep learning model based on an artificial neural network. An artificial neural network is a machine learning model created by mimicking the structure of the human brain and can be used to predict the location and direction of a user using location data.

[0163] According to one embodiment, the second model (1000) may be a model including a CNN. The CNN may be used to predict the user's location and direction using surrounding environment information along with location data (e.g., GPS coordinate data).

[0164] According to one embodiment, the second model (1000) may be a model including recurrent neural networks (RNN). An RNN is a neural network suitable for processing time-series data and can be used to predict the user's future movements by analyzing the user's past movement patterns and behaviors based on location data.

[0165] According to one embodiment, the second model (1000) may be a model including a graph neural network (GNN). A GNN is a neural network for modeling unstructured relational data and can be used to predict a user's movement path by considering complex relationships such as geographical characteristics.

[0166] According to one embodiment, the second model (1000) can be learned by a reinforcement learning method. A reinforcement learning method is a learning method in which a model interacts with an environment and selects an action in a direction that maximizes the reward, and can use a method of setting and optimizing a reward function according to the user's movement path.

[0167] According to one embodiment, the second model (1000) may be a model that combines at least some of the above-described models (CNN, RNN, GNN).

[0168] FIG. 11 is a diagram illustrating an operation in which an HMD device performs spatial structure mapping based on image data according to one embodiment of the present disclosure.

[0169] The operation of the embodiment of FIG. 11 may be performed, for example, by a map generation module of an HMD device (500) (e.g., the map generation module (531) of FIG. 5). According to one embodiment, the operation of the embodiment of FIG. 11 may be an example of the spatial structure mapping operation 820 of FIG. 8 or the spatial structure mapping operation of FIG. 9. In the embodiment of FIG. 11, for convenience of explanation, the space is described as an indoor space corresponding to a kitchen and a living room, but the embodiment is not limited thereto. For example, the description of the same embodiment may be applied to other indoor spaces such as a room or outdoor spaces.

[0170] According to one embodiment, as illustrated in part (a) of FIG. 11, the HMD device (500) can acquire (or collect) spatial location information of the user and / or information about the surrounding environment based on image data acquired by capturing the surrounding environment (e.g., an image in the front direction) of the user wearing the HMD device (500) through at least one camera of the HMD device (500) (e.g., camera (510) of FIG. 5). The HMD device (500) can map the user's location and surrounding environment in real time based on image data acquired in real time through the camera using a specified mapping technique (e.g., SLAM (simultaneous localization and mapping)).

[0171] According to one embodiment, the HMD device (500) can capture a point cloud of a three-dimensional space of the surrounding environment through a camera and acquire image data including data of the point cloud. A point cloud may refer to numerous points located in a three-dimensional space. These points can be continuously collected through the camera according to the direction in which the HMD device (500) moves.

[0172] According to one embodiment, the HMD device (500) can obtain structural information of a space (e.g., an indoor space) by analyzing collected point cloud data. For example, the HMD device (500) can obtain structural information of a space by analyzing point cloud data using vision technology and / or machine learning algorithms.

[0173] According to one embodiment, structural information may include placement information for at least one object placed in the space. The at least one object may include, but is not limited to, a first wall (OB1), a second wall (OB2), furniture (OB3), a floor (OB4), and / or a ceiling (OB5), for example, as illustrated in part (b) of FIG. 11. The placement information may include, for example, information about the location of the object (e.g., relative location) on a map (e.g., a 3D map) or in space, and / or information about the area of ​​the surface (or boundary surface) of the object. For example, placement information for the first wall (OB1) may include information about the location of the first wall (OB1) on a map or in space, and / or information about the area of ​​the surface of the first wall (OB1). Through such structural information, the exact structure of the space where the user is located can be identified, and a map can be created or updated based on this. The map thus created may be, for example, the map illustrated in FIG. 4a, 4b, FIG. 12, and FIG. 13. According to one embodiment, as illustrated in FIGS. 4a, 4b, 12, and 13, the map displays the location, size, and / or shape of each space (e.g., room, living room) and can display the user's location. The user can check their location through the map and set at least one virtual speaker object at a desired location. For example, as illustrated in FIGS. 4a, 4b, and 12, at least one virtual speaker object (VS1, VS2, VS3, and / or VS4) can be set on the map.

[0174] FIG. 12 is a diagram illustrating an operation in which an HMD device, according to one embodiment of the present disclosure, generates sound to be provided to a user wearing the HMD device using a second sound generation method.

[0175] As described above, when using a second sound generation method, unlike the first sound generation method, the HMD device (500) (e.g., the wearable electronic device (200) of FIG. 2 or the wearable electronic device (300) of FIG. 3a and 3b) can generate sound (e.g., surround sound) to be provided to a user wearing the HMD device by using the spatial location information of the user, along with structural information and material information of the space where the user is located.

[0176] In the embodiment of FIG. 12, it is assumed that a user wearing an HMD device (500) moves to another space while at least one virtual speaker (VS1, VS2, VS3 and / or VS4) is already set up in one space on the map, for example, as illustrated in FIG. 4a.

[0177] Referring to FIG. 12, a user wearing an HMD device (500) may move to a second space (SP2) (e.g., a room) that is different from a first space (SP1) (e.g., a living room) in which at least one virtual speaker (VS1, VS2, VS3 and / or VS4) is placed. That is, the environment or space in which the user is located may change due to the user's movement (or motion). In this case, unlike an HMD device using a first sound generation method (e.g., an HMD device (400) of FIG. 4), an HMD device (500) using a second sound generation method may detect the changed environment and generate a third sound suitable for the user's changed third spatial location (P3).

[0178] Unlike the sound of the embodiment of FIG. 4b generated using the first sound generation method, the third sound of the embodiment of FIG. 12 generated using the second sound generation method corresponds to a sound generated by taking into account the changed environment and the spatial location of the user. Therefore, the third sound can provide realistic and vivid sound to a user located in the changed environment or space. For example, the third sound is a sound provided to a user located in a space (SP2) different from the space (SP1) where at least one virtual speaker (VS1, VS2, VS3 and / or VS4) is placed, and corresponds to a sound generated by taking into account not only the user's relative position to the at least one virtual speaker (VS1, VS2, VS3 and / or VS4), but also the location of the wall between the two spaces, the door between rooms, and / or the sound characteristics of the changed space (e.g., sound boundary characteristics). Therefore, the third sound can continuously provide the user with realistic sound, such as hearing in the actual environment, despite changes in the environment or space due to changes in the user's location. In this way, when using the second sound generation method, the HMD device (500) can provide the user with a sound adapted to the changing environment.

[0179] FIG. 13 is a diagram illustrating the operation of an HMD device changing a virtual speaker placement setting according to one embodiment of the present disclosure.

[0180] According to one embodiment, as illustrated in part (a) of FIG. 13, an HMD device (500) (e.g., the wearable electronic device (200) of FIG. 2 or the wearable electronic device (300) of FIG. 3a and 3b) can identify whether the user's current location or predicted location belongs to a second space (SP2) different from a first space (SP1) in which at least one virtual speaker (VS1, VS2, VS3 and / or VS4) is placed based on current virtual speaker placement setting information.

[0181] According to one embodiment, as illustrated in part (b) of FIG. 13, the HMD device (500) may change virtual speaker placement setting information so that at least one of at least one virtual speaker (VS1, VS2, VS3 and / or VS4) is placed in the second space (SP2) based on the identification that the user's current location or predicted location belongs to the second space (SP2). Depending on the change in virtual speaker placement setting information, sound suitable for the second space (SP2) may be provided to the user.

[0182] According to one embodiment, a change in virtual speaker placement setting information may be performed automatically by the HMD device (500) based on user input or without user input. When the change in virtual speaker placement setting information is based on user input, the HMD device (500) may provide a visual or auditory notification to the user guiding the change in virtual speaker placement setting information in response to the identification that the user's current location or predicted location belongs to the second space (SP2). The user confirms through the notification that a change in virtual speaker placement setting information is necessary and can change the virtual speaker placement setting information through user input.

[0183] According to one embodiment of the present disclosure, an HMD device can generate sound by using spatial location information of a user, structural information of the space where the user is located, and material information of at least one object placed in the space.

[0184] According to one embodiment, the HMD device may include a camera; at least one processor including a processing circuit; and a memory including at least one storage medium for storing instructions.

[0185] According to one embodiment, the HMD device acquires structural information about the space where the user of the HMD device is located based on image data acquired through the camera, acquires material information about at least one object located in the space based on the image data using a learned first model, and causes sound to be generated based on the structural information, the material information, and virtual speaker placement setting information, and the material information may include sound characteristic information corresponding to the material of the object.

[0186] According to one embodiment, the HMD device can obtain predicted location information of the user based on the user's location data using a learned second model. The operation of generating the sound may include: generating a sound for a user at a predicted location based on the predicted location information using the structural information, the material information, and the virtual speaker placement setting information.

[0187] According to one embodiment, the first model is trained to receive image data for the at least one object as input data and to output material classification information for the at least one object as output data, wherein the material classification information includes information indicating the type of material of the object, and the operation of acquiring the material information may include an operation of acquiring sound characteristic information corresponding to the material of the object based on the material classification information.

[0188] According to one embodiment, the operation of acquiring sound characteristic information is performed using a preset material sound characteristic mapping table, and the material sound characteristic mapping table can be set using a machine learning model.

[0189] According to one embodiment, the first model can be trained to receive the image data as input data and output the sound characteristic information corresponding to the material of at least one object as output data.

[0190] According to one embodiment, the HMD device comprises at least one object, wherein the object includes at least one of a wall, ceiling, column, or floor located in the space, and the sound characteristic information may include information regarding the reflection coefficient or sound absorption coefficient of sound corresponding to the material of the object.

[0191] According to one embodiment, the HMD device causes to generate or update a three-dimensional or two-dimensional map of the space based on the image data, and the map may provide structural information about the space and location information of the user.

[0192] According to one embodiment, the HMD device identifies whether the user's current location or predicted location belongs to a second space different from a first space in which at least one virtual speaker is placed based on the virtual speaker placement setting information, and based on the identification that the user's current location or predicted location belongs to the second space, the device can change the virtual speaker placement setting information so that the at least one virtual speaker is placed in the second space.

[0193] According to one embodiment, the HMD device includes a plurality of speakers and can output the sound through the plurality of speakers.

[0194] According to one embodiment, the HMD device includes a communication circuit and can cause the sound data to be transmitted to a sound device including a plurality of speakers through the communication circuit.

[0195] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" each may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0196] The term “module” as used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0197] One embodiment of the present document may be implemented as software (e.g., program (140) of FIG. 1) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) of FIG. 1 or external memory (138) of FIG. 1) that is readable by a machine (e.g., electronic device (101) of FIG. 1). For example, a processor (e.g., processor (120) of FIG. 1) of the machine (e.g., electronic device (101) of FIG. 1) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0198] According to one embodiment, the method according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0199] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a head-mounted display (HMD) device, camera; At least one processor including a processing circuit; and It includes memory comprising at least one storage medium for storing instructions, and When the above instructions are executed individually or collectively by the at least one processor, the HMD device: Based on image data acquired through the camera, structural information regarding the space where the user of the HMD device is located is acquired, and Using a trained first model, material information for at least one object located in the space is obtained based on the image data, and Causes the generation of sound based on the above structural information, the above material information, and virtual speaker placement setting information for placing at least one virtual speaker, and An HMD device comprising sound characteristic information corresponding to the material of the object, wherein the above material information includes sound characteristic information.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: Using a trained second model, the user's predicted location information is obtained based on the user's location data, and The operation of generating the above sound is: An HMD device comprising an operation to generate sound for a user at a predicted location based on the predicted location information, using the above structural information, the above material information, and the above virtual speaker placement setting information.

3. In Paragraph 1 or 2, The first model is trained to receive image data for at least one object as input data and to output material classification information for at least one object as output data, wherein the material classification information includes information indicating the type of material of the object. An HMD device wherein the operation of acquiring the above material information includes the operation of acquiring sound characteristic information corresponding to the material of the object based on the above material classification information.

4. In Paragraph 3, An HMD device in which the operation of acquiring the above sound characteristic information is performed using a preset material sound characteristic mapping table, and the material sound characteristic mapping table is set using a machine learning model.

5. In Paragraph 1 or 2, The above-described first model is an HMD device trained to receive the above-described image data as input data and output the above-described sound characteristic information corresponding to the material of at least one object as output data.

6. In any one of paragraphs 1 through 5, The above at least one object includes at least one of a wall, ceiling, column, or floor located in the space, and An HMD device in which the above sound characteristic information includes information regarding the reflection coefficient or sound absorption coefficient of sound corresponding to the material of the object.

7. In any one of paragraphs 1 through 6, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: Based on the above image data, it causes to generate or update a 3D or 2D map of the above space, and The above map is an HMD device that provides structural information about the space and location information of the user.

8. In any one of paragraphs 1 through 7, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: Identifying whether the current location or predicted location of the user belongs to a second space different from the first space in which at least one virtual speaker is placed based on the virtual speaker placement setting information, and An HMD device that causes the virtual speaker placement setting information to be changed so that the at least one virtual speaker is placed in the second space based on the identification that the user's current location or predicted location belongs to the second space.

9. In any one of paragraphs 1 through 8, The above HMD device includes a plurality of speakers, and When the above instructions are executed individually or collectively by the at least one processor, the HMD device: An HMD device that causes the sound to be output through the plurality of speakers.

10. In any one of paragraphs 1 through 9, The above HMD device includes a communication circuit, and When the above instructions are executed individually or collectively by the at least one processor, the HMD device: An HMD device that causes the sound data to be transmitted to a sound device including a plurality of speakers through the above communication circuit.

11. In a method for an HMD (head mounted display) device, An operation to acquire structural information about the space where the user of the HMD device is located, based on image data acquired through the camera of the HMD device; An operation to acquire material information for at least one object located in the space based on the image data using the learned first model; and The method includes an operation of generating sound based on the above structural information, the above material information, and virtual speaker placement setting information for placing at least one virtual speaker, and A method comprising sound characteristic information corresponding to the material of the object, wherein the above material information includes sound characteristic information.

12. In paragraph 11, the above method is: The operation of obtaining predicted location information of the user based on the user's location data using a second learned model is further included. The operation of generating the above sound is: A method comprising the operation of generating sound for a user at a predicted location based on the predicted location information, using the above structural information, the above material information, and virtual speaker placement setting information.

13. In Paragraph 11 or 12, The first model is trained to receive image data for at least one object as input data and to output material classification information for at least one object as output data, wherein the material classification information includes information indicating the type of material of the object. A method in which the operation of acquiring the above material information includes the operation of acquiring sound characteristic information corresponding to the material of the object based on the above material classification information.

14. In Paragraph 13, A method in which the operation of acquiring the above sound characteristic information is performed using a pre-set material sound characteristic mapping table, and the material sound characteristic mapping table is set using a machine learning model.

15. In Paragraph 11 or 12, A method wherein the first model is trained to receive the image data as input data and output sound boundary characteristic information for at least one object as output data.

Images