Electronic device for acquiring user input data by sensing frictional sound and method for operating same

Abstract

An electronic device and a method for operating same are disclosed. An electronic device for acquiring user input data by sensing a sound propagated through a medium, according to the present disclosure, may comprise: a pad which is in contact with the surface of a medium through which a sound propagates; a sound sensor which includes two or more microphones, and which is disposed inside the pad or in one area of the surface thereof to sense sounds propagated through the medium and the pad; and a processor connected to the sound sensor to acquire user input data corresponding to sounds on the basis of sensing information acquired from the sound sensor, wherein the sounds may be frictional sounds related to the surface of the medium.

Description

Electronic device for acquiring user input data by sensing friction sound and method of operation thereof

[0001] The present disclosure relates to a user input data processing technology using frictional sound, and more specifically, to an electronic device that acquires input data by sensing frictional sound generated on the surface of a medium.

[0002] Friction sounds are distinctive noises generated when an object moves in contact with a surface, appearing in various forms depending on the surface condition of the medium, the intensity of friction, and the characteristics of the contacted object. By sensing these friction sounds and utilizing them as input signals, they can be applied to replace or complement existing touch-based input devices. For example, it is possible to develop technology that recognizes specific movements by analyzing friction sounds generated from glass, metal, or plastic surfaces.

[0003] However, in the process of processing input data based on frictional sounds, problems may arise where ambient noise interferes or the material and environmental conditions of the medium degrade input reliability.

[0004] In particular, when the frequency band of frictional sound overlaps with specific environmental noise, it may be difficult to extract the desired input data. To overcome these limitations, high-sensitivity sensors and sophisticated signal processing algorithms may be required.

[0005] An electronic device for acquiring user input data by sensing sound transmitted through a medium according to one embodiment of the present disclosure comprises: a pad in contact with the surface of the medium through which the sound is transmitted; a sound sensor comprising two or more microphones and disposed in an area of ​​the interior or surface of the pad to sense the sound transmitted through the medium and the pad; and a processor connected to the sound sensor and acquiring user input data corresponding to the sound based on sensing information acquired from the sound sensor, wherein the sound may be a friction sound on the surface of the medium.

[0006] A method of operation of an electronic device for acquiring user input data by sensing sound propagated through a medium according to one embodiment of the present disclosure may include: acquiring sensing information for a sound corresponding to a friction sound on the surface of the medium; acquiring user input data corresponding to the friction sound by inputting the acquired sensing information into a sound pattern recognition model; and performing a preset operation based on a control command corresponding to the acquired user input data.

[0007] An electronic device for acquiring user input data by sensing sound transmitted through a medium according to one embodiment of the present disclosure comprises: a pad in contact with the surface of the medium through which the sound is transmitted; a sound sensor disposed on a portion of the pad to sense the sound transmitted through the medium and the pad; and a processor connected to the sound sensor and acquiring user input data corresponding to the sound based on sensing information acquired from the sound sensor, wherein the sound may be a friction sound on the surface of the medium.

[0008] The above processor can acquire user input data corresponding to the sound by inputting the sensing information into a sound pattern recognition model.

[0009] The pad may be made of a wood-based material capable of transmitting sound propagated from the medium through the surface of the pad to a sound sensor placed on one area of ​​the pad.

[0010] The sound sensor may be placed on a region of the opposite side of the surface of the pad that the medium contacts.

[0011] The sound sensor is positioned at the center of the pad surface, and the center may be a point where the deviation in distance from the edge of the pad surface is minimized.

[0012] The processor can identify a control command corresponding to the user input data and perform a preset operation based on the identified control command.

[0013] The above pad may consist of at least one unit module.

[0014] A method of operation of an electronic device for acquiring user input data by sensing sound propagated through a medium according to one embodiment of the present disclosure may include: acquiring sensing information for a sound corresponding to a friction sound on the surface of the medium; acquiring user input data corresponding to the friction sound by inputting the acquired sensing information into a sound pattern recognition model; and performing a preset operation based on a control command corresponding to the acquired user input data.

[0015] It may be a non-transient computer-readable recording medium storing at least one instruction that is executed by a processor of an electronic device according to one embodiment of the present disclosure to perform a method of operation of the electronic device.

[0016] Through the present invention, user input data can be accurately acquired based on frictional sounds generated on the surface of a medium. This enables the use of mediums made of various materials as input devices and overcomes the limitations of existing input technologies. In particular, interference with ambient noise is effectively suppressed, thereby enhancing the reliability and accuracy of input data. These effects can be utilized in various application fields, such as smart devices, home appliances, and interactive displays, contributing to improved user experience and expanded diversity of input methods.

[0017] Aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description with reference to the accompanying drawings.

[0018] FIG. 1 is a perspective view illustrating the process of generating user friction sound on the surface of a medium located on a pad and an electronic device that senses friction sound to acquire user input data according to one embodiment of the present disclosure.

[0019] FIG. 2 is a plan view of a pad of an electronic device that acquires user input data by sensing frictional sound according to one embodiment of the present disclosure.

[0020] FIG. 3a is a plan view of a pad and a sensor constituting an electronic device according to one embodiment of the present disclosure.

[0021] FIG. 3b is a side view of a pad and a sensor constituting an electronic device according to one embodiment of the present disclosure.

[0022] FIG. 3c is a plan view of a medium, a pad, and a sensor positioned on a pad surface according to one embodiment of the present disclosure.

[0023] FIG. 3d is a side view of a medium, a pad, and a sensor positioned on a pad surface according to one embodiment of the present disclosure.

[0024] FIG. 4a is a plan view of a pad and a sensor constituting an electronic device according to one embodiment of the present disclosure.

[0025] FIG. 4b is a side view of a pad and a sensor constituting an electronic device according to one embodiment of the present disclosure.

[0026] FIG. 4c is a plan view of a medium, a pad, and a sensor positioned on a pad surface according to one embodiment of the present disclosure.

[0027] FIG. 4d is a side view of a medium, a pad, and a sensor positioned on a pad surface according to one embodiment of the present disclosure.

[0028] FIG. 5 is a block diagram illustrating the configuration of an electronic device that senses frictional sound and acquires user input data according to one embodiment of the present disclosure.

[0029] FIG. 6 is a flowchart illustrating the operation of an electronic device that senses frictional sound to acquire user input data according to one embodiment of the present disclosure.

[0030] FIG. 7 is a cross-sectional view illustrating an operation in which a user friction sound on the surface of a medium located on a pad, according to one embodiment of the present disclosure, propagates through the medium and the pad to reach a sound sensor and the sound sensor acquires sensing information.

[0031] FIG. 8a is a plan view illustrating the process of user input being made on the surface of a medium according to one embodiment of the present disclosure.

[0032] FIG. 8b is a plan view illustrating the process of user input being made on the surface of a medium according to one embodiment of the present disclosure.

[0033] FIG. 8c is a plan view illustrating the process of user input being made on the surface of a medium according to one embodiment of the present disclosure.

[0034] FIG. 9 is a diagram illustrating the principle of acquiring user input data based on frequency analysis of user friction sound according to one embodiment of the present disclosure.

[0035] FIG. 10 is a drawing for explaining the operation of an electronic device corresponding to various user inputs according to one embodiment of the present disclosure.

[0036] FIG. 11 is a flowchart illustrating the operation related to the preprocessing of sensing information and a sound pattern recognition model of an electronic device that acquires user input data by sensing frictional sound according to one embodiment of the present disclosure.

[0037] FIG. 12 is a flowchart illustrating the operation of an electronic device that acquires user input data by sensing frictional sound according to one embodiment of the present disclosure, acquiring user input data corresponding to the directionality of the sound.

[0038] The embodiments described herein are subject to various modifications and may have various forms; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals may be used for similar components.

[0039] In describing the present disclosure, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the present disclosure, such detailed description is omitted.

[0040] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concept of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to fully convey the technical concept of the present disclosure to those skilled in the art.

[0041] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of the rights. The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0042] In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0043] In the present disclosure, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0044] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0045] Where it is stated that a component (e.g., Component 1) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., Component 2), it should be understood that the component may be directly connected to the other component or connected through the other component (e.g., Component 3).

[0046] On the other hand, when it is stated that a certain component (e.g., a first component) is "directly connected" or "directly coupled" to another component (e.g., a second component), it may be understood that no other component (e.g., a third component) exists between the certain component and the other component.

[0047] As used in this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware.

[0048] Instead, in some situations, the expression “device configured to do something” may mean that the device is “capable of doing something” together with other devices or components. For example, the phrase “processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing those operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or application processor) capable of performing those operations by executing one or more software programs stored in a memory device.

[0049] In the embodiments, a 'module' or 'part' performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of 'modules' or a plurality of 'parts' may be integrated into at least one module and implemented by at least one processor, except for the 'module' or 'part' that needs to be implemented in specific hardware.

[0050] Meanwhile, the various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0051] Hereinafter, embodiments according to the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement them.

[0052] FIG. 1 is a perspective view illustrating the process of generating user friction sound on the surface of a medium located on a pad and an electronic device that senses friction sound to acquire user input data according to one embodiment of the present disclosure.

[0053] Referring to FIG. 1, the electronic device can perform the function of acquiring user input data by sensing sound transmitted through a medium (2). The pad (10) is in contact with the surface of the medium (2) through which the sound is transmitted.

[0054] Here, the medium (2) can be various objects capable of propagating sound, and may be objects made of various materials such as ceramics, books, ornaments, structures, etc., but is not limited thereto.

[0055] The pad (10) is a part that is in contact with the medium (2) and can detect the sound when the sound is propagated.

[0056] The pad (10) may be made of a wood-based material capable of transmitting sound transmitted from a medium (2) through the surface of the pad (10) to a sound sensor (110) placed on one area of ​​the pad (10).

[0057] Wood-based materials may have the characteristic of effectively transmitting sound vibrations generated in the medium (2). For example, there may be high-density wood with acoustic properties, lightweight plywood, and reinforced wood suitable for vibration propagation, but are not limited thereto.

[0058] However, the material of the pad (10) is not necessarily a wood-based material, and can be a variety of materials that can serve as a sound propagation medium (2), such as metal or ceramic.

[0059] The pad (10) may be composed of at least one unit module. A unit module may refer to a part having a minimum function that can operate independently as a component of the pad (10). For example, there may be an individual sensor module for detecting sound propagation, a structural module for effectively transmitting sound in a specific area, or a pad (10) configuration that can be expanded into multiple modules, but is not limited thereto.

[0060] The pad (10) composed of these unit modules can provide flexibility and expandability by allowing the user to add or replace modules as needed.

[0061] The pad (10) can be maintained spaced apart from the ground by having at least one support (20) attached to its bottom surface. The support (20) may be placed in the edge area of ​​the pad (10) as shown in FIG. 1, but is not limited thereto.

[0062] A sound sensor (110) is placed on a portion of the pad (10) to sense sound propagating through the medium (2) and the pad (10).

[0063] The sound may be a friction sound on the surface of the medium (2). The friction sound is a sound produced by actions such as scratching or rubbing the surface of the medium (2), and user input can be detected through the sound. For example, there may be a sound of scratching the surface of the medium (2) with a fingernail (1), a sound of rubbing the surface with the palm of the hand (1), or a sound produced by touching the medium (2) in a specific pattern, but is not limited thereto.

[0064] FIG. 2 is a plan view illustrating an electronic device that senses frictional sound and acquires user input data according to one embodiment of the present disclosure.

[0065] Referring to FIG. 2, the sound sensor (110) may be placed on a portion of the surface opposite to the surface where the medium (2) of the pad (10) contacts (or the bottom surface, in which case the surface where the medium (2) contacts the pad (10) may be the top surface of the pad (10), and may also be located inside rather than on the surface of the pad (10). In this case, the pad (10) may be in a form that surrounds or contains the sound sensor (110) inside.

[0066] The arrangement of the sound sensor (110) as described above may mean a method of installing the sound sensor (110) so as to effectively detect vibrations occurring at the contact surface with the medium (2).

[0067] According to various embodiments, there may be a sensor installed at the center of the bottom surface of the pad (10), a sensor attached to the edge portion of the pad (10), a sensor positioned to optimally detect vibration of the medium (2), etc., but are not limited thereto.

[0068] The sound sensor (110) may be positioned at the center of the surface of the pad (10). By positioning the sound sensor (110) at the center, it has the advantage of being able to uniformly detect sound vibrations occurring throughout the entire pad (10). For example, by positioning it at the center of the surface of the pad (10), uniform sensitivity can be maintained in all directions in which sound propagates, and there may be a sensor positioned at the center to optimally detect sound propagation, but it is not limited thereto.

[0069] The center may refer to the point where the deviation in distance from the edge of the pad (10) surface is minimized.

[0070] This point is a location where the distance from the corners or edges of the pad (10) is maintained uniformly, and where even sensitivity can be secured across the entire pad (10). For example, in the case of a square pad (10), it may be the intersection of diagonals, in the case of a circular pad (10), the center of the circle, or in the case of an irregular shape, the point with the smallest average distance, but is not limited thereto.

[0071] FIG. 3a is a plan view of a pad and a sensor constituting an electronic device according to one embodiment of the present disclosure.

[0072] FIG. 3b is a side view of a pad and a sensor constituting an electronic device according to one embodiment of the present disclosure.

[0073] Referring to FIGS. 3a and 3b, the cross-section of the pad (10) may be circular, and a square-shaped sound sensor (110) may be placed on the bottom surface or inside the pad (10) with a circular cross-section. Two microphones (111) may be placed in the sound sensor (110), but are not limited to the above shape.

[0074] FIG. 3c is a plan view of a medium, a pad, and a sensor positioned on a pad surface according to one embodiment of the present disclosure.

[0075] FIG. 3d is a side view of a medium, a pad, and a sensor positioned on a pad surface according to one embodiment of the present disclosure.

[0076] Referring to FIGS. 3c and 3d, a rectangular mediator (2) can be placed on a pad (10) with a circular cross section.

[0077] FIG. 4a is a plan view of a pad and a sensor constituting an electronic device according to one embodiment of the present disclosure.

[0078] FIG. 4b is a side view of a pad and a sensor constituting an electronic device according to one embodiment of the present disclosure.

[0079] Referring to FIGS. 4a and 4b, the cross-section of the pad (10) may be square, and a square-shaped sound sensor (110) may be placed on the bottom surface or inside the pad (10) with a square cross-section. Two microphones (111) may be placed in the sound sensor (110), but are not limited to the above form.

[0080] FIG. 4c is a plan view of a medium, a pad, and a sensor positioned on a pad surface according to one embodiment of the present disclosure.

[0081] FIG. 4d is a side view of a medium, a pad, and a sensor positioned on a pad surface according to one embodiment of the present disclosure.

[0082] Referring to FIGS. 4c and 4d, a cylindrical medium (2) with a circular cross-section can be placed on a pad (10) with a square cross-section.

[0083] FIG. 5 is a block diagram illustrating the configuration of an electronic device that senses frictional sound and acquires user input data according to one embodiment of the present disclosure.

[0084] Referring to FIG. 5, the electronic device may include a processor (140) including a sound sensor (110), a display (120), a communication interface (130), and a sound pattern recognition module (141), but the device configuration is not limited thereto and may include additional configurations or omit some configurations.

[0085] The processor (140) is connected to the sound sensor (110) and can obtain user input data corresponding to the sound based on the sensing information obtained from the sound sensor (110).

[0086] The sound sensor (110) may refer to a device that detects sound vibrations propagated through a medium (2) and converts them into an electrical signal. For example, it may include a piezoelectric element microphone (111), a vibration detection sensor, an acoustic transducer, etc., but is not limited thereto.

[0087] Additionally, the sound sensor (110) may include one or more microphones (111). Here, the microphone (111) may refer to a module that acquires sound and converts it into an electrical signal, and may be a condenser microphone (111), a ribbon microphone (111), a moving coil microphone (111), a piezoelectric element microphone (111), a carbon microphone (111), or a MEMS (Micro Electro Mechanical System) microphone (111). Furthermore, it may be implemented in omnidirectional, bidirectional, unidirectional, subcardioid, supercardioid, or hypercardioid modes.

[0088] The display (120) may include various types of display panels, such as an LCD (Liquid Crystal Display) panel, an OLED (Organic Light Emitting Diodes) panel, an AM-OLED (Active-Matrix Organic Light-Emitting Diode), an LcoS (Liquid Crystal on Silicon), a QLED (Quantum dot Light-Emitting Diode) and DLP (Digital Light Processing), a PDP (Plasma Display Panel) panel, an inorganic LED panel, and a Micro LED panel, but is not limited thereto. Meanwhile, the display (120) may form a touchscreen together with a touch panel and may be made of a flexible panel.

[0089] The display (120) can be implemented in a 2D shape such as a square or a rectangle, but is not limited thereto and can be implemented in various shapes such as a circle, a polygon, or a 3D shape.

[0090] The display (120) may be placed in one area of ​​the surface of the electronic device, but is not limited thereto, and may be implemented as a three-dimensional hologram projected in three-dimensional space or as a projection projected on a two-dimensional plane.

[0091] The display (120) may be included as a component of the electronic device, but is not limited thereto, and a separately provided display (120) device may be connected wirelessly / wiredly to the electronic device through a communication interface (130) or an input / output interface to output an image, video, or GUI according to a signal from the processor (140). In this case, the processor (140) may establish a wireless / wired connection with the display (120) device through the communication interface (130) or an input / output interface to transmit a signal for outputting an image, video, or GUI.

[0092] The processor (140) can provide user input data corresponding to the sensing information to the user by outputting it through the display (120).

[0093] For example, the processor (140) can visualize characters, alphabets, numbers, shapes, other control commands, etc., drawn by the user by rubbing them on the surface of the medium (2) with a hand (1) or a tool, and output them to the user through the display (120).

[0094] The communication interface (130) may include a wireless communication interface (130), a wired communication interface (130), or an input interface. The wireless communication interface (130) may communicate with various external devices using wireless communication technology or mobile communication technology. Such wireless communication technologies may include, for example, Bluetooth, Bluetooth Low Energy, CAN communication, Wi-Fi, Wi-Fi Direct, ultrawide band (UWB), Zigbee, infrared data association (IrDA), or near field communication (NFC), and mobile communication technologies may include 3GPP, Wi-Max, LTE (Long Term Evolution), 5G, etc.

[0095] The wireless communication interface (130) can be implemented using an antenna, a communication chip, a substrate, etc., capable of transmitting electromagnetic waves to the outside or receiving electromagnetic waves transmitted from the outside.

[0096] The wired communication interface (130) can communicate with various devices based on a wired communication network. Here, the wired communication network can be implemented using physical cables, such as, for example, a pair cable, a coaxial cable, a fiber optic cable, or an Ethernet cable.

[0097] Depending on the embodiment, either the wireless communication interface (130) and the wired communication interface (130) may be omitted. Accordingly, the electronic device may include only the wireless communication interface (130) or only the wired communication interface (130). In addition, the electronic device may be equipped with an integrated communication interface (130) that supports both wireless connection via the wireless communication interface (130) and wired connection via the wired communication interface (130).

[0098] The electronic device is not limited to having one communication interface (130) that performs a communication connection in one manner, but may include multiple communication interfaces (130) that perform communication connections in multiple manners.

[0099] The processor (140) can perform a communication connection with an external device, server, etc. through a communication interface (130) to transmit sensing information corresponding to a sound propagated through the medium (2) described above, user input data corresponding to the sensing information, or control commands corresponding to the user input data.

[0100] The processor (140) controls the overall operation of the electronic device. Specifically, the processor (140) is connected to the configuration of the electronic device including memory as described above, and can control the overall operation of the electronic device by executing at least one instruction stored in the memory as described above. In particular, the processor (140) can be implemented as a single processor (140) as well as as a plurality of processors (140).

[0101] The processor (140) may be implemented in various ways. For example, one or more processors (140) may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator. One or more processors (140) may control one or any combination of other components of an electronic device and may perform operations or data processing related to communication. One or more processors (140) may execute one or more programs or instructions stored in memory. For example, one or more processors (140) may perform a method according to one embodiment of the present disclosure by executing one or more instructions stored in memory.

[0102] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one processor (140) or by a plurality of processors (140). For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor (140), or the first operation and the second operation may be performed by a first processor (140) (e.g., a general-purpose processor (140)) and the third operation may be performed by a second processor (140) (e.g., an artificial intelligence dedicated processor (140)).

[0103] One or more processors (140) may be implemented as a single-core processor (140) including one core, or as one or more multi-core processors (140) including multiple cores (e.g., homogeneous multi-core or heterogeneous multi-core). When one or more processors (140) are implemented as multi-core processors (140), each of the multiple cores included in the multi-core processor (140) may include internal memory of the processor (140), such as on-chip memory, and a common cache shared by the multiple cores may be included in the multi-core processor (140). Additionally, each of the multiple cores (or some of the multiple cores) included in the multi-core processor (140) may independently read and execute program instructions for implementing a method according to one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one embodiment of the present disclosure.

[0104] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in the multi-core processor (140), or by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor (140), or the first operation and the second operation may be performed by a first core included in the multi-core processor (140), and the third operation may be performed by a second core included in the multi-core processor (140).

[0105] In the embodiments of the present disclosure, the processor (140) may mean a system-on-chip (SoC) in which one or more processors (140) and other electronic components are integrated, a single-core processor (140), a multi-core processor (140), or a core included in a single-core processor (140) or a multi-core processor (140), wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator or machine learning accelerator, etc., but the embodiments of the present disclosure are not limited thereto.

[0106] FIG. 6 is a flowchart illustrating the operation of an electronic device that acquires user input data by sensing frictional sound according to one embodiment of the present disclosure.

[0107] Referring to FIG. 6, the processor (140) can obtain sensing information for a sound corresponding to a friction sound on the surface of the medium (2) (S410).

[0108] FIG. 7 is a cross-sectional view illustrating an operation in which a user friction sound on the surface of a medium (2) located on a pad (10) according to one embodiment of the present disclosure propagates through the medium (2) and the pad (10) to reach a sound sensor (110) and the sound sensor (110) acquires sensing information.

[0109] Referring to FIG. 7, the medium (2) can be various objects capable of propagating sound, and may be objects made of various materials such as ceramics, books, ornaments, structures, etc., but is not limited thereto.

[0110] When a person rubs, rubs, or draws a specific character, alphabet, shape, number, etc. on a surface area of ​​the medium (2) with their hand (1) or other tool, frictional sound is transmitted through the medium (2) to the pad (10), and frictional sound is transmitted through the pad (10) in the form of a shading wave (900) and a surface wave (800) to a sound sensor (110) located on the opposite side. The processor (140) can obtain sensing information corresponding to the frictional sound through the sound sensor (110).

[0111] The processor (140) can input the acquired sensing information into a sound pattern recognition model (or sound pattern recognition module (141)) to acquire user input data corresponding to the friction sound (S420).

[0112] A sound pattern recognition model can refer to an algorithm that analyzes the pattern of an input sound and converts it into specific user input. Examples include, but are not limited to, speech recognition algorithms, pattern recognition models that respond to specific frictional sounds, and machine learning-based sound analysis systems.

[0113] Sound pattern recognition models can extract the temporal characteristics of sound signals using convolutional neural networks and recurrent neural networks (RNNs), but are not limited to these.

[0114] Furthermore, when a sound pattern recognition model is a transformer-based model, it serves to utilize sound signals in natural language processing by primarily extracting features from the sound signal and converting them into a sequence. Transformer models employ a self-attention mechanism to grasp the overall relationships within the input sound signal, thereby enabling language models to effectively recognize and understand the meaning associated with the sound.

[0115] When the sound pattern recognition model is a CNN-based model, the sound is converted into a frequency-time spectrogram and treated like a two-dimensional image, through which features of the sound signal are extracted and analyzed.

[0116] User input data may refer to information entered by the user by generating a friction sound on the surface of the medium (2). For example, there may be acts such as scratching or tapping the surface of the medium (2), commands entered through friction of a specific pattern, gesture input using sound, number input, alphabet input, etc., but are not limited thereto.

[0117] FIG. 8a is a plan view illustrating the process of user input being made on the surface of a medium according to one embodiment of the present disclosure.

[0118] Referring to FIG. 8a, the user can input the number 2 by rubbing the surface of the medium (2) placed on the pad (10) with a finger.

[0119] FIG. 8b is a plan view illustrating the process of user input being made on the surface of a medium according to one embodiment of the present disclosure.

[0120] Referring to FIG. 8b, the user can input a star shape by rubbing the surface of the medium (2) placed on the pad (10) with their finger.

[0121] FIG. 8c is a plan view illustrating the process of user input being made on the surface of a medium according to one embodiment of the present disclosure.

[0122] Referring to FIG. 8c, the user can input the alphabet a by rubbing the surface of the medium (2) placed on the pad (10) with a finger.

[0123] The processor (140) can perform a preset operation based on a control command corresponding to the acquired user input data (S430).

[0124] The processor (140) can identify control commands corresponding to user input data. A control command may refer to a specific action or instruction to be performed according to user input data. For example, there may be a command to turn on the device according to a specific pattern of scratching the medium (2), a volume control command corresponding to a specific friction sound, or a command to turn the light on or off according to the user's touch pattern, but are not limited thereto.

[0125] FIG. 9 is a diagram illustrating the principle of acquiring user input data based on frequency analysis of user friction sound according to one embodiment of the present disclosure.

[0126] Referring to FIG. 9, the processor (140) can obtain a user friction sound waveform and spectrogram (S1-1, S2-1) for cases where a user friction sound input is made directly on the surface of the pad (10) (S1) and where a user friction sound input is made on the surface of the medium (2) placed on the pad (10) (S2), and can identify the sound and the difference through frequency analysis (S3) for each.

[0127] The processor (140) can perform a preset operation based on an identified control command. The preset operation may refer to a specific task that the processor (140) must execute according to the control command. For example, it may include power control of a home appliance, operation of a media player, or operation of enabling or disabling a specific function, but is not limited thereto.

[0128] FIG. 10 is a drawing for explaining the operation of an electronic device corresponding to various user inputs according to one embodiment of the present disclosure.

[0129] Referring to FIG. 10, when the processor (140) identifies that the number 2 has been entered by user friction sound, it can obtain a control command to turn on the light and transmit it to the lighting device, and when it identifies that a star shape has been entered, it can obtain a control command to open or close the electric curtain and transmit it to the curtain control device. However, the form of user input for each command is not limited to the above, and various forms of user input may be made, and an electronic device control command corresponding thereto may be obtained and transmitted to each control device so that the corresponding control device performs an operation corresponding to the user input.

[0130] According to various embodiments, the processor (140) can analyze sound data detected by a sound sensor to interpret a specific gesture or input intention of the user. For example, when the user scratches or touches a medium with a finger, the processor can detect the friction sound generated by such action and analyze the characteristics of the sound to interpret a command or gesture that the user intends to input.

[0131] The processor (140) can also detect frictional sounds of a specific frequency band to distinguish commands intended by the user. For example, low-frequency frictional sounds may indicate a selection command, and high-frequency frictional sounds may indicate an execution command. Through this, the processor (140) can clearly distinguish various commands intended by the user.

[0132] For example, a low-frequency friction sound generated by a user scratching the surface of a medium at a speed below a threshold speed indicates a selection command, and the processor can recognize this to select or highlight a specific item on the screen. On the other hand, if a user scratches the surface of a medium at a speed above a threshold speed to generate a high-frequency friction sound, the processor (140) interprets this as an execution command and executes the item selected by the user or performs a related task.

[0133] The processor (140) can also filter the characteristics of the sound signal to separate the ambient noise from the user's input signal. For example, the processor (140) analyzes the frequency and amplitude of the input signal to remove ambient noise in a specific band, thereby enabling the user's input signal to be interpreted more accurately. Through this filtering process, the reliability of sound detection can be improved.

[0134] FIG. 11 is a flowchart illustrating the operation related to the preprocessing of sensing information and a sound pattern recognition model of an electronic device that acquires user input data by sensing frictional sound according to one embodiment of the present disclosure.

[0135] Referring to FIG. 11, the processor (140) can obtain sensing information for a sound corresponding to a friction sound on the surface of the medium (2) (S510).

[0136] The processor (140) can extract friction sound data included in the acquired sensing information (S520).

[0137] Friction sound data may refer to information about friction sound generated on the surface of the medium (2). For example, it may include the amplitude of the sound, frequency characteristics, duration of the sound, etc., but is not limited thereto.

[0138] The processor (140) can preprocess the acquired sensing information to extract the acquired preprocessed frictional sound data (S530).

[0139] Preprocessed fricative data refers to data from which unnecessary parts have been removed or transformed into the required form before analyzing a sound signal. Examples include, but are not limited to, sound signals with noise removed, data with specific frequency bands emphasized, and fricative data with the time axis normalized.

[0140] The processor (140) can input the frictional sound data and the preprocessed frictional sound data into a sound pattern recognition model to obtain user input data corresponding to the frictional sound (S540).

[0141] A sound pattern recognition model can refer to an algorithm that analyzes the pattern of input fricative sounds and converts them into specific user input. For example, examples may include, but are not limited to, gesture recognition models that respond to specific fricative sound patterns, machine learning-based sound analysis systems, and recognition models that analyze the frequency and amplitude of specific sounds to convert them into commands.

[0142] The processor (140) can perform a preset operation based on a control command corresponding to the acquired user input data (S550).

[0143] The pre-set operation may refer to a specific task that the processor (140) must execute according to a control command. For example, it may include power control of home appliances, operation of media playback devices, or operation of enabling or disabling specific functions, but is not limited thereto.

[0144] FIG. 12 is a flowchart illustrating the operation of an electronic device that acquires user input data by sensing frictional sound according to one embodiment of the present disclosure, acquiring user input data corresponding to the directionality of the sound.

[0145] Referring to FIG. 12, the processor (140) can obtain sensing information for a sound corresponding to a friction sound on the surface of the medium (2) (S610).

[0146] The processor (140) can analyze changes in the waveform of the sound over time based on the sensing information and identify the direction of the sound (S620).

[0147] Sound directionality refers to the location where the sound originated or the direction of its movement, which allows for the identification of the user's intention or gesture. For example, this may include, but is not limited to, the direction of sound movement following a scratching motion in a specific direction, identifying the location of the sound source, and comparing sound patterns from multiple directions.

[0148] The processor (140) can obtain user input data corresponding to the directionality of the identified sound (S630).

[0149] User input data is data that reflects the intention or command conveyed by the user through sound, and various inputs can be interpreted through the directional information of the sound. For example, there may be inputs that activate a specific function through a gesture of rubbing the hand (1) in a specific direction, methods that distinguish inputs through the start and end points of the sound, and methods that recognize complex commands through a pattern of rubbing in multiple directions, but are not limited to these.

[0150] The processor (140) can perform a preset operation based on a control command corresponding to the acquired user input data (S640).

[0151] The pre-set operation may refer to a specific task that the processor (140) must perform according to user input. For example, there may be various operations such as turning the power on / off of a home appliance, controlling media playback, or switching to a specific mode, but are not limited thereto.

[0152] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0153] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. An electronic device that acquires user input data by sensing sound transmitted through a medium, A pad that comes into contact with the surface of the medium through which sound propagates; A sound sensor comprising two or more microphones and disposed in an area of ​​the interior or surface of the pad to sense sound propagated through the medium and the pad; and A processor connected to the sound sensor and acquiring user input data corresponding to the sound based on sensing information acquired from the sound sensor; The above sound is, An electronic device that is a friction sound on the surface of the above medium.

2. In Paragraph 1, The above processor is, An electronic device that inputs the sensing information into a sound pattern recognition model to obtain user input data corresponding to the sound.

3. In Paragraph 1, The above pad is, An electronic device made of a material capable of transmitting sound propagated from the above medium through the surface of the pad to a sound sensor disposed on a region of the pad.

4. In Paragraph 1, The above sound sensor is, An electronic device disposed on a surface area of ​​the pad or inside the pad.

5. In Paragraph 4, The above sound sensor is, It is positioned at the center of the pad surface mentioned above, and The above central part is, An electronic device that is a point where the deviation in distance from the edge of the pad surface is minimized.

6. In Paragraph 1, The above processor is, Identifying a control command corresponding to the user input data generated based on interaction with the surface of the medium, and An electronic device that performs a preset operation based on the above-identified control command.

7. In Paragraph 1, The above pad is, An electronic device comprising at least one unit module.

8. A method of operation of an electronic device for acquiring user input data by sensing sound transmitted through a medium, An operation to acquire sensing information regarding a sound corresponding to a friction sound on the surface of a medium; The operation of inputting the above-mentioned acquired sensing information into a sound pattern recognition model to acquire user input data corresponding to the friction sound; and A method of operation comprising: an operation that performs a preset operation based on a control command corresponding to the user input data obtained above.

9. A non-transient computer-readable recording medium storing at least one instruction that is executed by a processor of an electronic device to cause said electronic device to perform the method of operation of claim 8.

Images