Health monitoring method and apparatus based on sound and vibration, and device and storage medium

Abstract

A health monitoring method and apparatus based on sound and vibration, and a device and a storage medium. The method comprises: in response to an information acquisition request for a detection site, acquiring test information collected by an electronic device from the detection site of a subject, wherein the test information comprises sound test information and tremor test information; in response to the test information meeting a predetermined test requirement, acquiring detection information collected by the electronic device from the detection site of the subject, wherein the detection information comprises sound detection information and tremor detection information; and in response to the acquisition time of the detection information exceeding a predetermined threshold value and the detection information meeting a predetermined detection requirement, determining, on the basis of the detection information, a first health prediction result related to the detection site. Thus, the accuracy of an electronic device monitoring subject data can be improved.

Description

Methods, apparatus, devices, and storage media for detecting health based on sound and vibration.

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411554393.0, filed on November 1, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to a method, apparatus, device, and storage medium for detecting health based on sound and vibration. Background Technology

[0004] Health monitoring devices help people monitor their health status. For example, the stethoscope has been an important screening and monitoring tool for many diseases, including cardiopulmonary diseases, for many years, and there are currently various electronic stethoscopes that can assist in recording and analyzing auscultation signals. However, existing electronic stethoscopes are expensive and have barriers to use, only usable in medical institutions or by medical professionals, and cannot be used in a wider range of disease screening scenarios, thus limiting their application scope. Summary of the Invention

[0005] In a first aspect of this disclosure, a method for detecting health based on sound and vibration is provided. The method includes: in response to an information acquisition request for a detection location, acquiring test information collected by an electronic device from the detection location of an object, the test information including sound test information and vibration test information; in response to the test information meeting predetermined test requirements, acquiring detection information collected by the electronic device from the detection location of the object, the detection information including sound detection information and vibration detection information; and in response to the acquisition time of the detection information exceeding a predetermined threshold and the detection information meeting predetermined detection requirements, determining a first health prediction result related to the detection location based on the detection information.

[0006] In a second aspect of this disclosure, an apparatus for detecting health based on sound and vibration is provided. The apparatus includes: a test information acquisition unit configured to acquire test information collected by an electronic device from the detection location of an object, the test information including sound test information and vibration test information, in response to an information acquisition request for a detection location; a detection information acquisition unit configured to acquire detection information collected by the electronic device from the detection location of the object, the detection information including sound detection information and vibration detection information, in response to the test information meeting predetermined test requirements; and a first health prediction result determination unit configured to determine a first health prediction result related to the detection location based on the detection information, in response to the acquisition time of the detection information exceeding a predetermined threshold and the detection information meeting predetermined detection requirements.

[0007] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0008] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0009] In a fifth aspect of this disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0010] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0012] Figure 1 illustrates a schematic diagram of application scenarios according to some embodiments of the present disclosure;

[0013] Figure 2 shows a schematic block diagram of a method for detecting health based on sound and vibration according to some embodiments of the present disclosure;

[0014] Figure 3 shows a flowchart of a method for detecting health based on sound and vibration according to some embodiments of the present disclosure;

[0015] Figure 4 shows an overall schematic diagram of a health detection device based on sound and vibration according to an embodiment of the present disclosure; and

[0016] Figure 5 shows a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0019] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0020] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," a "machine learning network," or simply a "network," and these terms are used interchangeably. A model can also include different types of processing units or networks.

[0021] As used herein, a “unit,” “operation unit,” or “subunit” can consist of any suitable machine learning model or network. As used herein, a set of elements or similar expressions can include one or more such elements. For example, “a set of convolutional units” can include one or more convolutional units.

[0022] As briefly mentioned earlier, traditional electronic stethoscopes suffer from drawbacks such as high operating costs and limited application scope. Furthermore, since microphone units integrated into electronic devices such as smartphones can collect sound wave signals, the use of electronic devices to collect body data is attracting increasing attention.

[0023] Current solutions that use electronic devices as electronic stethoscopes rely solely on microphone units to collect sound wave signals, resulting in low accuracy of the information obtained.

[0024] This disclosure presents a scheme for health detection based on sound and vibration. According to various embodiments of this disclosure, by bringing an electronic device close to or into contact with a detection location, the electronic device can acquire test information of the detection location, including sound test information and vibration test information. In response to the test information meeting predetermined test requirements, detection information is collected from the detection location. The detection information includes sound detection information and vibration detection information. A first health prediction result related to the detection location is determined based on the detection information. In this manner, based on sound wave signals collected by a microphone unit, the electronic device can collect vibration signals at the detection location using an inertial measurement unit, thereby determining the condition of the detection location based on the sound wave signals and vibration signals. This improves the accuracy of the first health prediction result related to the detection location. Furthermore, before collecting the detection information, the electronic device also tests the signal quality at the detection location, which also improves the accuracy of the detection information acquired by the electronic device. By using an inertial measurement unit to collect vibration information, mechanical vibration signals and low-frequency (e.g., below 20 Hz) sound signals of the target part of the object can be more accurately identified. This further improves the accuracy of information recognition.

[0025] Figure 1 illustrates an application scenario 100 according to some embodiments of the present disclosure. As shown in Figure 1, application scenario 100 includes an object 110 and an electronic device 120 coupled to a detection point on the object 110. The object 110 may be a living organism, such as a human or animal. In some other embodiments, the object 110 may also be a device suitable for emitting sound and / or vibration during operation, such as an engine or a speed reducer. The electronic device 120 includes a microphone unit suitable for detecting sound and an inertial measurement unit suitable for detecting mechanical vibration. The electronic device 120 is arranged at the detection position of the object 110, so that the electronic device 120 can collect sound signals at the detection position of the object 110 through the microphone unit and can collect vibration signals at the detection position of the object 110 through the inertial measurement unit. In some embodiments, the inertial measurement unit may include an accelerometer, a gyroscope, etc.

[0026] It should be understood that the electronic device 120 being positioned at the detection location of the object 110 includes the electronic device 120 being close to the detection location of the object 110 or the electronic device 120 being in contact with the detection location of the object 110. For example, in some embodiments, the electronic device 120 can be attached to the surface of the object 110, such as attaching the electronic device 120 to the skin of a living organism. Through contact between the electronic device 120 and the skin, the electronic device 120 can more accurately collect skin tremor information. In some other embodiments, the electronic device 120 can also maintain a predetermined distance from the surface of the object 110. For example, clothing, hair, etc., can be placed between the electronic device 120 and the living organism's skin.

[0027] Electronic device 120 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PDS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 120 may also support any type of user-facing interface (such as "wearable" circuitry).

[0028] In some embodiments, the electronic device 120 can process the collected sound signals and vibration signals based on its own computing power to output prediction results. For example, the electronic device 120 can process the sound signals and vibration signals of an organism to determine the organism's health status.

[0029] In some embodiments, application scenario 100 further includes a server 130. The electronic device 120 can send the acquired sound signal and vibration signal to the server 130, and the server 130 processes the signals. The processing result of the server 130 on the sound signal and vibration signal can be returned to the electronic device 120. The electronic device 120 can also display the processing result to the user.

[0030] Server 130 can be any type of computing system / server 130 capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0031] Figure 2 shows a schematic block diagram 200 of a method for detecting health based on sound and vibration according to some embodiments of the present disclosure. As shown in Figure 2, if a user uses electronic device 120 to obtain a first health prediction result for a detection location, in block 210, the user first needs to open the application in electronic device 120 and select the desired function in the application. In block 220, the user places electronic device 120 at the detection location as prompted in the application. In block 230, electronic device 120 acquires test signals at the detection location; if the acquired test signals meet predetermined requirements, it begins acquiring detection signals. In block 240, during the acquisition of detection signals by electronic device 120, the microphone unit of electronic device 120 acquires sound detection signals. In block 250, the inertial measurement unit acquires vibration detection signals. In block 241, the sound detection signals are denoised and preprocessed. In block 251, the vibration detection signals are denoised and preprocessed. In block 260, multimodal signal processing is performed on the sound detection signals and vibration detection signals, and in block 270, the first health prediction result for the detection location is output. The following will explain in detail how the sound and vibration-based health detection method of this disclosure processes the detection location of object 110.

[0032] Figure 3 shows a flowchart of a method 300 for detecting health based on sound and vibration according to some embodiments of the present disclosure. As shown in Figure 3, method 300 can be implemented by server 130 in the application scenario. Test information and / or detection information (described in detail below) collected by electronic device 120 can be sent to server 130 via a data network, and server 130 processes the test information and / or detection information. The following description will mainly focus on server 130 executing method 300. In addition, in some cases, such as when the computing power of electronic device 120 allows, method 300 can also be implemented by electronic device 120 in application scenario 100. Electronic device 120 can process the test information and / or detection information while collecting it, which will not be described in detail in this disclosure. It should be understood that method 300 may also include additional boxes not shown and / or some (or some) of the boxes shown may be omitted, and the scope of the present disclosure is not limited in this respect.

[0033] In box 310, server 130 responds to an information acquisition request for a detection location by acquiring test information collected by electronic device 120 from the detection location of object 110, including sound test information and vibration test information.

[0034] In some embodiments, while a user sends an information acquisition request for a detection location to the electronic device 120, the electronic device 120 may present introductory information through a display module (e.g., a display or a speaker) based on the user's information acquisition request, thereby assisting the user in moving the electronic device 120 to a detection location close to or near the object 110.

[0035] In some embodiments, in response to an information acquisition request for a target area, the electronic device 120 displays introductory information related to one or more detection locations in the target area of ​​the object 110, the introductory information including at least one of image information, sound information and text information.

[0036] For example, if a user needs to obtain information about the lungs of object 110, the user can send an information retrieval request for the lungs as the target area to electronic device 120. Based on the information retrieval request for the target area, electronic device 120 displays multiple location information related to the target area. For example, electronic device 120 can display introductory information about lung-related detection locations (e.g., the upper left lung, lower left lung, upper right lung, and lower right lung on the front of the object's chest, and the lower left lung and lower right lung on the back of the object) through a display screen or speaker output module. In this way, the user can move electronic device 120 to the detection location according to the prompts from electronic device 120. In some other examples, for examinations of the target area to the heart, the electronic device can prompt the user for heart-related detection locations (e.g., the first aortic valve auscultation area, the second aortic valve auscultation area, the mitral valve area, the tricuspid valve area, and the pulmonary valve area). This facilitates the user in accurately moving the electronic device to the heart-related detection location. By displaying the detection position related to the target area through the electronic device 120, the accuracy of the electronic device 120 being placed at the detection position related to the target area can be improved, thereby improving detection efficiency and the accuracy of outputting the second health prediction result for the target area.

[0037] In some embodiments, the introductory information may include at least one of voice introduction, text introduction, image introduction, and video introduction.

[0038] If the electronic device 120 moves to the detection position of the object 110, it acquires the signal at the detection position based on the information acquisition request for the detection position to determine whether the signal at the detection position meets the detection requirements. For example, the microphone unit of the electronic device 120 can acquire sound test information at the detection position, and the inertial measurement unit of the electronic device 120 can acquire vibration test information at the detection position.

[0039] In some embodiments, if the electronic device 120 moves to the detection position, the user can send an information acquisition request for the detection position to the electronic device 120 through the electronic device 120's input module (e.g., buttons, touch screen, etc.). Upon receiving the information acquisition request for the detection position, the electronic device 120 begins to collect test signals for the detection position. In some embodiments, the electronic device 120 can also determine whether it has moved to the detection position using its own sensors or other modules. For example, the electronic device 120 can determine whether its predetermined position is in contact with the object 110 using its own distance sensor or other similar means, and then begin collecting test signals for the detection position after the electronic device 120 comes into contact with the detection position of the object 110.

[0040] In some embodiments, a user can place the electronic device 120 at the detection location of the object 110 and issue an information acquisition request to the electronic device 120 through its input module (e.g., keyboard, touch screen, etc.). After receiving the information acquisition request for the detection location, the electronic device 120 can collect test information from the detection location.

[0041] In box 320, server 130, in response to test information meeting predetermined test requirements, acquires detection information collected by electronic device 120 from the detection location of object 110, including sound detection information and vibration detection information.

[0042] In some embodiments, the server 130 detects the collected sound test information and vibration test information. If both the sound test information and vibration test information meet predetermined conditions, it indicates that the signal quality at the detection location meets the requirements. Therefore, the server 130 can control the electronic device 120 to continue collecting detection information at that detection location.

[0043] In some embodiments, the server 130 can determine whether the test information meets predetermined conditions based on the signal-to-noise ratio (SNR) of the test information. For example, the server 130 can determine that the sound test information meets predetermined requirements based on the SNR of the sound test information exceeding a first predetermined threshold within a first predetermined time period. As another example, the server 130 can also determine that the vibration test information meets predetermined requirements based on the SNR of the vibration test information exceeding a second predetermined threshold within a second predetermined time period. It should be understood that in some other embodiments, the server 130 can also determine whether the test information meets the requirements based on other appropriate physical parameters, which will not be elaborated upon here.

[0044] In some embodiments, in response to test information indicating that predetermined test requirements are met, server 130 sends a prompt message, which includes at least one of visual, auditory, and tactile prompts. Upon receiving the prompt message, electronic device 120 issues corresponding visual prompts, such as flashing indicator lights or displaying relevant content on a screen; auditory prompts, such as a speaker or buzzer emitting a prompt tone; and tactile prompts, such as vibration from a vibration motor. Through the prompt message, electronic device 120 can inform the user of its current detection location.

[0045] In some other embodiments, if the test information deviates from the predetermined detection requirements, the electronic device 120 can issue an alarm to alert the user that the current detection location is abnormal. For example, if the signal-to-noise ratio of the sound test information acquired by the electronic device 120 is less than a first predetermined threshold, or the duration of the sound test information acquired by the electronic device 120 is less than the first predetermined time, the electronic device 120 can send an alarm to the user. Similarly, if the signal-to-noise ratio of the vibration test information collected by the electronic device 120 is less than a second predetermined threshold, or the duration of the vibration test information collected by the electronic device 120 is less than the second predetermined time, the electronic device 120 can also send an alarm to the user within a second predetermined time (e.g., 5s, 10s, 15s, 20s, etc.).

[0046] Warning information includes at least one of auditory, visual, and tactile information. For example, electronic device 120 can alert the user based on warning information through sound, flashing lights, text or video reminders, vibration, etc., to prompt the user to correct the detection position or the way the electronic device 120 is worn.

[0047] In box 330, server 130 responds to the fact that the acquisition time of detection information exceeds a predetermined threshold and the detection information meets predetermined detection requirements, and determines a first health prediction result related to the detection location based on the detection information.

[0048] Similar to determining whether test information meets predetermined conditions as described above, server 130 can determine whether detection information is usable by judging whether detection information meets predetermined requirements. Specifically, in some embodiments, server 130 can confirm that sound detection information meets predetermined requirements based on the fact that the signal-to-noise ratio of sound detection information exceeds a third predetermined threshold within a third predetermined time period (e.g., 5s, 10s, 15s, 20s, etc.). In some other embodiments, server 130 can also confirm that tremor detection information meets predetermined requirements based on the fact that the signal-to-noise ratio of tremor detection information exceeds a fourth predetermined threshold within a fourth predetermined time period (e.g., 5s, 10s, 15s, 20s, etc.).

[0049] In some embodiments, the time for the electronic device 120 to collect test information and detection information varies depending on the target site. For example, if the target site is the heart, the electronic device 120 may collect sound test information, vibration test information, sound detection information, and vibration detection information for 10 seconds. That is, the first predetermined time, the second predetermined time, the third predetermined time, and the fourth predetermined time can be arbitrarily selected as 10 seconds. If the target site is the lungs, the electronic device 120 may collect sound test information, vibration test information, sound detection information, and vibration detection information for 20 seconds. That is, the first predetermined time, the second predetermined time, the third predetermined time, and the fourth predetermined time can be arbitrarily selected as 20 seconds.

[0050] To ensure the accuracy of signal (test signal and / or detection signal) acquisition during use, the electronic device 120 can also prompt the user under test to remain still. Furthermore, if the user's movements are too large during testing or detection (i.e., the inertial measurement unit of the electronic device 120 detects vibrations exceeding a predetermined range), the electronic device 120 will also remind the user under test to remain still via voice, video, or other means.

[0051] If the detection information collected by the electronic device 120 does not meet the predetermined requirements, such as the signal-to-noise ratio of the sound detection information being less than a predetermined threshold, or the signal-to-noise ratio of the vibration detection information being less than a predetermined threshold, the server 130 can provide warning information to the electronic device 120. Based on the warning information, the electronic device 120 can alert the user through sound, light, text or video reminders, vibration, etc., to prompt the user to correct the detection position or the way the electronic device 120 is worn.

[0052] If the server 130 determines that both the sound detection information and the tremor detection information meet the predetermined requirements, the server 130 can output a first health prediction result related to the detection location based on the sound detection information and the tremor detection information.

[0053] In some embodiments, if the electronic device 120 is positioned in the first aortic valve auscultation area of ​​the subject's chest, the server can determine that a murmur has occurred in the second aortic valve auscultation area based on the detection information from the second aortic valve auscultation area.

[0054] In some embodiments, in response to obtaining detection information, server 130 performs an adjustment operation on the detection information to obtain adjusted detection information. The adjustment operation includes at least one of the following: performing noise reduction processing on the detection information based on a filter, and performing alignment processing on the detection information; and determining whether the adjusted detection information meets predetermined detection requirements.

[0055] In some embodiments, after the electronic device 120 collects test information or detection information, it can upload the test information or detection information to the server 130. The server 130 uses a Butterworth filter to reduce noise in the sound information (sound test information or sound detection information) and the tremor information (tremor test information or tremor detection information). Then, an automatic alignment algorithm is used to align multiple cycles (e.g., multiple respiratory cycles or multiple cardiac cycles) to finally obtain the median signal of one cycle.

[0056] In some embodiments, the process of determining a first health prediction result related to the detection location based on the detection information further includes: the server 130 extracting feature representation information from the detection information. The server 130 applies a cross-attention mechanism and a bimodal fusion mechanism to the feature representation information to determine the first health prediction result related to the detection location.

[0057] In some embodiments, the server 130 can use a convolutional neural network to process the sound detection information and tremor detection information to extract features, and then perform cross-attention and bimodal fusion to form a fusion layer, and finally output a first health prediction result related to the detection location.

[0058] In some embodiments, method 300 further includes server 110 acquiring first health prediction results for multiple detection locations of object 110. Server 110 determines second health prediction results for target parts corresponding to the multiple detection locations based on the first health prediction results for each of the multiple detection locations.

[0059] In some embodiments, a user can send an information acquisition request to an electronic device 120 for a target area, and the electronic device 120 determines multiple detection locations based on the information acquisition request. Thus, the user can send information acquisition requests to the electronic device 120 for multiple detection locations respectively, and the server 110 can determine multiple first health prediction results related to the multiple detection locations in response to the information acquisition requests for the multiple detection locations. Furthermore, the server 110 determines a second health prediction result based on the multiple first health prediction results.

[0060] For example, if electronic device 120 detects the target area of ​​the heart of the subject, the server can determine that the subject's heart function is normal based on the first health prediction results of multiple detection locations related to the heart (e.g., the first aortic valve auscultation area, the second aortic valve auscultation area, the mitral valve area, the tricuspid valve area, the pulmonary valve area, etc.).

[0061] In some embodiments, server 130 can also determine a second health prediction result for the target region based on detection information from multiple detection locations related to the target region. For example, server 130 can directly filter and denoise the detection information from multiple detection locations separately, and then align multiple periods of the multiple detection information using an automatic alignment algorithm to obtain the median signal of one period of the target region. Finally, a convolutional neural network is used to process and extract features, and after cross-attention and dual-modal fusion, a fusion layer is formed, ultimately outputting the second health prediction result related to the target region.

[0062] By employing a microphone unit to acquire sound detection signals and an inertial measurement unit to acquire vibration detection signals, the accuracy of detection information acquired by electronic devices can be improved. In particular, by using an inertial measurement unit to collect vibration information, mechanical vibration signals and low-frequency sound signals from the target object can be more accurately identified, thereby further improving the accuracy of information recognition.

[0063] Figure 4 shows an overall schematic diagram of a health detection device 400 based on sound and vibration according to an embodiment of the present disclosure. As shown in Figure 4, the device 400 includes a test information acquisition unit 410, configured to acquire test information collected by the electronic device 120 from the detection location of the object 110 in response to an information acquisition request for a detection location. The test information includes sound test information and vibration test information.

[0064] The detection information acquisition unit 420 is configured to acquire detection information collected by the electronic device 120 from the detection position of the object 110 in response to the test information meeting the predetermined test requirements. The detection information includes sound detection information and vibration detection information.

[0065] The first health prediction result determination unit 430 is configured to determine a first health prediction result related to the detection location based on the detection information in response to the acquisition time of the detection information exceeding a predetermined threshold and the detection information meeting predetermined detection requirements.

[0066] In some embodiments, the device 400 further includes a second health prediction result determination module, configured to determine a second health prediction result for a target part corresponding to the multiple detection locations based on the first health prediction results for each of the multiple detection locations.

[0067] In some embodiments, the device 400 further includes an introductory information display module, configured to display introductory information related to one or more detection locations in the target part of the object 110 in response to an information acquisition request for the target part, wherein the introductory information includes at least one of image information, sound information and text information.

[0068] In some embodiments, the apparatus 400 further includes a test information comparison module configured to compare test information based on predetermined test requirements, wherein the test information satisfies the predetermined test requirements including: within a first predetermined time period, the signal-to-noise ratio of the sound test information exceeds a first predetermined threshold; and within a second predetermined time period, the signal-to-noise ratio of the vibration test information exceeds a second predetermined threshold.

[0069] In some embodiments, the device 400 further includes a detection information comparison module configured to compare detection information based on predetermined detection requirements, wherein the detection information satisfies the predetermined detection requirements as follows: within a third predetermined time period, the signal-to-noise ratio of the sound detection information exceeds a third predetermined threshold; and within a fourth predetermined time period, the signal-to-noise ratio of the tremor detection information exceeds a fourth predetermined threshold.

[0070] In some embodiments, the device 400 further includes an alarm information providing module, configured to provide alarm information in response to a deviation between test information and predetermined test requirements, or a deviation between detection information and predetermined detection requirements.

[0071] In some embodiments, the first health prediction result determination module is further configured to extract feature representation information of the detection information; and apply a cross-attention mechanism and a bimodal fusion mechanism to the feature representation information to determine a first health prediction result related to the detection location.

[0072] In some embodiments, the apparatus 400 further includes an adjustment module configured to perform an adjustment operation on the detection information in response to acquiring the detection information, thereby obtaining adjusted detection information. The adjustment operation includes at least one of the following: performing noise reduction processing on the detection information based on a filter, and performing alignment processing on the detection information; and determining whether the adjusted detection information meets a predetermined detection requirement.

[0073] In some embodiments, the device 400 further includes a prompting information module configured to send prompting information in response to test information meeting predetermined test requirements. The prompting information includes at least one of visual prompting information, auditory prompting information, and tactile prompting information.

[0074] Figure 5 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. This electronic device 500 may, for example, be used to implement the server 130 shown in Figure 1. It should be understood that the electronic device 500 shown in Figure 5 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein.

[0075] As shown in Figure 5, the electronic device 500 is in the form of a general-purpose electronic device. Components of the electronic device 500 may include, but are not limited to, one or more processors 510 or processing units, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0076] Electronic device 500 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 500.

[0077] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 5, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include a computer program product having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0078] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0079] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0080] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores one or more computer instructions, wherein one or more computer instructions are executed by a processor to implement the methods described above.

[0081] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0082] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0083] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0085] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for detecting health based on sound and vibration, comprising: In response to a request to acquire information about the detection location, test information collected by the electronic device from the detection location of the object is acquired, the test information including sound test information and vibration test information; In response to the test information meeting the predetermined test requirements, the electronic device acquires detection information collected from the detection location of the object, the detection information including sound detection information and vibration detection information; as well as In response to the acquisition time of the detection information exceeding a predetermined threshold and the detection information meeting predetermined detection requirements, a first health prediction result related to the detection location is determined based on the detection information.

2. The method according to claim 1, wherein at least the microphone unit of the electronic device is used to acquire the sound test information and / or the sound detection information.

3. The method according to claim 1, wherein at least the inertial measurement unit of the electronic device is used to acquire the tremor test information and / or the tremor detection information.

4. The method according to claim 1, further comprising: Obtain the first health prediction results for multiple detection locations of the object; Based on the first health prediction results of each of the multiple detection locations, a second health prediction result is determined for the target body part corresponding to the multiple detection locations.

5. The method according to claim 4, further comprising: In response to an information acquisition request for the target region, introductory information related to one or more detection locations in the target region of the object is displayed, the introductory information including at least one of image information, sound information and text information.

6. The method according to claim 1, wherein the test information satisfies predetermined test requirements, including: Within a first predetermined time period, the signal-to-noise ratio of the sound test information exceeds a first predetermined threshold. as well as Within a second predetermined time period, the signal-to-noise ratio of the tremor test information exceeds a second predetermined threshold.

7. The method according to claim 1, wherein the detection information satisfies the predetermined detection requirements, comprising: Within a third predetermined time period, the signal-to-noise ratio of the sound detection information exceeds a third predetermined threshold. as well as Within a fourth predetermined time period, the signal-to-noise ratio of the tremor detection information exceeds a fourth predetermined threshold.

8. The method according to claim 6 or 7, further comprising: An alarm message is provided in response to a deviation between the test information and the predetermined test requirements, or a deviation between the detection information and the predetermined detection requirements.

9. The method of claim 1, wherein determining a first health prediction result related to the detection location based on the detection information comprises: Extract the feature representation information of the detection information; as well as A cross-attention mechanism and a bimodal fusion mechanism are applied to the feature representation information to determine a first health prediction result related to the detection location.

10. The method according to claim 1, further comprising: In response to obtaining the detection information, an adjustment operation is performed on the detection information to obtain adjusted detection information. The adjustment operation includes at least one of the following: noise reduction processing of the detection information based on a filter, and alignment processing of the detection information. as well as Determine whether the adjusted detection information meets the predetermined detection requirements.

11. The method according to claim 1, further comprising: In response to the test information meeting the predetermined test requirements, a prompt message is sent, the prompt message including at least one of visual prompt message, auditory prompt message and tactile prompt message.

12. A device for detecting health based on sound and vibration, comprising: The test information acquisition unit is configured to acquire test information collected by the electronic device from the detection location of the object in response to an information acquisition request for the detection location, the test information including sound test information and vibration test information; The detection information acquisition unit is configured to acquire detection information collected by the electronic device from the detection location of the object in response to the test information meeting a predetermined test requirement, the detection information including sound detection information and vibration detection information; The first health prediction result determination unit is configured to determine a first health prediction result related to the detection location based on the detection information in response to the acquisition time of the detection information exceeding a predetermined threshold and the detection information meeting predetermined detection requirements.

13. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 11.

15. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 11.

Images