Medium-free holographic device for musical instruments and control method therefor

Abstract

The embodiments of the present application relate to the technical field of medium-free music-score-information display. Provided are a medium-free holographic device for musical instruments and a control method therefor. The medium-free holographic device for musical instruments comprises: a medium-free holographic module, which is configured to generate and display a medium-free holographic musical score image; a first sensor, which is configured to collect musical-instrument action information and / or musical-instrument sound information during the performance of a musical instrument; and a control module, which is signally connected to the first sensor and the medium-free holographic module and is configured to determine musical score information on the basis of the musical-instrument action information and / or the musical-instrument sound information, and adjust the medium-free holographic musical score image on the basis of the musical score information. By means of the present application, the problem of the complexity of music score operations being high is solved, thereby improving the effect of performance and reducing errors in terms of performance.

Description

A medium-free holographic device for musical instruments and its control method

[0001] This application claims priority to Chinese Patent Application No. 202510188590.3, filed on February 20, 2025, entitled "A mediumless holographic device for musical instruments and its control method", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of media-free display of musical scores, and more specifically, to a media-free holographic device for musical instruments and its control method. Background Technology

[0003] In the field of musical instrument performance, sheet music is a crucial reference for performers. Traditionally, performers manually turn the pages of paper sheet music during a performance. This not only easily distracts the performer, affecting the continuity and accuracy of the playing, but can also create significant psychological pressure during complex movements or fast-paced performances. Furthermore, for large-scale performances or professional settings, specialized staff are often required to assist with page turning, which increases labor costs and can lead to interruptions due to operational errors, thus impacting the performance's effectiveness.

[0004] With the development of technology, electronic sheet music display devices have gradually been introduced into musical instrument performance. These devices display sheet music on a screen, enabling automatic page turning and adjustments, thus addressing some of the shortcomings of traditional paper sheet music. However, most existing electronic sheet music display devices rely on touchscreen operation or external control devices, requiring performers to still be distracted during performance. Furthermore, the devices typically require a physical connection to the instrument, limiting the performer's range of motion and performance flexibility.

[0005] There is currently no good solution to the above problems. Summary of the Invention

[0006] This application provides a medium-free holographic device for musical instruments and its control method, so as to at least solve the problem of high complexity in music score operation in related technologies.

[0007] According to one embodiment of this application, a medium-free holographic device for musical instruments is provided, comprising:

[0008] A medium-free holographic module, configured to generate and display medium-free holographic musical score images;

[0009] The first sensor is configured to collect instrument movement information and / or instrument sound information during instrument operation, wherein the instrument movement information includes the vibration or pressing / pulling actions performed by the keys or strings of the instrument when the instrument is in operation, and the instrument sound information includes information about the sound emitted by the instrument.

[0010] The control module, connected to the first sensor and the medium-free holographic module, is configured to determine the score information based on the instrument's movement information and / or the instrument's sound information, and adjust the medium-free holographic score image based on the score information.

[0011] In one exemplary embodiment, it further includes:

[0012] The second sensor is connected to the control module and is configured to collect external action information of external forces operating the musical instrument.

[0013] The control module determines the score information based on the external action information, the instrument action information, and / or the instrument sound information, and adjusts the mediumless holographic score image according to the score information.

[0014] In one exemplary embodiment, the medium-free holographic module includes:

[0015] The display unit is signal-connected to the control module and is configured to generate musical score images according to the instructions of the control module;

[0016] An optical waveguide is configured to perform optical processing on the musical score image to generate a medium-free holographic musical score image.

[0017] In one exemplary embodiment, the musical instrument is connected to the body of the musical instrument using a medium-free holographic device;

[0018] or,

[0019] The medium-free holographic device for the musical instrument is not connected to the instrument body.

[0020] According to another embodiment of this application, a method for controlling a medium-free holographic device for musical instruments is provided, comprising:

[0021] The instrument's movement information and / or sound information are acquired through the first sensor. The instrument's movement information includes the vibration or pressing / pulling actions of the instrument's keys or strings when the instrument is in operation, and the instrument's sound information includes information about the sound emitted by the instrument.

[0022] The control module determines the score information based on the instrument movement information and / or instrument sound information, and the score information is used to indicate the score chapter corresponding to the instrument movement information or the instrument sound information;

[0023] The mediumless holographic module adjusts the pre-generated mediumless holographic score image based on the score information.

[0024] In one exemplary embodiment, it further includes:

[0025] External motion information is collected by a second sensor, wherein the external motion information includes external motion information of external actions that operate the musical instrument;

[0026] The control module determines the score information based on the external action information, the instrument action information, and / or the instrument sound information, and adjusts the mediumless holographic score image according to the score information.

[0027] In one exemplary embodiment, determining the musical score information based on the instrument movement information includes:

[0028] Based on the instrument movement information, determine the syllable information corresponding to the instrument movement;

[0029] The syllable information is matched with a preset musical score to determine the musical score information.

[0030] In one exemplary embodiment, determining the musical score information based on the instrument sound information includes:

[0031] Based on the instrument sound information, determine the syllable information corresponding to the instrument sound;

[0032] The syllable information is matched with a preset musical score to determine the musical score information.

[0033] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0034] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0035] This application enables automatic music score recognition and page turning by collecting and recognizing instrument sounds or movements, eliminating the need for manual operation. Therefore, it solves the problem of high complexity in music score operation, thereby improving the performance effect of instruments and reducing the difficulty of playing instruments. Attached Figure Description

[0036] Figure 1 is a structural block diagram of a medium-free holographic device for a musical instrument according to an embodiment of this application;

[0037] Figure 2 is a schematic diagram of a specific embodiment of this application;

[0038] Figure 3 is a schematic diagram of a specific embodiment of this application;

[0039] Figure 4 is a schematic diagram of a specific embodiment of this application;

[0040] Figure 5 is a flowchart of a control method for a mediumless holographic device for musical instruments according to an embodiment of this application. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0042] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0043] Furthermore, in this application, directional terms such as "upper," "lower," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and may change accordingly depending on the orientation of the components in the accompanying drawings.

[0044] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "coupled" can refer to an electrical connection that enables signal transmission.

[0045] As used herein, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from the given value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the given quantity (i.e., the limitations of the measurement system).

[0046] This embodiment also provides a medium-free holographic device for musical instruments. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0047] In general, instrumentalists need to consult sheet music when performing. This method places high demands on performers, especially for complex or fast-paced pieces. However, if staff remotely operate the sheet music display, it increases labor costs and is prone to errors. Therefore, a new sheet music display method is needed to adapt to this performance scenario.

[0048] Example 1

[0049] Figure 1 is a structural block diagram of a medium-free holographic device 1 for musical instruments according to an embodiment of this application. As shown in Figure 1, the medium-free holographic device 1 for musical instruments includes:

[0050] The system comprises a medium-free holographic module 11, a first sensor 12, and a control module 13. The first sensor 12 is configured to collect instrument movement information and / or instrument sound information. The instrument movement information includes actions performed by the keys or strings of the instrument 2 during operation, such as the vibration of the keys or strings themselves or the pressing or pulling of the keys or strings. The instrument sound information includes information about the sounds emitted by the instrument 2. The control module 13 is signal-connected to the first sensor 12 and the medium-free holographic module 11. The medium-free holographic module 11 is configured to generate and display a medium-free holographic musical score image 113 (as shown in Figures 2 and 3). The medium-free holographic musical score image 113 can be a staff notation or other types of musical score (e.g., dynamic operations used to indicate actions or key presses). (Prompts, etc.); The control module 13 determines the score information based on the instrument's movement information and / or sound information, and adjusts the medium-free holographic score image 113 according to the score information. For example, if the first sensor 12 collects a sound or movement, the control module 13 then identifies that the sound or movement corresponds to a certain movement in the score, and then instructs the medium-free holographic module 11 to turn the pages of the score or change the movement, so that the performer can play the movement without turning the pages himself; In particular, during the process of turning the pages or changing the movement, prompts can also be made based on the sound or movement to indicate the current movement or syllable position, ensuring the correctness of the performance; Of course, the prompts can be made using triangular indicators, special colors, etc., which are not limited here, as long as prompts can be made, they fall within this scope.

[0051] Among them, musical instrument 2 can be a keyboard instrument with certain keys such as a piano or electronic keyboard, or a wind instrument such as a saxophone, suona, or wind pipe, a string instrument such as an erhu, violin, or harp, or a percussion instrument such as a yangqin or drum. That is, idiophones, membranophones, aerophones, stringed instruments, and electrophones are all within the scope of musical instruments referred to in this application. The first sensor 12 can be a combination of an array microphone and a sound sensor for sound acquisition, or a high-pixel camera or light sensor for motion acquisition, or other devices capable of sound or motion acquisition, such as a combination of radar, camera, and sound sensor, an infrared sensor, camera, and sound sensor, or a combination of radar, camera, infrared sensor, and sound sensor, etc., which are not limited here. The control module 13 can be a CPU / GPU combination for image and sound processing. The actions acquired here include the actions of the instrument itself, such as the changes in the vibration of the keys / strings when the keys are pressed or the strings are plucked or pulled.

[0052] Of course, to recognize actions or sounds, a pre-trained neural network model or other deep learning model is needed. Specifically, this involves first denoising the collected sound and / or action information to remove background noise and retain clear instrument sounds or movements; then normalizing the data to standardize it to a uniform range for better model processing; next, extracting useful features from the raw data—for example, spectral features (such as MFCC, Mel-frequency cepstral coefficients) for sound data, and motion trajectories for action data; then feeding this data into the corresponding model, such as a convolutional neural network (CNN) model suitable for processing image and video data (which can be used to recognize instrument movements), or a recurrent neural network (RNN) model suitable for processing time-series data (which can recognize sound signals), or a Transformer architecture-related model (not limited here); finally, the model automatically extracts features and makes predictions. The system outputs the corresponding score or syllable information and matches the identified syllables or score with a preset score, which can be stored in a database. Then, a search algorithm (such as Dynamic Time Warping (DTW)) is used to compare the identified syllable sequence with the syllable sequence in the score to find the most matching section. Based on the matched score information, instructions for adjusting the medium-free holographic score image are generated. For example, if the currently played movement requires a page turn, a page-turning instruction is generated; if the current syllable position needs to be indicated, a syllable indication instruction is generated. The adjustment instructions are then transmitted to the medium-free holographic module to update the content of the medium-free holographic score image by controlling the optical waveguide and display components, ensuring it matches the performer's playing progress. During model optimization, if the match is successful, the current score position is determined; if the match fails, a prompt message is issued to remind the performer or system to perform further processing, such as further optimizing and updating the model based on the output and matching results.

[0053] As shown in Figure 2, the medium-free holographic module 11 includes a display element 111 and an optical waveguide plate 112. The display element 111 is signal-connected to the control module 13 and is configured to generate a medium-free holographic music score image 113 according to the instructions of the control module. The display element 111 can be a display device or equipment that can generate light source images, such as a display screen or monitor. The optical waveguide plate 112 is used to perform optical processing on the music score image to generate the medium-free holographic music score image 113. The optical waveguide plate 112 can be an optical element or device described in any of the Chinese patents with application numbers 202210060077.2, 202221492951.1, and 201920104395.8, or other optical elements or devices that can perform levitation imaging, which are not limited here.

[0054] It should be noted that, due to the different types of musical instruments 2, the medium-free holographic device 1 for musical instruments can be connected to the body of the musical instrument 2 or not. This connection can be a direct connection, such as a fixed connection or a detachable connection, or a signal connection via cables or communication equipment. Fixed connections include welding, gluing, and connections made difficult to disassemble after installation using specific mounting structures. Detachable connections include bolt fixing, Velcro, and other connection methods that allow for easy assembly and disassembly. Signal connections can involve mounting the first sensor 12 on the body of the musical instrument 2, while other modules are mounted in other locations, and transmitting signals via signal protocols or cables; this is not limited here. It is easy to understand that "not connected" here means that the medium-free holographic device 1 for musical instruments does not have direct contact with the body of the musical instrument 2, but directly collects the sound or movement information of the musical instrument 2. This solution directly identifies the sound or movement information after filtering it. The body of instrument 2 refers to the part that can be played, such as the body of a guitar, piano, violin, or wind instrument. For instruments with sufficient installation space, such as a piano, the instrument-free holographic device 1 can be installed inside the body. For instruments without installation space, such as a guitar, the instrument-free holographic device 1 is not connected to the body of instrument 2, as shown in Figures 3 and 4. In Figure 3(A), the instrument-free holographic device 1 is external to the body of instrument 2 and connected to it. In Figure 3(B), the instrument-free holographic device 1 is external to the body of instrument 2 and not connected to it. In Figure 4, the instrument-free holographic device 1 is external to the body of instrument 2, but can be installed internally or pushed / pulled through a hinge. In Figure 4, the instrument-free holographic device 1 is internally installed inside the body of instrument 2 without hinges or push / pull mechanisms.

[0055] In particular, due to differences in installation location and usage requirements, the image display methods and usage methods of medium-free holographic devices for musical instruments also differ. Take the piano as an example:

[0056] As shown in structure (B) of Figure 4, the instrument can be mounted inside the piano body with a built-in medium-free holographic device 1. The resulting medium-free musical score image can be located above the piano body or above the keys (i.e., in front of the piano body), or it can be mounted on the surface of the piano body. Alternatively, the instrument can be built into the piano body with the device 1 hinged to it (as shown in structure (A) of Figure 4; the dotted line represents the hinged rotation and push-pull state). This allows the player to adjust the position according to their needs. Pulling or flipping the mediumless holographic device for musical instruments changes the display position or angle of the mediumless musical score image. The specific hinge or installation method is not limited here, as long as the above function can be achieved. Of course, in addition to hinges, other methods that can change the position of the built-in mediumless holographic device 1 for musical instruments and thus change the display position of the mediumless holographic musical score image 113 are all within the protection scope of this application. For example, changing the position of the mediumless holographic device 1 for musical instruments through a specially designed sliding structure, a fixed electronic push rod or cylinder, etc., are all within the aforementioned scope and are not limited here.

[0057] Example 2

[0058] The difference from Embodiment 1 is that, to further accurately identify the musical movement, the performer's movements can also be captured, and the progress of the musical movement can be identified based on the performer's movements; specifically, the medium-free holographic device for musical instruments also includes:

[0059] The second sensor 14 is signal-connected to the control module 13 and is configured to collect external action information of external forces operating the musical instrument.

[0060] The control module 13 determines the score information based on the external action information, the instrument action information and / or the instrument sound information, and adjusts the mediumless holographic score image 113 according to the score information.

[0061] Generally, in a fixed musical movement, the performer's movements on the instrument are fixed. Whether it is striking, blowing, plucking, or pressing, they need to maintain a fixed position to achieve accurate performance of the movement. Therefore, by capturing and recognizing the changes and distribution of the performer's movements over a period of time, the current musical movement can be determined, and it can be determined whether it is necessary to turn the page or change the movement.

[0062] As is easily understood, the second sensor 14 can also be a sensor capable of capturing the performer's movements, such as an infrared sensor, a light sensor, a camera, radar, or at least one of these, without limitation here; and the installation position and method of the second sensor 14 should preferably be based on the ability to capture the performer's limb movements, such as the movements of the fingers, elbows, arms, etc., and for more accurate recognition, multiple sensors should be installed in multiple locations; external motion information refers to the limb movements performed by the performer on the instrument, such as striking, blowing, plucking, pressing, and pulling.

[0063] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0064] Example 3

[0065] In this embodiment, a method for controlling a medium-free holographic device for musical instruments is provided. Figure 5 is a flowchart of a method for controlling a medium-free holographic device for musical instruments according to an embodiment of this application. As shown in Figure 5, the process includes the following steps:

[0066] Step S51: Acquire instrument movement information and / or instrument sound information of the instrument through the first sensor 12, wherein the instrument movement information includes the vibration or pressing / pulling action of the instrument's keys or strings when the instrument is working, and the instrument sound information includes the information of the sound emitted by the instrument.

[0067] Step S52, the control module 13 determines the score information based on the instrument movement information and / or instrument sound information, the score information being used to indicate the score chapter corresponding to the instrument movement information or the instrument sound information;

[0068] Step S53: The medium-free holographic module 11 adjusts the pre-generated medium-free holographic score image according to the score information.

[0069] The entities that perform the above steps can be base stations, terminals, etc., but are not limited to these.

[0070] In an optional embodiment, the method further includes:

[0071] Step S54: Collect external motion information through the second sensor 14, wherein the external motion information includes external motion information of external actions that operate the musical instrument;

[0072] In step S55, the control module 13 determines the score information based on the external action information, the instrument action information, and / or the instrument sound information, and adjusts the mediumless holographic score image based on the score information.

[0073] Example 4

[0074] The step of determining the musical score information based on the instrument movement information includes:

[0075] Step S521: Determine the syllable information corresponding to the instrument movement based on the instrument movement information;

[0076] Step S522: Match the syllable information with a preset musical score to determine the musical score information.

[0077] In this embodiment, the instrument's movements are collected to determine the progress of the musical movement based on the instrument's own movements, thereby effectively providing the performer with progress prompts or automatic page turning.

[0078] Specifically, the collected instrument movement information includes the movements of the instrument keys or strings, such as the speed, force, and duration of key presses, or the amplitude and frequency of string plucking. The collected movement information is then analyzed to extract syllable-related feature parameters. For example, for keyboard instruments, the note being played can be inferred from the order, speed, and duration of key presses; for string instruments, the pitch and intensity can be determined from the frequency and amplitude of string plucking; for guitar playing, the corresponding note can be determined by detecting the plucking action and the position of the finger press. Then, based on the analyzed feature parameters, the syllable information corresponding to the instrument movement is determined. For example, by analyzing the order and time interval of key presses, the sequence of notes played can be determined; by analyzing the frequency of string plucking, the pitch can be determined, and thus the syllable. Syllable information can include parameters such as the pitch, duration, and intensity of the note; these parameters together constitute the syllable information corresponding to the instrument movement.

[0079] The preset musical score can be a standard score stored in a database or a user-defined score; the preset musical score information is stored in the system's database; the extracted syllable information is matched with the preset musical score. Dynamic Time Warping (DTW), Longest Common Subsequence (LCS) algorithms, or other similarity matching algorithms can be used to compare the similarity between the syllable information and the musical score. During the matching process, the system compares the pitch, duration, intensity, and other parameters of the syllable information with the notes in the musical score one by one to find the most matching part; the determined musical score information includes the pitch, duration, intensity, and other parameters of the notes, as well as the arrangement order of the notes and the movement structure. The musical score information can be stored in a digital format, such as music XML or MIDI format, for easy reading and matching by the system; by matching the syllable information with the preset musical score, the musical score information currently being played can be determined. The matching process can include comparisons of note duration, pitch, rhythm, etc., to ensure the accuracy of the generated musical score information.

[0080] Example 5

[0081] The difference from Example 4 is that, in order to better ensure the accuracy of data matching, the collected instrument movement information can be further analyzed, and the progress of the movement can be judged by analyzing whether the instrument movement information is normal.

[0082] Specifically, the collected instrument movement information (such as key speed, force, duration, movement frequency, and movement amplitude) can be normalized. Then, the normalized instrument movement information is used to construct an instrument movement matrix. The correlation value of the instrument movement matrix is ​​then calculated based on the Pearson coefficient. If the correlation value is within a preset range, the relevant movement is preliminarily determined to be reasonable. Subsequently, based on the instrument movement matrix, the matrix corresponding to the preset musical score with the highest similarity value to the instrument movement matrix is ​​matched, and the preset musical score corresponding to the matrix is ​​used as the optional musical score information. If the subsequent actual performance of the musical score changes the same as the optional musical score information, then the optional musical score information is used as the final musical score information, and so on.

[0083] Example 6

[0084] In an optional embodiment, determining the musical score information based on the instrument sound information includes:

[0085] Step S523: Determine the syllable information corresponding to the instrument sound based on the instrument sound information;

[0086] Step S524: Match the syllable information with a preset musical score to determine the musical score information.

[0087] In this embodiment, the instrument movements are collected to determine the progress of the musical movement based on the sound of the instruments, thereby effectively providing the performer with prompts on the progress of the musical movement or automatically turning pages.

[0088] This process involves extracting the fundamental frequency of a sound signal using methods such as autocorrelation algorithms and cepstral analysis to determine the pitch of a note; analyzing the amplitude of the sound signal to determine the loudness of the note; determining the duration of the note using methods such as short-time energy analysis; and distinguishing the sound characteristics of different instruments by analyzing the spectral features of the sound signal. Subsequently, based on the extracted feature vectors, a pre-trained sound recognition model (such as a deep learning model) is used to classify or regress the sound signal to determine the syllable information corresponding to the instrument's sound. For example, for a piano performance sound signal, the model can determine the pitch based on the extracted pitch... Features such as intensity and duration are used to identify specific notes (such as C4, D4, etc.) and their durations. Syllable information can be represented as a series of notes and their corresponding durations and intensities, for example: [C4, 1 / 4], [E4, 1 / 4], [G4, 1 / 2], indicating that the notes C4 (duration 1 / 4 beat), E4 (duration 1 / 4 beat), and G4 (duration 1 / 2 beat) were played. Then, according to the matching results, the extracted syllable information is successfully matched with the note sequence of beats 1-3 of the first measure in the score. At this point, there is no need to turn the page, and so on.

[0089] Example 7

[0090] The difference from Example 6 is that, in order to better ensure the accuracy of data matching, the collected instrument sound information can be further analyzed, and the progress of the movement can be judged by analyzing whether the instrument sound information is normal.

[0091] Specifically, the collected instrument movement information (such as fundamental frequency, loudness, amplitude, duration, spectral characteristics, etc.) can be normalized. Then, the normalized instrument sound information is used to construct an instrument sound matrix. The correlation value of the instrument sound matrix is ​​then calculated based on the Pearson coefficient. If the correlation value is within a preset range, the related sound is preliminarily determined to be reasonable. Subsequently, based on the instrument sound matrix, the matrix corresponding to the preset score with the highest similarity value to the instrument sound matrix is ​​matched, and the preset score corresponding to the matrix is ​​used as the optional score information. If the subsequent actual performance score changes are the same as the optional score information, the optional score information is used as the final score information, and so on.

[0092] Example 8

[0093] The difference from Embodiments 5 and 7 is that, after obtaining the optional score information based on the instrument action matrix and the instrument sound matrix, the similarity of the two scores is calculated. If the similarity meets the requirements, the optional score information can be directly used as the final score information without comparing it with the actual performance; alternatively, the optional score information can be compared with the actual performance to ensure the accuracy of the performance.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0095] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0096] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0097] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0098] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0103] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A medium-free holographic device for musical instruments, comprising: A medium-free holographic module for generating and displaying medium-free holographic musical score images; The first sensor is used to collect instrument movement information and / or instrument sound information when the instrument is in operation, wherein the instrument movement information includes the vibration or pressing / pulling actions performed by the keys or strings of the instrument when the instrument is in operation, and the instrument sound information includes information about the sound emitted by the instrument. The control module, connected to the first sensor and the medium-free holographic module, is used to determine the score information based on the instrument's movement information and / or the instrument's sound information, and to adjust the medium-free holographic score image based on the score information.

2. The medium-free holographic device for musical instruments according to claim 1, wherein, Also includes: The second sensor is connected to the control module and is used to collect external action information of external forces operating the musical instrument. The control module determines the score information based on the external action information, the instrument action information, and / or the instrument sound information, and adjusts the mediumless holographic score image according to the score information.

3. The medium-free holographic device for musical instruments according to claim 1, wherein, The medium-free holographic module includes: The display unit is signal-connected to the control module and is used to generate musical score images according to the instructions of the control module; An optical waveguide plate is used to perform optical processing on the musical score image to generate a medium-free holographic musical score image.

4. The medium-free holographic device for musical instruments according to claim 1, characterized in that, The musical instrument is connected to its body using a medium-free holographic device; or, The medium-free holographic device for the musical instrument is not connected to the instrument body.

5. A method for controlling a medium-free holographic device for musical instruments, comprising: The instrument's movement information and / or sound information are acquired through the first sensor. The instrument's movement information includes the vibration or pressing / pulling actions of the instrument's keys or strings when the instrument is in operation. The instrument's sound information includes information about the sound emitted by the instrument. The control module determines the score information based on the instrument movement information and / or instrument sound information, and the score information is used to indicate the score chapter corresponding to the instrument movement information or the instrument sound information; The mediumless holographic module adjusts the pre-generated mediumless holographic score image based on the score information.

6. The method according to claim 5, wherein, Also includes: External motion information is collected by a second sensor, wherein the external motion information includes external motion information of external actions that operate the musical instrument; The control module determines the score information based on the external action information, the instrument action information, and / or the instrument sound information, and adjusts the mediumless holographic score image according to the score information.

7. The method according to claim 5, wherein, The step of determining the musical score information based on the instrument movement information includes: Based on the instrument movement information, determine the syllable information corresponding to the instrument movement; The syllable information is matched with a preset musical score to determine the musical score information.

8. The method according to claim 5, wherein, Determining the musical score information based on the instrument sound information includes: Based on the instrument sound information, determine the syllable information corresponding to the instrument sound; The syllable information is matched with a preset musical score to determine the musical score information.

9. A computer-readable storage medium storing a computer program, wherein, The computer program is configured to execute the method described in any one of claims 5 to 8 when it is run.

10. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor being configured to run the computer program to perform the method of any one of claims 5 to 8.

Images