Music score recognition method, music score reconstruction method, system, and computer program product

Abstract

The present invention relates to the field of digital processing of music scores, and in particular to a music score recognition method, a music score reconstruction method, a system, and a computer program product. The method comprises: providing a music score, wherein the music score comprises a plurality of first semantic elements; recognizing a first semantic form of each of the first semantic elements; determining whether each of the first semantic elements is a primary semantic element by using a first evaluation rule; if yes, grouping primary semantic elements used for forming a complete identifier into a semantic element subset; recognizing at least two termination lines from among the plurality of first semantic elements; and classifying the plurality of first semantic elements into a plurality of semantic element sets on the basis of the termination lines. The present invention provides a method for conversion from semantic elements to a MusicXML format file, capable of assisting a user in rapidly integrating non-electronic music score data into mainstream music platform software, and then performing electronic editing on non-electronic music scores.

Description

A music score recognition method, music score reconstruction method, system, and computer program product.

[0001] Priority application

[0002] This application claims priority to Chinese Invention Patent Application No. [2025102318623] "A Music Score Recognition Method and System", filed on February 26, 2025, which is incorporated herein by reference in its entirety. Technical Field

[0003] This invention relates to the field of electronic music notation technology, specifically to a music notation recognition method, a music notation reconstruction method, a system, and a computer program product. Background Technology

[0004] Sheet music is an important medium for music creation and learning, so the recording and analysis of electronic sheet music data is also crucial for music users' learning process.

[0005] Currently, traditional music score input and editing typically rely on manual operations, which are time-consuming and labor-intensive. Although existing technologies have proposed electronic recognition methods for music score data, they still have shortcomings in terms of the universality and reliability of music score data recognition.

[0006] For example, patent application CN110852375A discloses an end-to-end musical note recognition method based on deep learning. This method includes: data preprocessing: downloading the corresponding dataset from MuseScore and re-encoding pitch and duration labels; data augmentation: augmenting the re-encoded musical score data; and an end-to-end model: a deep convolutional neural network model applied to end-to-end musical note recognition, inputting the augmented data into the model, and outputting the note duration and pitch. However, this note recognition method can only recognize limited information such as note duration and pitch.

[0007] For example, patent application CN114359946A discloses an optical music score image recognition method based on residual attention Transformer. The steps of this recognition process are: establishing a music score image recognition model based on residual attention Transformer to extract note sequence features from the music score image; and digitizing the note sequence in the music score image based on the captured note sequence features. This invention uses a pre-trained shallow differential convolutional neural network to initialize parameters and extract basic note features. Then, a recurrent neural network is used to encode and decode the obtained note features and label sequences respectively to extract the association information of the note sequences. However, this method requires removing pixels outside the note range and can only recognize note pixels.

[0008] In summary, existing sheet music image recognition methods have a very limited depth of information they can recognize. Furthermore, when faced with complex scores (such as multi-layered notes, handwritten scores, and complex symbols), they often suffer from insufficient accuracy and low processing efficiency, failing to meet user needs. Summary of the Invention

[0009] The purpose of this invention is to provide a music score processing method that partially solves or alleviates the above-mentioned shortcomings in the prior art and can improve the reliability of data conversion in the process of converting music score images into electronic music data.

[0010] To solve the aforementioned technical problems, the present invention specifically adopts the following technical solution: The first aspect of the present invention is to provide a method for reconstructing musical scores, comprising the following steps:

[0011] S201, providing a musical score, the musical score including a plurality of first ideographic elements;

[0012] S202, identify the first ideographic form of the first ideographic element, and the first ideographic form is also associated with a confidence level; wherein, the first ideographic form can express the meaning of at least one musical sound element in the musical score, the first ideographic form includes one or more of the following: shape, position, quantity, direction, the musical sound element includes one or more of the following: duration, dynamics, pitch, volume, rhythm, meter, tonality, mode, and the confidence level is used to represent the recognition accuracy of the first ideographic form.

[0013] S203, the first evaluation rule is used to determine whether the first ideographic element is a first-level ideographic element; the first evaluation rule requires that when a first ideographic element must be connected with another first ideographic element on the musical score to form a complete identifier, the corresponding first ideographic element is considered to be the first-level ideographic element.

[0014] If so, the first-level ideographic elements used to form a complete identifier are grouped into a subset of ideographic elements.

[0015] S204, Identify at least two termination lines from a plurality of the first semantic elements;

[0016] S205, classify the plurality of first ideographic elements into a plurality of ideographic element sets according to the termination line, wherein the first ideographic element located between two adjacent termination lines is classified into one ideographic element set, and one ideographic element set includes at least one ideographic element subset.

[0017] S206, Generate an electronic data sequence for describing the musical score based on the plurality of said ideographic element sets.

[0018] In some embodiments, the method further includes the steps of: identifying at least one subset of the ideographic elements; determining the identifier type of the complete identifier based on the first ideographic form of a plurality of the first-level ideographic elements in the subset of the ideographic elements; and recording the identifier type and the musical element of the complete identifier in the corresponding set of ideographic elements.

[0019] In some embodiments, the method further includes the step: S207, verifying the completeness of the semantic information of the semantic element set using a time value evaluation model; wherein, S207 includes: S2071, identifying beat information from the semantic element set, and determining the expected musical note duration of the semantic element set based on the beat information; S2072, identifying a first type of time value semantic element and a second type of time value semantic element from the semantic elements, wherein the first type of time value semantic element includes one or more of the following types: rest, note, dotted note, and the second type of time value... The types of ideographic elements include: slurs; S2073, determine the first performance duration based on the first type of time value ideographic elements, and determine the second performance duration based on the second type of time value ideographic elements; S2074, determine whether the expected musical note duration, the first performance duration, and the second performance duration satisfy the time value evaluation model, the time value evaluation model including: first performance duration - second performance duration = expected musical note duration; if yes, it is preliminarily considered that the ideographic information of the set of ideographic elements is complete; if not, the ideographic information of the set of ideographic elements will be supplemented.

[0020] In some embodiments, the step of supplementing the semantic information of the semantic element set includes: (1) obtaining the first confidence level of the first type of time value semantic element and the second confidence level of the second type of time value semantic element; (2) determining whether the first confidence level and the second confidence level are both greater than the set first confidence value; if yes, then the step is executed: (3) obtaining the time value difference, where the time value difference = musical duration - (first performance duration - second performance duration); (4) generating at least one supplementary rest information based on the time value difference; if no, then the step is executed: (5) outputting a prompt signal to prompt the user to correct the semantic information of the semantic element set.

[0021] In some embodiments, the method further includes the steps of: obtaining a first position of the first ideographic element; determining whether the first ideographic element crosses at least one termination line based on the first position; if so, disassembling the first ideographic element into at least two ideographic sub-elements based on the corresponding termination line, and recording the second position of the ideographic sub-elements; and classifying the at least two second ideographic sub-elements into corresponding sets of at least two ideographic elements according to the second position.

[0022] In some embodiments, the method further includes the steps of: determining whether the first ideographic element is an auxiliary ideographic element, wherein the auxiliary ideographic element is an ideographic element used to supplement auxiliary information for the note; if so, then the steps of: obtaining the first position of the first ideographic element; identifying at least one first ideographic element at a third position associated with the first position; and associating the auxiliary information with the corresponding first ideographic element. In some embodiments, the auxiliary ideographic element includes one or more of the following: slur, tie, rest, staccato mark, sustain mark, pipa mark, and ornament.

[0023] In some embodiments, S202 includes the step of: S2021, using a first target detection model to identify a first-level first semantic element; wherein, the first target detection model includes: an input terminal for receiving a first musical score image; a feature extraction network for extracting feature information from the first musical score image; wherein, the first feature information includes: pixel information of the first-level first semantic element; a feature fusion network for fusing the first feature information and outputting a first feature map; and a target detection network for identifying the first feature map according to a corresponding minimum recognition unit, so as to classify and output the fusion result, wherein, the minimum recognition unit is the minimum resolution for the target detection network to identify the first feature map, and correspondingly, the target detection network sequentially includes: a second feature detection layer, a third feature detection layer, and a fourth feature detection layer connected to each other, and the size of the minimum recognition unit from the second feature detection layer to the fourth feature detection layer gradually increases;

[0024] S2022, a second target detection model is used to identify the second-level first ideographic element, wherein the second-level first ideographic element is an ideographic element that can be independently drawn in the musical score; wherein the second target detection model includes: an input terminal for receiving the second musical score image; a feature extraction network for extracting second feature information from the second musical score image; wherein the second feature information includes: information of the second-level first ideographic element; a feature fusion network for fusing the second feature information and outputting a corresponding second feature map; and a target detection network for identifying the second feature map according to the corresponding minimum recognition unit, and classifying and outputting the fusion result, wherein the minimum recognition unit is the minimum resolution for the target detection network to identify the second feature map, and correspondingly, the target detection network includes, in sequence: a first feature detection layer, a second feature detection layer, a third feature detection layer, and a fourth feature detection layer connected together; wherein the size of the first minimum recognition unit of the first feature detection layer is 4×4 to 8×8 pixels, and the size of the minimum recognition unit from the first feature detection layer to the fourth feature detection layer gradually increases.

[0025] This invention also provides a music score reconstruction system, comprising: a music score acquisition module for providing a music score, the music score including multiple first ideographic elements; an ideographic element identification module for identifying a first ideographic form of the first ideographic element, wherein the first ideographic form is also associated with a confidence level; wherein the first ideographic form can express the meaning of at least one musical sound element in the music score, the first ideographic form including one or more of the following: shape, position, quantity, direction, the musical sound element including one or more of the following: duration, dynamics, pitch, volume, rhythm, meter, tonality, mode, the confidence level being used to represent the recognition accuracy of the first ideographic form; and an ideographic element evaluation module for determining whether the first ideographic element is a first-level ideographic element using a first evaluation rule. The first evaluation rule requires that when a first ideographic element must be connected to another first ideographic element on the musical score to form a complete identifier, the corresponding first ideographic element is considered to be the first-level ideographic element; if so, the first-level ideographic elements used to form a complete identifier are classified into a subset of ideographic elements; the termination line identification module is used to identify at least two termination lines from a plurality of first ideographic elements; the ideographic element classification module is used to classify a plurality of first ideographic elements into a plurality of ideographic element sets according to the termination lines, wherein a first ideographic element located between two adjacent termination lines is classified into one set of ideographic elements, and one set of ideographic elements includes at least one subset of ideographic elements.

[0026] In another aspect, the present invention also provides a music score recognition method, comprising: S101, constructing a target detection model, wherein the target detection model comprises, in sequence:

[0027] The input terminal is used to receive a musical score image; the feature extraction network is used to extract feature information from the musical score image; wherein, the feature information includes: (1) basic information, the basic information includes: note information, clef information, key signature information, slur information, polyphonic information; (2) technique information, the technique information includes: performance technique information, fingering information; (3) emotional style information, the emotional style information includes: expression marking information, dynamic marking information, time signature information; the feature fusion network is used to fuse the feature information and output a corresponding feature map; the target detection network is used to identify the feature map according to the corresponding minimum recognition unit, so as to classify and output the fusion result, wherein, the minimum recognition unit is the target detection unit. The target detection network is defined as the minimum resolution at which the network identifies the feature map. Correspondingly, the target detection network sequentially includes a first feature detection layer, a second feature detection layer, a third feature detection layer, and a fourth feature detection layer connected to each other. The first feature detection layer performs target detection on the feature map based on a first minimum recognition unit, the second feature detection layer performs target detection on the feature map based on a second minimum recognition unit, the third feature detection layer performs target detection on the feature map based on a third minimum recognition unit, and the fourth feature detection layer performs target detection on the feature map based on a fourth minimum recognition unit. The size of the first minimum recognition unit is 4 pixels × 4 pixels to 8 pixels × 8 pixels, and the sizes of the first minimum recognition unit, the second minimum recognition unit, the third minimum recognition unit, and the fourth minimum recognition unit gradually increase.

[0028] The output detection results include: a data stream formed by multiple feature sequences, wherein the feature sequences include: the type of feature information, the location of feature information, and the confidence level of feature information;

[0029] S102, the musical score image to be identified is input into the target detection model, and the target detection network outputs the corresponding detection result.

[0030] In some embodiments, before S102, the method further includes the step of: performing detail feature enhancement processing on the sheet music image to be identified; wherein the detail feature enhancement processing includes the steps of: performing grayscale processing on the sheet music image to be identified to obtain a grayscale image; performing binarization processing on the grayscale image to obtain a binarized image; performing shadow removal operation on the binarized image to obtain a shadow-removed image; identifying outliers in the shadow-removed image, and using contrast enhancement operation to reduce the intensity of the visual features of the outliers, thereby obtaining a contrast-enhanced image; wherein the outliers include: a type of point, and the interval between the type of point and the position of the target feature in the shadow-removed image is greater than a first interval threshold, and the target feature includes one or more of the following: notes, clefs, rests, fermata, staccato marks, dynamic marks, and performance techniques.

[0031] In some embodiments, the outlier further includes: a binary point. Correspondingly, the step of identifying outliers in the deshaded image includes: identifying target points in the deshaded image, wherein the target point is a set of consecutive pixels whose pixel value is less than a set threshold; when the position of the target point and the corresponding target feature in the deshaded image is less than or equal to a first interval threshold, the target point is identified as an associated point of the target feature; when the number of associated points of the target feature is greater than the target number, a binary point is selected from the associated points.

[0032] In some embodiments, the step of selecting a binary point from the associated points includes: calculating the intensity difference of visual features between at least two associated points associated with the same target feature; wherein the visual features include one or more of the following: grayscale, brightness, size; when the intensity difference is greater than a set intensity threshold, the associated point with the smaller intensity of the visual feature is selected as a binary point; when the intensity difference is less than or equal to the intensity threshold, a user input instruction is received, the input instruction being used to define a binary point from at least two associated points.

[0033] In some embodiments, when the intensity difference is less than or equal to the intensity threshold, the method further includes the steps of: comparing the binarized image and the deshaded image to determine whether there is a newly added target point in the original shadow region of the deshaded image, and marking the target point as a newly added point; determining whether the associated point is the newly added point; if so, outputting a suggestion signal to the user to define the corresponding associated point as the binary point.

[0034] In some embodiments, the method further includes the step of tilt correction on the contrast-enhanced image.

[0035] In some embodiments, the selectable range of the second minimum identification unit is 8×8 pixels to 16×16 pixels, and / or the selectable range of the third minimum identification unit is 16×16 pixels to 32×32 pixels, and / or the selectable range of the fourth minimum identification unit is 32×32 pixels to 64×64 pixels.

[0036] The present invention also provides a music score recognition system, comprising:

[0037] A model providing module is used to provide an object detection model, wherein the object detection model comprises, in sequence:

[0038] The input terminal is used to receive musical score images;

[0039] A feature extraction network is used to extract feature information from the musical score image; wherein the feature information includes: (1) basic information, which includes: note information, clef information, key signature information, slur information, and polyphonic information; (2) technical information, which includes: performance technique information and fingering information; (3) emotional style information, which includes: expression marking information, dynamic marking information, and time signature information;

[0040] A feature fusion network is used to fuse the feature information and output a feature map accordingly.

[0041] An object detection network is provided, which identifies the feature map based on a corresponding minimum recognition unit to classify and output the fusion result. The minimum recognition unit is the minimum resolution at which the object detection network identifies the feature map. Correspondingly, the object detection network sequentially includes a first feature detection layer, a second feature detection layer, a third feature detection layer, and a fourth feature detection layer connected in sequence. The first feature detection layer performs object detection on the feature map based on a first minimum recognition unit; the second feature detection layer performs object detection on the feature map based on a second minimum recognition unit; the third feature detection layer performs object detection on the feature map based on a third minimum recognition unit; and the fourth feature detection layer performs object detection on the feature map based on a fourth minimum recognition unit. The size of the first minimum recognition unit is 4 pixels × 4 pixels to 8 pixels × 8 pixels, and the sizes of the first minimum recognition unit, the second minimum recognition unit, the third minimum recognition unit, and the fourth minimum recognition unit gradually increase.

[0042] The output detection results include: a data stream formed by multiple feature sequences, wherein the feature sequences include: the type of feature information, the location of feature information, and the confidence level of feature information;

[0043] The detection module is used to input the sheet music image to be identified into the target detection model, and the target detection network outputs the detection result accordingly.

[0044] In some embodiments, the image further includes: an image processing module for performing detail feature enhancement processing on the sheet music image to be identified; wherein the image processing module includes: a grayscale processing unit for performing grayscale processing on the sheet music image to be identified to obtain a grayscale image; a binarization processing unit for performing binarization processing on the grayscale image to obtain a binarized image; a shadow processing unit for performing shadow removal operation on the binarized image to obtain a shadow-free image; and an anomaly processing unit for identifying anomalies in the shadow-free image and using a contrast enhancement operation to reduce the intensity of the visual features of the anomalies, thereby obtaining a contrast-enhanced image; wherein the anomalies include: a type of point, and the interval between the type of point and the position of the target feature in the shadow-free image is greater than a first interval threshold, and the target feature includes one or more of the following: notes, clefs, rests, fermata, staccato marks, dynamic marks, and performance techniques.

[0045] In some embodiments, the outlier further includes: a binary point. Correspondingly, the outlier processing unit includes: an identification subunit, configured to identify a target point in the deshaded image, wherein the target point is a set of consecutive pixels whose pixel value is less than a set threshold; an association subunit, configured to identify the target point as an associated point of the target feature when the position of the target point and the corresponding target feature in the deshaded image is less than or equal to a first interval threshold; and a binary point selection unit, configured to select a binary point from the associated points when the number of associated points of the target feature is greater than the target number.

[0046] The present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the music score reconstruction method as described in any of the present invention.

[0047] Beneficial technical effects: This invention proposes a music score recognition method. This method comprehensively and deeply extracts data from three dimensions of the music score: basic information, technical information, and emotional style information. The recognition model outputs a standardized feature sequence (or standardized data stream) at the target detection model's output end. This feature sequence is represented by three dimensions: feature type, position, and confidence level.

[0048] Furthermore, the comprehensive collection of basic information, technical information, and emotional style information helps to identify the core information of the score. At the same time, information such as type, position, and confidence level helps to completely reconstruct and restore the score, so as to complete the reverse reconstruction from feature sequence to score file (such as MusicXML file).

[0049] In another aspect, this invention also provides a method for reconstructing musical scores. This method can group numerous ideographic elements (i.e., pre-classify them) by combining the type of the ideographic elements and measure information (such as the ideographic form of the end line). This pre-classification method not only conforms to general music theory rules, but also helps to generate XML elements based on each set in a standardized manner, reducing the pressure of the data conversion process. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. The elements or parts in the drawings are not necessarily drawn to scale. Obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0051] Figure 1 is a schematic flowchart of the music score recognition method in an exemplary embodiment of the present invention;

[0052] Figure 2 is a flowchart illustrating the detailed feature enhancement processing steps of the music score recognition method in an exemplary embodiment of the present invention;

[0053] Figure 3 is a schematic diagram of a musical score photograph in a specific embodiment of the present invention;

[0054] Figure 4 is a schematic diagram of the effect after detail enhancement processing of Figure 3;

[0055] Figure 5 is a schematic diagram of the module structure of the music score recognition system in an exemplary embodiment of the present invention;

[0056] Figure 6 is a flowchart illustrating an exemplary reconstruction method of the present invention;

[0057] Figure 7 is a schematic diagram of the identification method of the first-level semantic element in an exemplary embodiment of the present invention;

[0058] Figure 8 is a schematic diagram of a musical score in an exemplary embodiment of the present invention;

[0059] Figure 9 is a schematic diagram of the identification method of the first-level semantic element in another exemplary embodiment of the present invention.

[0060] Figure 10 is a schematic diagram of the content of an XML file in an exemplary embodiment of the present invention.

[0061] Summary of reference numerals in the attached diagram: Identifier 10, First ideographic element A11, First ideographic element B12, Confidence level 20, First symbol head 11a, Second symbol head 11b, First stem 12a, Second stem 12b, First hyphen 13a, Second hyphen 13b. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0063] In this document, suffixes such as "module," "component," or "unit" used to denote elements are used solely for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "component," or "unit" may be used interchangeably. In this document, terms such as "upper," "lower," "inner," "outer," "front," "rear," "one end," and "the other end," indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0064] In this document, "and / or" includes any and all combinations of one or more of the listed related items. "A plurality of" in this document means two or more, i.e., it includes two, three, four, five, etc. As used in this specification, the term "about" typically means + / -5% of the value, more typically + / -4%, more typically + / -3%, more typically + / -2%, even more typically + / -1%, even more typically + / -0.5%. In this specification, certain embodiments may be disclosed in a range format. It should be understood that this "range" description is merely for convenience and brevity and should not be construed as a rigid limitation on the disclosed range. Therefore, the description of the range should be considered as having specifically disclosed all possible subranges and independent numerical values ​​within those ranges. For example, a description of the range 1-6 should be considered as having specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., as well as individual numbers within this range, such as 1, 2, 3, 4, 5, and 6. The above rules apply regardless of the breadth of the range.

[0065] In this article, "musical score" or "musical notation" refers to a regular combination of written symbols used to record musical pitch or rhythm. Common examples of musical scores include numbered musical notation and staff notation. Taking the piano as an example, the musical score used on the piano is usually staff notation. Staff notation is a method of recording music by marking notes of different durations and other symbols (i.e., written symbols) on five equally spaced parallel horizontal lines. Each line of the staff and the spaces between the lines are, from bottom to top, called the first line, second line, third line, fourth line, fifth line, and first space, second space, third space, fourth space. If there are not enough lines and spaces, additional lines and spaces can be added above or below the staff. These ledger lines and spaces are called ledger first line, ledger first space, ledger first line, ledger first space, etc., each representing a pitch. The fixed pitch of these pitches is determined by the clef used.

[0066] In musical notation, a syllable is the smallest unit of sound that can be produced independently, used to represent rhythm and melody. Syllables are represented by specific symbols (or identifiers) in sheet music, such as notes, rests, and slurs. The length, pitch, and dynamics of each syllable are specifically represented by these symbols. The combination of these symbols constitutes a complete musical work. Generally, a voice part refers to an independent line of sound in polyphonic music. When a work has two or more melodies playing simultaneously, or when there is a vertical harmony composed of two or more notes, each horizontal melody line or each vertical pitch line constituting that harmony can be called a voice part. For example, in four-part harmony writing, the soprano, alto, tenor, and bass parts are four independent voice parts. Similarly, in piano music, the main melody played by the right hand and the bass accompaniment played by the left hand belong to different voice parts.

[0067] In other words, syllables are the basic elements that make up a voice part. Different combinations of syllables form different melodic lines, which in turn constitute different voice parts. Each voice part is composed of a series of orderly arranged syllables, which are arranged according to specific rhythms and pitches to form a unique melody.

[0068] Currently, mainstream music editing software typically uses MusicXML files to digitize and edit sheet music data. However, for sheet music without digitized data, such as new scores, users often need to manually input the data into the software. This manual input method is inefficient and has a high error rate. While some AI network models for recognizing digitized sheet music have been proposed, these methods are limited in their ability to recognize diverse data types, making them difficult to integrate with mainstream music editing software platforms. Furthermore, as sheet music becomes more complex, involving multiple layers of notes, handwritten scores, and complex symbols, traditional AI network models exhibit significant shortcomings in accuracy and reliability.

[0069] Referring to Figure 1, the present invention provides a music score recognition method, comprising:

[0070] S101, Construct a target detection model. Preferably, the target detection model includes, in sequence:

[0071] The input terminal is used to receive a musical score image (e.g., a photograph taken).

[0072] A feature extraction network is used to extract feature information from the musical score image; the feature information can be the type of feature and the position of the feature on the musical score.

[0073] In this embodiment, "feature" refers to an identifier (or written symbol, or simply "symbol") used in the musical score to express music theory information. For example, the identifier can be a note, a horizontal line, a rest, a clef, a text mark, etc. In this embodiment, the feature information includes: (1) basic information, which includes one or more of the following: note information, clef information, key signature information, slur information, and polyphonic information; (2) technical information, which includes one or more of the following: performance technique information and fingering information; (3) emotional style information, which includes one or more of the following: expression marking information, dynamic marking information, and time signature information.

[0074] Generally, a musical note is a symbol used to record the progression of sounds of different durations. For example, note information can include the note's duration, pitch, and position. A clef is a symbol that identifies the specific pitch of each line (space), and it is typically divided into three categories: F clef, C clef, and G clef. Clef information can include the clef type and its position. A key signature is a symbol indicating the key range used in a piece of music. Slurs can be divided into two types: tie lines and legato lines. Multivoice can be identified using symbols such as slurs. Playing techniques can refer to staccato, sustain, arpeggios, glissando, accents, appoggiaturas, and turnes. Fingering can include staccato markings, sustain markings, glissando markings, accent markings, trill markings, tremolo markings, and repeat markings (such as DC (repeat from the beginning) and DS (repeat from S)). For example, expression markings can be tempo markings, embellishment markings, tonality, tempo terms, touch terms, expression terms, etc., used to reflect playing style. Dynamic markings are identifiers used to indicate the strength of a playing sound (for example, they can include one or more of the following: pp (very soft), mp (medium-pian), mf (medium-forte), ff (very strong)). Time signatures are identifiers used to reflect the periodicity of the dynamic pattern of notes.

[0075] For example, in some embodiments, the identified feature information may not include note information; that is, notes can be identified separately using a separate object detection model. In other embodiments, the identified feature information includes note information to improve the model's ability to distinguish between different types of identifiers. In other words, in some embodiments, S101 may provide two object detection models: one for detecting note information and the other for detecting other feature information.

[0076] A feature fusion network is used to fuse the feature information and output a feature map accordingly.

[0077] An object detection network is used to identify the feature map according to the corresponding minimum recognition unit, so as to classify and output the fusion result, wherein the minimum recognition unit is the minimum resolution by which the object detection network identifies the feature map.

[0078] For example, an object detection network may sequentially include at least two feature detection layers, and the size of the smallest recognition unit in the at least two feature detection layers gradually increases.

[0079] Preferably, the target detection network may sequentially include: a first feature detection layer, a second feature detection layer, a third feature detection layer, and a fourth feature detection layer connected in series. The first feature detection layer performs target detection on the feature map based on a first minimum recognition unit; the second feature detection layer performs target detection on the feature map based on a second minimum recognition unit; the third feature detection layer performs target detection on the feature map based on a third minimum recognition unit; and the fourth feature detection layer performs target detection on the feature map based on a fourth minimum recognition unit. The size of the first minimum recognition unit can be 4 pixels × 4 pixels to 8 pixels × 8 pixels, and the sizes of the first minimum recognition unit, the second minimum recognition unit, the third minimum recognition unit, and the fourth minimum recognition unit gradually increase in size.

[0080] Therefore, by setting the first to fourth feature detection layers, this embodiment can sequentially identify information from small targets to large targets in the feature map during the downsampling process, and then further fuse the identified large target information and small target information through the upsampling process.

[0081] Ultimately, the output detection result includes a data stream formed by multiple feature sequences. Each feature sequence includes: the type of feature information, the location of the feature information, and the confidence level of the feature information. The confidence level describes the reliability of the identification results for the type and location of the feature information.

[0082] S102, the sheet music image to be identified is input into the target detection model, and the target detection network outputs the corresponding detection result. The detection result is a sequence of multiple feature features from the sheet music image (such as a photograph of the sheet music).

[0083] In some embodiments, multiple feature detection layers can be combined with a feature pyramid network (FPN) to construct feature maps at different scales and fuse them using lateral connections, thereby achieving effective detection of targets at different scales.

[0084] Further, in some embodiments, the selectable range of the second minimum recognition unit of the second feature detection layer is 8×8-16×16 pixels. In some embodiments, the selectable range of the third minimum recognition unit of the third feature detection layer is 16×16-32×32 pixels. In some embodiments, the selectable range of the fourth minimum recognition unit of the fourth feature detection layer is 32×32-64×64 pixels. It is acceptable as long as the size of the minimum recognition unit gradually increases.

[0085] It is worth noting that in this embodiment, the feature extraction network of the target detection model comprehensively and deeply extracts data from the three dimensions of the musical score: basic information, technical information, and emotional style information. This data is then uniformly output as a standardized feature sequence (or standardized data stream) at the output of the target detection model (i.e., the output of the target detection network). Furthermore, this feature sequence is represented by three dimensions: feature type, position, and confidence level.

[0086] The comprehensive collection of basic information, technical information, and emotional style information helps to identify the core information of the score. At the same time, information such as type, position, and confidence level helps to completely reconstruct and restore the score, so as to complete the reverse reconstruction from feature sequence to score file (such as MusicXML file).

[0087] Preferably, in some embodiments, the object detection network can support the recognition output of at least 136 classes of notes and symbols (equivalent to identifiers).

[0088] In some embodiments, after inputting the captured musical score image (i.e., the image of the musical score to be identified) into the target detection model and outputting a standardized data stream, the user can choose to compare this data stream with musical score data in an existing database (e.g., standard musical scores with pre-recorded information) to select the standard musical score that matches the musical score image. For example, in some embodiments, when the musical score image does not have a corresponding standard musical score, a standard MusicXML file can be directly reconstructed from the data stream.

[0089] Furthermore, it is worth noting that in this embodiment, the recognition result (i.e., the type and location of the feature) and the recognition reliability (i.e., confidence level) are preferably used for dual output, which also facilitates the improvement of the comparison reliability during the musical score comparison process.

[0090] For example, in some embodiments, the similarity between the data stream and the standard sheet music can be calculated to perform sheet music comparison, and features with different confidence levels can have different weights in the similarity calculation process. Furthermore, the present invention can also match the photograph with the standard sheet music in the database based on the detection results. For example, in some embodiments, cosine similarity or deep learning-based contrastive learning methods can be used to perform sheet music photograph and standard sheet music surface matching, outputting the optimal matching result.

[0091] For example, the CLIP model can be used to calculate the predicted feature values ​​of the musical score image based on the feature sequence; and the predicted feature values ​​can be used to query a pre-stored musical score database to see if there is matching standard musical score data. The musical score database includes multiple standard musical score data sets, and each musical score data set is associated with a feature value.

[0092] Preferably, the target detection model in this embodiment adopts the YOLOv5 network structure, and a first feature detection layer is introduced into the target detection network of the YOLOv5 network structure to identify small-scale targets (i.e., small targets) in the musical score, while a second to fourth feature detection layer is introduced to identify large-sized targets (i.e., large targets) in the musical score; thereby ensuring effective classification and recognition of multi-dimensional information in the musical score. Simultaneously, a Coordinate Attention (CA) mechanism is introduced after concat in the feature fusion network to guide the model to focus more on long-distance information and small target information in the musical score, completing in-depth capture and learning of multi-dimensional information. Preferably, a multi-layer convolutional neural network (CNN) can be introduced to classify the detected target regions. Preferably, the first to fourth feature detection layers can respectively adopt P2 feature layers to P5 feature layers.

[0093] In some embodiments, the model structure uses YOLOv5, and the network can be optimized to incorporate the characteristics of the musical score, adding a feature pyramid module to enhance the detection capability of small targets (such as ornaments). In some embodiments, image feature extraction algorithms (such as SIFT and ORB algorithms) can be used in the feature extraction network to extract key feature points from the musical score image. During the application of this recognition method, an image acquisition device (such as a mobile phone, tablet, camera, etc.) can be used to photograph the musical score. The photograph is then input into the target detection model, which outputs the detection result of the photograph, i.e., the feature sequence.

[0094] In some embodiments, the sheet music image uploaded by the user can be one or more of the following formats, such as JPEG, PNG, and PDF.

[0095] Furthermore, in order to improve the accuracy of the detection results, the present invention also provides a method for enhancing the details of musical scores in photographs, so as to reduce the number of interfering pixels in the photographs to a large extent.

[0096] For example, in some embodiments, referring to Figure 2, before S102, the method further includes the step of: performing detail feature enhancement processing on the sheet music image (i.e., sheet music photograph) to be identified; wherein, the detail feature enhancement processing includes the following steps:

[0097] S1021, The music score image to be identified is processed into grayscale to obtain a grayscale image;

[0098] S1022, The grayscale image is binarized to obtain a binary image;

[0099] S1023, Perform a shadow removal operation on the binarized image to obtain a shadow-free image;

[0100] S1024, identify abnormal points in the deshaded image, and perform a contrast enhancement operation to weaken the intensity of the visual features of the abnormal points (i.e., perform a feature weakening operation) to obtain a contrast-enhanced image.

[0101] In some embodiments, the contrast enhancement operation may employ one or more of the following methods: histogram equalization, linear transformation, histogram matching, and adaptive histogram equalization.

[0102] In this embodiment, visual features refer to the visual characteristics of points (such as outliers or target points), such as color (e.g., pixel value) and shape. The intensity of the corresponding visual features refers to the shade of color or the size of the shape (e.g., the number of pixels). Reducing the intensity of the visual features of outliers helps to decrease the probability of the target detection model falsely detecting outliers.

[0103] For example, reducing the intensity of a visual feature means decreasing the depth of a color. Or, reducing the intensity of a visual feature means decreasing the size of a shape (e.g., reducing the number of pixels).

[0104] In this embodiment, the abnormal points may include: a type of point, and the interval between the type of point and the position of the target feature in the deshaded image is greater than a first interval threshold. The target feature includes one or more of the following: musical notes, clefs, rests, fermata, staccato, dynamics, and playing techniques.

[0105] In other words, a type of point usually refers to points that do not belong to the target feature. Or, a type of point usually refers to points that are a certain distance away from the normal target feature.

[0106] In this embodiment, feature weakening operations are performed on anomaly points in specific types of written symbol regions. This allows for the concentrated removal of major anomalies (e.g., impurities / stains in the original score file due to printing equipment or improper storage, or interference pixels generated during binarization and deshading processes) based on a limited amount of weakened data.

[0107] It is worth noting that the "centralization" in this embodiment is reflected in at least two levels: 1. At the level of operation steps, after performing binarization and shadow removal operations, abnormal points in the de-shadowed image are removed in a concentrated manner; 2. At the level of image processing area, abnormal points in a limited area of ​​a limited type of written symbols are removed in a concentrated manner.

[0108] From another perspective, this concentrated feature weakening operation can, on the one hand, remove existing abnormalities in the musical score and interference factors during image processing, reducing the problem of residual interference factors. On the other hand, the applicant noted that focusing on the removal of limited written symbol areas can ensure the accuracy of subsequent photo processing (i.e., remove the main interference factors) while reducing the processing pressure on abnormal data (i.e., reducing the amount of data processing).

[0109] Understandably, in the process of building an object detection model, a large sample set is needed to optimize and train the model. The sample set can be sample photos of multiple musical scores. Correspondingly, the sample photos can also be preprocessed using the concentrated feature weakening operation of this invention before being input into the model for training.

[0110] In other words, the present invention also provides a separate method for preprocessing photographs, which may include steps S1021-S1024.

[0111] In some embodiments, the outlier points further include: binary points; correspondingly, the step of identifying outlier points in the deshaded image includes:

[0112] Identify target points in the deshaded image. A target point is a set of consecutive pixels whose pixel value is less than a set threshold. Target points include outliers and valid points. Valid points are written symbols on a musical score whose size is less than a set size threshold; for example, they can be small written symbols such as bass notes, treble notes, or dotted notes.

[0113] When the position (or interval) between the target point and the corresponding target feature in the deshaded image is less than or equal to a first interval threshold, the target point is identified as an associated point of the target feature.

[0114] When the number of associated points of the target feature is greater than the target number, a second-class point (i.e., an outlier) will be selected from the associated points. Correspondingly, one or more other associated points can be selected as valid points.

[0115] In other words, in this embodiment, real valid points and false valid points (such as outliers) can be identified uniformly first, and then filtered to select the real valid points. That is, the target point can be the total set of points that are not distinguished as real or false.

[0116] For example, in some embodiments, in a binarized image, each pixel value is between 0 and 255. 0 represents black, and 255 represents white. Therefore, the larger the pixel value, the brighter the color; the smaller the pixel value, the darker the color. When multiple consecutive pixels in an image are all less than a set threshold, these multiple consecutive pixels can be identified as a single target point.

[0117] For example, in some embodiments, the target point may typically include anomalies.

[0118] In this embodiment, the identification and processing of associated points can filter out abnormal points caused by printing or contamination issues from the target points. Furthermore, by collaboratively analyzing the location of target features and the number of target points, this embodiment can centrally process abnormal points with a high degree of interference in key areas, thereby improving the accuracy of abnormal point identification while reducing the burden on data processing.

[0119] In some embodiments, the step of selecting a binary point from the associated points includes:

[0120] Calculate the intensity difference of a visual feature between at least two associated points that are associated with the same target feature; wherein the visual feature may include one or more of the following: grayscale, brightness, and size;

[0121] When the intensity difference is greater than the set intensity threshold, the associated point with the lower intensity of the visible feature is selected as a Class II point.

[0122] Furthermore, in some embodiments, when the intensity difference is less than or equal to the intensity threshold, a user input instruction is received. This input instruction is used to define a Class II point (i.e., an anomaly) from at least two associated points. In other words, in this embodiment, when the differences in visual features between valid points and anomalies are small, the anomaly is manually defined by the user through a manual input instruction.

[0123] Preferably, in this embodiment, the image can be processed using a semi-automatic human-machine collaborative recognition method to address the interference level of outliers. For example, when the difference between outliers and valid points is large, the distinction between outliers and valid points can be automated, while when the difference between outliers and valid points is small, the image can be forcibly converted to a user-defined mode.

[0124] In this embodiment, the distinction between outliers and valid points is further enhanced by introducing the visual feature intensity of points for semi-automatic differentiation, taking into account the location of the feature target and the number of points. This semi-automatic differentiation scheme can balance overall recognition efficiency and recognition accuracy.

[0125] In some embodiments, when the intensity difference is less than or equal to the intensity threshold, the method further includes the step of:

[0126] By comparing the binarized image and the deshaded image, it is determined whether there are any newly added target points in the original shadow region of the deshaded image, and the target points are marked as newly added points;

[0127] Determine whether the associated point is the newly added point; if so, output a suggestion signal to the user to define the newly added point as a type II point. That is, consider the newly added point to have a high probability of being an anomaly.

[0128] Referring to Figure 3, during the photographing process, shadows may be generated on the sheet music due to occlusion or other reasons. For example, the finger on the left side of Figure 3 will create a shadow area on the left edge of the sheet music, and the photographing device on the right side will also create a large shadow area on the sheet music photograph. These shadow areas will affect the feature recognition accuracy of the target detection model. To address this, the present invention can use binarization and shadow removal operations in combination to weaken or eliminate the shadow areas. However, the applicant notes that the difficulty of shadow removal operations increases significantly in complex areas of the sheet music, and new impurity points may be generated during the shadow removal process. For example, some residual pixels may form on the edges of the shadow area, or some residual pixels may form where the shadow area overlaps with complex areas such as overlapping notes. When the number of these residual pixels is large, it will also bring new technical difficulties to the feature weakening operation.

[0129] To address this, and to improve the efficiency of feature weakening operations—specifically, to enhance the identification efficiency of Class II outliers—this invention assesses the anomaly probability of associated points by integrating the effects of shadow removal operations. This provides suggested signals for user selection processes within a semi-automatic identification mechanism. This reduces the difficulty of manual selection, thereby improving selection efficiency.

[0130] In some embodiments, the method further includes the step of tilt correction on the contrast-enhanced image. In this embodiment, tilt correction is preferably performed after a concentrated feature weakening operation. This helps improve the accuracy of the image reconstruction process while reducing the workload of image processing operations.

[0131] As can be seen from the transition from Figure 3 to Figure 4, the concentrated feature weakening operation in this invention can reliably process photographs. Figure 4 shows the accurate removal of interfering pixels generated during the photographing process, while ensuring that the written symbols in the musical score are preserved completely and clearly.

[0132] It is understood that the musical score images in this invention may include standardized staff notation information or handwritten annotations. For example, for handwritten annotations in the musical score, OCR technology can be used to recognize the handwritten annotations in the photograph, including lyrics, notes, and symbols. Combined with semantic analysis technology, the handwritten annotations are associated with the corresponding positions in the musical score to extract meaningful information (such as performance prompts, modification instructions, etc.). For example, OCR technology combined with a deep learning model can recognize the content of the handwritten annotations. At the same time, semantic segmentation algorithms can be used to locate the annotation positions and semantically associate them with the content of the musical score.

[0133] The beneficial technical effects of this application are as follows: First, this invention proposes a recognition model that comprehensively and deeply captures three dimensions of data, including basic information, technical information, and emotional style information of musical scores, and outputs them in the form of a standardized feature sequence (or standardized data stream) at the output end of the target detection model. The feature sequence is represented by three dimensions of parameters: feature type, position, and confidence.

[0134] Preferably, the first to fourth feature detection layers with gradually increasing minimum recognition units are used to achieve fusion detection from small targets to large targets, so as to ensure that the above-mentioned multi-dimensional musical score feature information is classified and learned in a refined manner.

[0135] Furthermore, the comprehensive collection of basic information, technical information, and emotional style information helps to identify the core information of the score. At the same time, information such as type, position, and confidence level helps to completely reconstruct and restore the score, so as to complete the reverse reconstruction from feature sequence to score file (such as MusicXML file).

[0136] This invention further provides a method for enhancing the details of musical score photographs, thereby significantly reducing interfering pixels in the photographs. Specifically, this invention performs feature weakening operations on outlier points in specific types of written symbol regions, enabling the concentrated removal of major anomalies (e.g., impurities / stains in the original musical score file due to printing equipment issues or improper storage, or interfering pixels generated during binarization and shadow removal processes) based on a limited amount of weakened data.

[0137] It is worth noting that the "centralization" in this invention is reflected in at least two levels: 1. At the level of operation steps, after performing binarization and shadow removal operations, abnormal points in the de-shadowed image are removed in a concentrated manner; 2. At the level of image processing area, abnormal points in a limited area of ​​a limited type of written symbols are removed in a concentrated manner.

[0138] From another perspective, this concentrated feature weakening operation can, on the one hand, remove existing abnormalities in the musical score and interference factors during image processing, reducing the problem of residual interference factors. On the other hand, the applicant noted that focusing on the removal of limited written symbol areas can ensure the accuracy of subsequent photo processing (i.e., remove the main interference factors) while reducing the processing pressure on abnormal data (i.e., reducing the amount of data processing).

[0139] This invention performs collaborative analysis on the location of target features and the number of target points, enabling centralized processing of anomalies with high interference levels in key locations. This improves the accuracy of anomaly identification while reducing the burden of data processing.

[0140] In other words, this embodiment provides a music score recognition method, including:

[0141] S101, Construct a target detection model, wherein the target detection model includes, in sequence:

[0142] The input terminal is used to receive musical score images;

[0143] A feature extraction network is used to extract feature information from the musical score image; wherein the feature information includes at least one of the following: (1) basic information, which includes: note information, clef information, key signature information, slur information, and polyphonic information; (2) technical information, which includes: performance technique information and fingering information; (3) emotional style information, which includes: expression marking information, dynamic marking information, and time signature information;

[0144] A feature fusion network is used to fuse the feature information and output a feature map accordingly.

[0145] An object detection network is used to identify the feature map based on the corresponding smallest recognition unit, so as to classify and output the fusion result;

[0146] The output detection results include: a data stream formed by multiple feature sequences, wherein the feature sequences include: the type of feature information, the location of feature information, and the confidence level of feature information;

[0147] S102, the musical score image to be identified is input into the target detection model, and the target detection network outputs the corresponding detection result.

[0148] Example 2

[0149] Referring to Figure 5, the present invention also provides a music score recognition system corresponding to the above-described music score recognition method, comprising:

[0150] Target model module 101 is used to provide a target detection model, wherein the target detection model includes, in sequence:

[0151] The input terminal is used to receive musical score images;

[0152] A feature extraction network is used to extract feature information from the musical score image. Preferably, the feature information may include: note information, time signature, clef, key signature, playing technique, fingering, tie lines, slurs, and polyphonic information;

[0153] A feature fusion network is used to fuse the feature information and output a feature map accordingly.

[0154] An object detection network is used to identify the feature map based on the corresponding smallest recognition unit, so as to classify and output the fusion result.

[0155] Wherein, the minimum recognition unit is the minimum resolution at which the target detection network recognizes the feature map. Preferably, the target detection network sequentially includes: a first feature detection layer, a second feature detection layer, a third feature detection layer, and a fourth feature detection layer connected in series. The first feature detection layer performs target detection on the feature map based on the first minimum recognition unit, the second feature detection layer performs target detection on the feature map based on the second minimum recognition unit, the third feature detection layer performs target detection on the feature map based on the third minimum recognition unit, and the fourth feature detection layer performs target detection on the feature map based on the fourth minimum recognition unit. Preferably, the size of the first minimum recognition unit is 4 pixels × 4 pixels to 8 pixels × 8 pixels, and the sizes of the first minimum recognition unit, the second minimum recognition unit, the third minimum recognition unit, and the fourth minimum recognition unit gradually increase.

[0156] The output detection results include: a data stream formed by multiple feature sequences, wherein the feature sequences include: the type of feature information, the location of feature information, and the confidence level of feature information;

[0157] The detection output module 102 is used to input the sheet music image to be identified into the target detection model, and the target detection network outputs the detection result accordingly.

[0158] In some embodiments, the image further includes: an image processing module 103, configured to perform detail feature enhancement processing on the sheet music image to be identified; wherein the image processing module includes:

[0159] The grayscale processing unit 1031 is used to perform grayscale processing on the music score image to be identified to obtain a grayscale image;

[0160] Binarization processing unit 1032 is used to perform binarization processing on the grayscale image to obtain a binarized image;

[0161] The shadow processing unit 1033 is used to perform shadow removal operation on the binarized image to obtain a shadow-free image;

[0162] Anomaly processing unit 1034 is used to identify anomalies in the deshaded image and to perform a contrast enhancement operation to reduce the intensity of the visual features of the anomalies, thereby obtaining a contrast-enhanced image.

[0163] The abnormal points include: a type of point, and the interval between the type of point and the position of the target feature in the deshaded image is greater than a first interval threshold. The target feature includes one or more of the following: musical notes, clefs, rests, fermata, staccato, dynamics, and playing techniques.

[0164] In some embodiments, the anomaly points further include: type II points; correspondingly, the anomaly point processing unit 1034 includes:

[0165] The first identification subunit 10341 is used to identify target points in the deshaded image, wherein the target point is a set of continuous pixels with at least one pixel value less than a set threshold.

[0166] The association subunit 10342 is used to identify the target point as the associated point of the target feature when the position of the target point and the corresponding target feature in the deshaded image is less than or equal to a first interval threshold.

[0167] The binary point selection subunit 10343 is used to select a binary point from the associated points when the number of associated points of the target feature is greater than the target number.

[0168] In some embodiments, the second-type point selection subunit 10343 further includes:

[0169] A visual feature comparison subunit is used to calculate the intensity difference of visual features between at least two associated points related to the same target feature; wherein the visual features include one or more of the following: grayscale, brightness, and size;

[0170] The second identification subunit is used to select the associated point with the lower intensity of the visible feature as a second-class point when the intensity difference is greater than the set intensity threshold.

[0171] The third identification subunit is used to receive a user input instruction when the intensity difference is less than or equal to the intensity threshold. The input instruction is used to define a binary point from at least two associated points.

[0172] In some embodiments, the third identification subunit is further configured to compare the binarized image and the deshaded image, determine whether there is a newly added target point in the original shadow region of the deshaded image, and mark the target point as a newly added point; determine whether the associated point is the newly added point; if so, output a suggestion signal to the user to define the newly added point as the binary point.

[0173] In some embodiments, the image further includes a correction unit for tilt correction of the contrast-enhanced image.

[0174] Furthermore, to verify the accuracy of the recognition method provided by this invention (i.e., different target detection models separately identify musical notes and non-musical note identifiers), this invention also conducted reliability tests using a selected test set. The test results are shown in Table 1:

[0175] Table 1 - Comparison of Identification Accuracy for Different Sample Types

[0176] Furthermore, the feature sequences identified from the sheet music photos are matched with pre-stored standard sheet music. The standard sheet music contains pre-recorded feature sequence information; a match is considered complete when the overlap of the feature sequences of two sheet music photos exceeds a matching threshold. Therefore, the sheet music recognition technology proposed in this invention allows users to efficiently retrieve corresponding standard sheet music data from a massive sheet music database using only sheet music photos. Verification has shown that the matching accuracy between sheet music photos and standard sheet music exceeds 95%.

[0177] This exemplary object detection model employs the YOLOv5 network structure, and uses the P2-P5 feature layers within the YOLOv5 network to identify both small and large objects. Furthermore, a Coordinate Attention (CA) mechanism is introduced after the concat function in the feature fusion network.

[0178] Below, this invention proposes a method for reconstructing standardized musical score files (such as musicXML files) based on the feature sequence of the smallest unit (i.e., the semantic detection result of the first-level semantic element). This reconstruction method, in conjunction with musical score recognition methods, can greatly assist users in rapidly converting musical score images into standardized musical score files.

[0179] It is worth noting that the music score recognition technology and reconstruction method in this invention can serve as a MusicXML conversion interface, thereby enabling it to interface with many mainstream music editing software programs, facilitating users to electronically edit any paper music score.

[0180] Example 3

[0181] Referring to Figure 6, the present invention provides a method for reconstructing musical scores, which may include the following steps:

[0182] S201, providing a musical score (preferably a photograph of the musical score), the musical score including a plurality of first ideographic elements;

[0183] For example, when a photograph of a musical score is provided, the first semantic element contained in the photograph can be identified through an object detection model.

[0184] S202, identify the first ideographic form of the first ideographic element. The first ideographic form may also be associated with a confidence level. The first ideographic form can express the meaning of at least one musical note element in the musical score, and the first ideographic form includes one or more of the following: shape, position, quantity, and direction. The musical note element includes one or more of the following: duration, dynamics, pitch, volume, rhythm, meter, tonality, and mode. The confidence level is used to indicate the accuracy of identifying the first ideographic form.

[0185] In this embodiment, the first semantic element can be acquired using a pre-trained AI recognition model (such as an object detection model). The detection sequence output by the AI ​​recognition model includes the type, position, and confidence level of the first semantic element. Based on the type and / or position of the first semantic element, information about the musical element labeled by the first semantic element can be determined.

[0186] S203, using the first evaluation rule to determine whether the first ideographic element is a first-level ideographic element; the first evaluation rule requires that when a first ideographic element must be connected and drawn (or drawn continuously) with another first ideographic element on the score to form a complete identifier (also called a second ideographic element), then the corresponding first ideographic element is considered to be the first-level ideographic element.

[0187] If so, the first-level ideographic elements used to form a complete identifier are grouped into a subset of ideographic elements.

[0188] In this embodiment, "connected drawing" refers to at least two pixels of the first ideographic element being connected on the musical score image (e.g., if the distance between pixels is less than the set drawing interval, it is considered to be connected). In this embodiment, a typical complete identifier is a musical note, such as a whole note, half note, quarter note, etc.

[0189] For example, when two or more primary ideographic elements must be connected together to form a complete identifier (or when the primary ideographic element cannot be presented alone in the score), then the two or more primary ideographic elements can be defined as first-level ideographic elements. For example, when the primary ideographic element is a notehead, it needs to be drawn in connection with the stem and note tail.

[0190] For example, in some embodiments, the category of the first ideographic element can be determined by recognizing its shape, and then it can be determined whether the first ideographic element belongs to a first-level ideographic element.

[0191] S204, Identify at least two termination lines from a plurality of the first semantic elements;

[0192] S205, the plurality of first ideographic elements are classified into a plurality of ideographic element sets according to the termination lines, wherein the first ideographic elements located between two adjacent termination lines are classified into one ideographic element set. Furthermore, each ideographic element set may include at least one subset of ideographic elements.

[0193] It is understandable that, in some embodiments, the classification of ideographic element subsets can be performed after the classification of the ideographic element set. Preferably, a ideographic element subset is assigned to a ideographic element set.

[0194] In other words, in this embodiment, before converting the data results obtained based on the AI ​​model into musicXML files, numerous ideographic elements can first be grouped (i.e., pre-classified) based on their type and measure information (such as the ideographic form of the end line). This pre-classification method not only conforms to general music theory rules but also helps to generate XML elements based on each set in a standardized manner, reducing the pressure on the data conversion process.

[0195] In some embodiments, the step of: identifying at least one subset of the ideographic elements is further included;

[0196] Obtain the first ideographic form (i.e., ideographic information) of multiple first-level ideographic elements in the subset of ideographic elements, and determine the identifier type of the complete identifier according to the corresponding first ideographic form.

[0197] The identifier type and the musical element of the complete identifier are recorded in the corresponding set of ideographic elements.

[0198] For example, in some embodiments, the identifier type of the subset of ideographic elements and the corresponding musical elements can be recorded before generating the electronic data sequence.

[0199] For example, in some embodiments, to facilitate the conversion of musicXML files, it is preferable to determine the identifier type of the identifier represented by the identifier based on the semantic information of the subset of semantic elements. The data in the original set of semantic elements is then replaced with the identifier type and the corresponding musical element (such as duration, pitch, dynamics, etc.).

[0200] Taking a quarter note as an example, the data information of the ideographic element subset is modified to the quarter note type.

[0201] Referring to Figures 7-9, which illustrate the identification methods of some identifiers 10, Figure 8 shows a schematic diagram of the detection results of a musical score. This score includes multiple identifiers 10, and each identifier 10 can be composed of one or more first ideographic elements. For example, identifier 10 can be composed of first ideographic elements A11 and B12, and each of the first ideographic elements A11 and B12 records a confidence level of 20. As shown in Figures 8 and 9, the numerical markers displayed in the images are used to represent the type of ideographic element and the confidence level, respectively.

[0202] It is important to note that in complex musical scores, multiple ideographic elements may be drawn adjacently or overlappingly. This complex arrangement poses a significant challenge to the reliable identification of these elements. To address this, the identification method based on the smallest unit (i.e., the first ideographic element) proposed in this invention can effectively improve the accuracy of score recognition. Here, the smallest unit refers to the shape of a musical note element that can be stored independently in its ideographic form (i.e., the smallest decomposition unit of the identifier).

[0203] In some embodiments, the method further includes the step of: S207, verifying the completeness of the semantic information of the semantic element set using a time-value evaluation model. S207 includes:

[0204] S2071, Identify beat information from the set of semantic elements, and determine the expected musical duration of the set of semantic elements based on the beat information;

[0205] S2072, Identify a first type of time value ideographic element and a second type of time value ideographic element from the ideographic elements. The first type of time value ideographic element includes one or more of the following types: rest, note, dot. The second type of time value ideographic element includes the following type: slur.

[0206] S2073, determine the first performance duration based on the first type of time value representation element, and determine the second performance duration based on the second type of time value representation element;

[0207] S2074, determine whether the expected musical duration, the first performance duration, and the second performance duration satisfy the time value evaluation model, wherein the time value evaluation model includes: first performance duration - second performance duration = expected musical duration;

[0208] If yes, it is preliminarily considered that the semantic information of the set of semantic elements is complete; if not, it will be (prompted) to supplement the semantic information of the set of semantic elements.

[0209] For example, in some embodiments, a rest signifies a pause, a note signifies a playing time, and a dot signifies a sustained duration. Correspondingly, the combined duration represented by one or more identifiers such as rests, notes, and dots can be statistically analyzed to determine the first performance duration. The duration of a slur is calculated by dividing the note into n equal parts, resulting in an n-slur. The duration of the slur can be calculated according to fixed rules, and this duration can be statistically analyzed to determine the second performance duration. The duration of the musical note can be set by the time signature of the measure.

[0210] In this embodiment, beat information can be selected from the time value dimension and combined with one type of time value semantic element and two types of time value semantic elements to evaluate the completeness of the semantic information of the semantic element set. This can, to some extent, compensate for the differences between the sheet music format in the photograph and the electronic digital format, improving the reliability and accuracy of the electronic digital format conversion process.

[0211] Furthermore, to address the problem of incomplete semantic information in the set of semantic elements, this invention also provides a semi-automatic supplementation mechanism. For example, the supplementation mechanism can automatically generate rest supplementation information by combining the confidence level and time value difference of the two types of time-value semantic elements, and can force users to manually judge the supplementation information when the confidence level is low.

[0212] In some embodiments, the step of supplementing the semantic information of the semantic element set includes:

[0213] (1) Obtain the first confidence level of the first type of time value representation element and the second confidence level of the second type of time value representation element; (2) Determine whether the first confidence level and the second confidence level are both greater than the set first confidence value;

[0214] If so, then execute the following steps: (3) Obtain the time difference, where time difference = musical duration - (first performance duration - second performance duration); (4) Generate at least one supplementary rest information based on the time difference;

[0215] For example, in some embodiments, supplementary rest information can be the type of rest and / or the number of rests. It is understood that this supplementary rest information can be automatically generated using preset music theory rules.

[0216] If not, then proceed to step (5) output a prompt signal to prompt the user to correct the semantic information of the set of semantic elements.

[0217] For example, in some embodiments, the output prompt signal includes at least one supplementary pause information (i.e., an automatic recommendation result) generated based on the time value difference. Correspondingly, the user can select correction information based on the automatic recommendation result. For example, the user can choose to receive the recommended supplementary pause information, or they can manually enter other pause information.

[0218] In this embodiment, when the confidence level of the time value representation element is high, the corresponding rest information can be automatically supplemented based on the time value difference. When the confidence level of the time value representation element is low, manual information correction by the user will be introduced.

[0219] In this embodiment, by combining the two dimensions of time and confidence, information can be quickly supplemented for a set of semantic elements with incomplete semantic information, so as to ensure that the set of semantic elements can be converted into a standard musicXML file according to music theory rules (as shown in Figure 10), avoiding conversion errors caused by format mismatch.

[0220] Furthermore, this invention can also perform sequence splitting on complex spectral elements to improve the conversion accuracy of XML files. Specifically, this invention can split semantic elements spanning multiple sections into different sets.

[0221] For example, in some embodiments, the steps include:

[0222] Get the first position of the first semantic element;

[0223] Based on the first position, determine whether the first ideographic element crosses at least one termination line;

[0224] If so, the first ideographic element is decomposed into at least two ideographic sub-elements according to the corresponding termination line, and the second position of the ideographic sub-elements is recorded; the at least two second ideographic sub-elements are classified into the corresponding sets of at least two ideographic elements according to the second position.

[0225] For example, in some embodiments, in S205, the first ideographic element can be disassembled and classified according to whether it crosses the termination line.

[0226] In this embodiment, the first semantic element that spans multiple sections will be pre-splitled to be categorized into different semantic element sets, thereby further simplifying the automated generation of XML elements and improving the accuracy of digital transformation.

[0227] Furthermore, in some embodiments, the method further includes the step of: determining whether the first ideographic element is an auxiliary ideographic element, wherein the auxiliary ideographic element is an ideographic element used to supplement auxiliary information for the musical note;

[0228] If so, then perform the following steps: obtain the first position of the first ideographic element;

[0229] Identify at least one first semantic element in a third position associated with the first position;

[0230] The auxiliary information is associated with the corresponding first semantic element.

[0231] In some embodiments, the third position refers to the area covered by a predetermined length extending downward from the first position. Specifically, this predetermined length can be preset by the display scale of the sheet music or the size of the sheet music identifier.

[0232] In some embodiments, the auxiliary expressive elements include one or more of the following: slurs, tie lines, rests, staccato marks, sustain marks, lute marks, sharps / flats, and ornaments. For example, ornaments may include one or more of the following: trill marks, mordent marks, and appoggiaturas. Typically, the type definition of auxiliary expressive elements can be set by the user in conjunction with music theory knowledge.

[0233] In some embodiments, auxiliary ideographic elements may cover multiple musical notes. In this embodiment, the auxiliary ideographic information of the auxiliary ideographic elements can be pre-assigned and associated with multiple first ideographic elements to further strengthen the data association within the ideographic element set, thereby improving the accuracy of XML element generation.

[0234] In an exemplary embodiment, the process of constructing an XML file from the detection sequence of the first semantic element is as follows:

[0235] Obtain the set of semantic elements obtained from the classification;

[0236] For each set of ideographic elements, a Measure object is created to represent each ideographic element. Information from the set of ideographic elements is then inserted into the Measure object. Each Measure object contains all the necessary musical elements for that set of ideographic elements (i.e., a measure). Musical attributes such as time signature, key signature, and clef can be inserted into the Measure object.

[0237] For each Measure object, a first XML element is generated for each note, including pitch and duration information, and a second XML element is generated for each rest. Other special symbols, such as gestalts and slurs, can also be processed to generate a third XML element.

[0238] Furthermore, once all the ideographic elements have been converted into their corresponding XML elements, an XML document can be created. For example, necessary information, such as the title and author, can be added to the document header. Simultaneously, the XML elements for all sections are inserted into the document in sequence. Finally, the generated MusicXML document is saved to the file system. A validation tool can then be used to check if the file format is correct.

[0239] In some embodiments, S202 includes the step of:

[0240] S2021, the first target detection model (equivalent to an AI recognition model) is used to identify the first level of semantic elements;

[0241] The first target detection model includes:

[0242] The input terminal is used to receive a first musical score image;

[0243] A feature extraction network is used to extract first feature information from the musical score image; wherein, the first feature information includes: pixel information of a first-level ideographic element;

[0244] A feature fusion network is used to fuse the first feature information and output a first feature map accordingly.

[0245] An object detection network is provided, which is used to identify the first feature map according to the corresponding minimum recognition unit, so as to classify and output the fusion result. Preferably, the minimum recognition unit is the minimum resolution by which the object detection network identifies the first feature map. Correspondingly, the object detection network includes, in sequence, a second feature detection layer, a third feature detection layer, and a fourth feature detection layer connected together.

[0246] Preferably, the minimum recognition units of the second to fourth feature detection layers gradually increase in size. For example, in some embodiments, the selectable range of the second minimum recognition unit of the second feature detection layer is 8×8-16×16 pixels, the selectable range of the third minimum recognition unit of the third feature detection layer is 16×16-32×32 pixels, and the selectable range of the fourth minimum recognition unit of the fourth feature detection layer is 32×32-64×64 pixels.

[0247] S2022, The second target detection model is used to identify the second-level first ideographic element, wherein the second-level first ideographic element is an ideographic element that can be drawn independently in the musical score;

[0248] Preferably, the second target detection model includes:

[0249] The input terminal is used to receive musical score images;

[0250] A feature extraction network is used to extract second feature information from the musical score image; wherein the second feature information includes: information of a second-level first semantic element;

[0251] A feature fusion network is used to fuse second feature information and output a second feature map accordingly.

[0252] An object detection network is used to identify the second feature map according to the corresponding minimum recognition unit, so as to classify and output the fusion result. The minimum recognition unit is the minimum resolution for the object detection network to identify the second feature map. Correspondingly, the object detection network includes, in sequence, a first feature detection layer, a second feature detection layer, a third feature detection layer and a fourth feature detection layer connected to each other.

[0253] The size of the first minimum recognition unit in the first feature detection layer is 4×4 to 8×8 pixels, and the size of the minimum recognition unit gradually increases from the first feature detection layer to the fourth feature detection layer. Preferably, the selectable range of the second minimum recognition unit in the second feature detection layer is 8×8-16×16 pixels, the selectable range of the third minimum recognition unit in the third feature detection layer is 16×16-32×32 pixels, and the selectable range of the fourth minimum recognition unit in the fourth feature detection layer is 32×32-64×64 pixels.

[0254] In this embodiment, the configuration of the second to fourth feature networks can improve the learning depth of the target detection network for detailed units of the same major category of identifiers (such as musical notes), while the combination of the first to fourth feature detection networks can improve the learning breadth of the data for identifiers of different major categories. Therefore, by focusing on different aspects of detail depth and scope when learning primary and secondary semantic elements, the overall recognition accuracy of musical score images can be increased, while simultaneously improving model construction and detection efficiency.

[0255] It is understood that the first target detection model in this embodiment can be the target detection model constructed in the music score recognition method described in Embodiment 1.

[0256] Figure 7 illustrates the detection results of the first object detection model. Figure 7 shows the morphological structure of two sixteenth notes. In the process of recognizing these two sixteenth notes, they are decomposed into the following basic elements: first notehead 11a, second notehead 11b, first stem 12a, second stem 12b, first short horizontal line 13a, and second short horizontal line 13b. Correspondingly, the detection results (or detection sequence) output by the object detection network are: notehead, position 1, confidence 1; notehead, position 2, confidence 2... short horizontal line, position 5, confidence 5; short horizontal line, position 6, confidence 6. The position of the notehead on the staff indicates the pitch. The stem connects the notehead and the flag, and can be upward or downward. The direction of the stem determines the note value and rhythm. When the stem is upward, it is written to the right of the notehead; when downward, it is written to the left. The flag is a curved mark extending from the stem, usually one or more flags. The number and direction of the flags vary depending on the duration of the note.

[0257] Preferably, in some embodiments, the first object detection model can be a YOLOv model, such as an instance segmentation model. Preferably, the first musical score image is a sample photograph, which can be obtained through the following steps: acquiring electronic data of the musical score (e.g., a musicXML file); generating a staff image with a staff format based on the electronic data, displaying the notes in the staff format with a first color and other ideographic elements with a second color, wherein the first and second colors are different; and taking a photograph of the staff image to serve as the sample photograph.

[0258] Example 4

[0259] The present invention also provides a music score reconstruction system, comprising:

[0260] The music score acquisition module is used to provide music scores, which include multiple first ideographic elements;

[0261] The semantic element recognition module is used to recognize the first semantic form of the first semantic element, and the first semantic form is also associated with a confidence level. Preferably, the first semantic form can express the meaning of at least one musical sound element in the musical score, and the first semantic form includes one or more of the following: shape, position, quantity, direction, and the musical sound element includes one or more of the following: duration, dynamics, pitch, volume, rhythm, meter, tonality, and mode, and the confidence level is used to represent the recognition accuracy of the first semantic form;

[0262] The ideographic element evaluation module is used to determine whether the first ideographic element is a first-level ideographic element using a first evaluation rule. The first evaluation rule requires that when a first ideographic element must be connected with another first ideographic element on the musical score to form a complete identifier, the corresponding first ideographic element is considered to be the first-level ideographic element. If so, the first-level ideographic elements used to form the complete identifier are assigned to a subset of ideographic elements.

[0263] The termination line identification module is used to identify at least two termination lines from a plurality of the first semantic elements;

[0264] The semantic element classification module is used to classify multiple first semantic elements into multiple semantic element sets according to the termination line, wherein the first semantic elements located between two adjacent termination lines are classified into one semantic element set, and one semantic element set includes at least one semantic element subset.

[0265] The present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the music score reconstruction method or music score recognition method as described in any embodiment of the present invention.

[0266] It is understood that the step numbers in any embodiment of the present invention are merely for descriptive convenience and are not a limitation on the order of steps. It is also understood that the music score recognition system of the present invention can implement any of the above methods or steps, and the methods or steps of different embodiments can be combined arbitrarily as long as there is no conflict; these will not be elaborated further here.

[0267] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware, but in many cases the former is a preferred implementation. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a computer terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0268] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for reconstructing musical scores, characterized in that, Including the following steps: S201, providing a musical score, the musical score including a plurality of first ideographic elements; S202, identify the first ideographic form of the first ideographic element, and the first ideographic form is also associated with a confidence level; wherein, the first ideographic form can express the meaning of at least one musical sound element in the musical score, the first ideographic form includes one or more of the following: shape, position, quantity, direction, the musical sound element includes one or more of the following: duration, dynamics, pitch, volume, rhythm, meter, tonality, mode, and the confidence level is used to represent the recognition accuracy of the first ideographic form. S203, the first evaluation rule is used to determine whether the first ideographic element is a first-level ideographic element; the first evaluation rule requires that when a first ideographic element must be connected with another first ideographic element on the musical score to form a complete identifier, the corresponding first ideographic element is considered to be the first-level ideographic element; if so, the first-level ideographic element used to form a complete identifier is assigned to a subset of ideographic elements. S204, Identify at least two termination lines from a plurality of the first semantic elements; S205, classify the plurality of first ideographic elements into a plurality of ideographic element sets according to the termination line, wherein the first ideographic element located between two adjacent termination lines is classified into one ideographic element set, and one ideographic element set includes at least one ideographic element subset. S206, Generate an electronic data sequence for describing the musical score based on the plurality of said ideographic element sets.

2. The method according to claim 1, characterized in that, It also includes the following steps: Identify at least one subset of the ideographic elements; determine the identifier type of the complete identifier based on the first ideographic form of the plurality of first-level ideographic elements in the subset of the ideographic elements; record the identifier type and the musical element of the complete identifier in the corresponding subset of the ideographic elements.

3. The method according to claim 2, characterized in that, The method also includes step S207, which verifies the completeness of the semantic information of the semantic element set using a time-value evaluation model; wherein, S207 includes: S2071, Identify rhythm information from the set of signifying elements, and determine the expected musical note duration of the set of signifying elements based on the rhythm information; S2072, Identify a first type of time value signifying element and a second type of time value signifying element from the signifying elements, wherein the first type of time value signifying element includes one or more of the following types: rest, note, dotted; and the second type of time value signifying element includes: slur; S2073, determine a first performance duration based on the first type of time value signifying element, and determine a second performance duration based on the second type of time value signifying element; S2074, determine whether the expected musical note duration, the first performance duration, and the second performance duration satisfy the time value evaluation model, wherein the time value evaluation model includes: first performance duration - second performance duration = expected musical note duration; if yes, it is preliminarily considered that the signifying information of the set of signifying elements is complete; if not, the signifying information of the set of signifying elements will be supplemented.

4. The method according to claim 3, characterized in that, The steps for supplementing the semantic information of the set of semantic elements include: (1) obtaining the first confidence level of the first type of time value semantic element and the second confidence level of the second type of time value semantic element; (2) determining whether the first confidence level and the second confidence level are both greater than the set first confidence value; If yes, then execute the following steps: (3) Obtain the time difference, where the time difference = musical duration - (first performance duration - second performance duration); (4) Generate at least one supplementary rest information based on the time difference; if no, then execute the following steps: (5) Output a prompt signal to prompt the user to correct the semantic information of the set of semantic elements.

5. The method according to claim 1, characterized in that, It also includes the following steps: Obtain the first position of the first ideographic element; determine whether the first ideographic element crosses at least one termination line based on the first position; if so, decompose the first ideographic element into at least two ideographic sub-elements based on the corresponding termination line, and record the second position of the ideographic sub-elements. The at least two second ideographic sub-elements are classified into the corresponding sets of at least two ideographic elements according to the second position.

6. The method according to claim 1, characterized in that, It also includes the following steps: Determine whether the first ideographic element is an auxiliary ideographic element, wherein the auxiliary ideographic element is an ideographic element used to supplement auxiliary information for the note; if so, execute the following steps: obtain the first position of the first ideographic element; identify at least one first ideographic element in a third position associated with the first position; The auxiliary information is associated with the corresponding first semantic element.

7. The method according to claim 6, characterized in that, The auxiliary ideographic elements include one or more of the following: slurs, tie lines, rests, staccato marks, sustain marks, pipa marks, and ornaments.

8. The method according to claim 1, characterized in that, S202 includes the following steps: S2021, The first target detection model is used to identify the first level of the first semantic element; The first target detection model includes: an input terminal for receiving a first musical score image; a feature extraction network for extracting feature information from the first musical score image; wherein the first feature information includes: pixel information of a first-level first semantic element; and a feature fusion network for fusing the first feature information and outputting a first feature map. An object detection network is used to identify the first feature map according to the corresponding minimum recognition unit, so as to classify and output the fusion result. The minimum recognition unit is the minimum resolution for the object detection network to identify the first feature map. Correspondingly, the object detection network includes a second feature detection layer, a third feature detection layer and a fourth feature detection layer connected in sequence, and the size of the minimum recognition unit from the second feature detection layer to the fourth feature detection layer gradually increases. S2022, The second target detection model is used to identify the second-level first ideographic element, wherein the second-level first ideographic element is an ideographic element that can be drawn independently in the musical score; The second target detection model includes: an input terminal for receiving the second musical score image; a feature extraction network for extracting second feature information from the second musical score image; wherein the second feature information includes: information of a second-level first semantic element; and a feature fusion network for fusing the second feature information and outputting a second feature map. An object detection network is used to identify the second feature map according to the corresponding minimum recognition unit, so as to classify and output the fusion result. The minimum recognition unit is the minimum resolution for the object detection network to identify the second feature map. Correspondingly, the object detection network includes, in sequence, a first feature detection layer, a second feature detection layer, a third feature detection layer and a fourth feature detection layer connected to each other. The size of the first smallest recognition unit in the first feature detection layer is 4×4 to 8×8 pixels, and the size of the smallest recognition unit from the first feature detection layer to the fourth feature detection layer gradually increases.

9. A musical score reconstruction system, characterized in that, include: The music score acquisition module is used to provide music scores, which include multiple first ideographic elements; The semantic element recognition module is used to recognize the first semantic form of the first semantic element, and the first semantic form is also associated with a confidence level; wherein, the first semantic form can express the meaning of at least one musical sound element in the musical score, and the first semantic form includes one or more of the following: shape, position, quantity, direction, and the musical sound element includes one or more of the following: duration, dynamics, pitch, volume, rhythm, meter, tonality, and mode, and the confidence level is used to represent the recognition accuracy of the first semantic form; The ideographic element evaluation module is used to determine whether the first ideographic element is a first-level ideographic element using a first evaluation rule. The first evaluation rule requires that when a first ideographic element must be connected with another first ideographic element on the musical score to form a complete identifier, the corresponding first ideographic element is considered to be the first-level ideographic element. If so, the first-level ideographic elements used to form a complete identifier are grouped into a subset of ideographic elements. The termination line identification module is used to identify at least two termination lines from a plurality of the first semantic elements; The semantic element classification module is used to classify multiple first semantic elements into multiple semantic element sets according to the termination line, wherein the first semantic elements located between two adjacent termination lines are classified into one semantic element set, and one semantic element set includes at least one semantic element subset.

10. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the music score reconstruction method as described in any one of claims 1 to 8.

11. A method for recognizing musical scores, characterized in that, include: S101, Construct an object detection model, wherein the object detection model includes, in sequence: The input terminal is used to receive a musical score image; the feature extraction network is used to extract feature information from the musical score image; wherein, the feature information includes: (1) basic information, which includes: note information, clef information, key signature information, slur information, and polyphonic information; (2) technique information, which includes: performance technique information and fingering information; (3) emotional style information, which includes: expression marking information, dynamic marking information, and time signature information; the feature fusion network is used to fuse the feature information and output a corresponding feature map; An object detection network is provided, which identifies the feature map based on a corresponding minimum recognition unit to classify and output the fusion result. The minimum recognition unit is the minimum resolution at which the object detection network identifies the feature map. Correspondingly, the object detection network sequentially includes a first feature detection layer, a second feature detection layer, a third feature detection layer, and a fourth feature detection layer connected together. The first feature detection layer performs object detection on the feature map based on a first minimum recognition unit; the second feature detection layer performs object detection on the feature map based on a second minimum recognition unit; the third feature detection layer performs object detection on the feature map based on a third minimum recognition unit; and the fourth feature detection layer performs object detection on the feature map based on a fourth minimum recognition unit. The size of the first minimum recognition unit is 4 pixels × 4 pixels to 8 pixels × 8 pixels, and the size of the first, second, third, and fourth minimum recognition units gradually increases. The output detection result includes a data stream formed by multiple feature sequences, wherein the feature sequence includes the type of feature information, the position of the feature information, and the confidence level of the feature information. S102, the music score image to be identified is input into the target detection model, and the target detection network outputs the corresponding detection result.

12. The method according to claim 11, characterized in that, Before S102, the method further includes the step of: performing detail feature enhancement processing on the score image to be identified; wherein, the detail feature enhancement processing includes the following steps: The sheet music image to be identified is converted to grayscale to obtain a grayscale image; the grayscale image is binarized to obtain a binarized image; a shadow removal operation is performed on the binarized image to obtain a shadow-free image; outliers in the shadow-free image are identified, and a contrast enhancement operation is used to reduce the intensity of the visual features of the outliers, thereby obtaining a contrast-enhanced image; wherein, the outliers include: a type of point, and the interval between the type of point and the position of the target feature in the shadow-free image is greater than a first interval threshold, and the target feature includes one or more of the following: notes, clefs, rests, fermata, staccato marks, dynamic marks, and performance techniques.

13. The method according to claim 12, characterized in that, The abnormal points also include: Class II points. Correspondingly, the step of identifying abnormal points in the deshaded image includes: identifying target points in the deshaded image, wherein the target point is a set of continuous pixels with at least one pixel value less than a set threshold; when the position of the target point and the corresponding target feature in the deshaded image is less than or equal to a first interval threshold, the target point is identified as an associated point of the target feature; when the number of associated points of the target feature is greater than the target number, a Class II point is selected from the associated points.

14. The method according to claim 13, characterized in that, The step of selecting a binary point from the associated points includes: calculating the intensity difference of visual features between at least two associated points associated with the same target feature; wherein the visual features include one or more of the following: grayscale, brightness, size; when the intensity difference is greater than a set intensity threshold, the associated point with the smaller intensity of the visual feature is selected as a binary point; when the intensity difference is less than or equal to the intensity threshold, a user input instruction is received, the input instruction being used to define a binary point from at least two associated points.

15. The method according to claim 14, characterized in that, When the intensity difference is less than or equal to the intensity threshold, the method further includes the following steps: comparing the binarized image and the deshaded image, determining whether there is a newly added target point in the original shadow region of the deshaded image, and marking the target point as a newly added point; determining whether the associated point is the newly added point; if so, outputting a suggestion signal to the user to define the corresponding associated point as the binary point.

16. The method according to claim 15, characterized in that, It also includes the following steps: The contrast-enhanced image is then tilt-corrected.

17. The method according to claim 11, characterized in that, The selectable range of the second minimum recognition unit is 8×8 pixels to 16×16 pixels, and / or the selectable range of the third minimum recognition unit is 16×16 pixels to 32×32 pixels, and / or the selectable range of the fourth minimum recognition unit is 32×32 pixels to 64×64 pixels.

18. A music score recognition system, characterized in that, include: A model providing module is used to provide an object detection model, wherein the object detection model comprises, in sequence: The input terminal is used to receive a musical score image; the feature extraction network is used to extract feature information from the musical score image; wherein, the feature information includes: (1) basic information, the basic information includes: note information, clef information, key signature information, slur information, polyphonic information; (2) technical information, the technical information includes: performance technique information, fingering information; (3) emotional style information, the emotional style information includes: expression marking information, dynamic marking information, time signature information; the feature fusion network is used to fuse the feature information and output a corresponding feature map; the target detection network is used to identify the feature map according to the corresponding minimum recognition unit, so as to classify and output the fusion result, wherein, the minimum recognition unit is the minimum resolution of the target detection network to identify the feature map, and correspondingly, the target detection network The network sequentially includes: a first feature detection layer, a second feature detection layer, a third feature detection layer, and a fourth feature detection layer connected to each other. The first feature detection layer performs target detection on the feature map based on a first minimum recognition unit; the second feature detection layer performs target detection on the feature map based on a second minimum recognition unit; the third feature detection layer performs target detection on the feature map based on a third minimum recognition unit; and the fourth feature detection layer performs target detection on the feature map based on a fourth minimum recognition unit. The size of the first minimum recognition unit is 4 pixels × 4 pixels to 8 pixels × 8 pixels, and the size of the first minimum recognition unit, the second minimum recognition unit, the third minimum recognition unit, and the fourth minimum recognition unit gradually increases. The output detection result includes: a data stream formed by multiple feature sequences, wherein the feature sequence includes: the type of feature information, the position of the feature information, and the confidence level of the feature information. The detection module is used to input the sheet music image to be identified into the target detection model, and the target detection network outputs the detection result accordingly.

19. The system according to claim 18, characterized in that, Also includes: An image processing module is used to perform detail feature enhancement processing on the musical score image to be identified; wherein, the image processing module includes: The system includes a grayscale processing unit for converting the sheet music image to grayscale to obtain a grayscale image; a binarization processing unit for binarizing the grayscale image to obtain a binarized image; a shadow processing unit for performing shadow removal on the binarized image to obtain a de-shadowed image; and an outlier processing unit for identifying outliers in the de-shadowed image and using contrast enhancement to reduce the intensity of the visible features of the outliers, thereby obtaining a contrast-enhanced image. The outliers include: a type of point, where the interval between the type of point and the position of the target feature in the de-shadowed image is greater than a first interval threshold. The target features include one or more of the following: musical notes, clefs, rests, fermata, staccato marks, dynamic markings, and performance techniques.

20. The system according to claim 19, characterized in that, The abnormal points also include: binary points. Correspondingly, the abnormal point processing unit includes: an identification subunit, used to identify target points in the deshaded image, wherein the target point is a set of continuous pixels with at least one pixel value less than a set threshold; an association subunit, used to identify the target point as an associated point of the target feature when the position of the target point and the corresponding target feature in the deshaded image is less than or equal to a first interval threshold; and a binary point selection unit, used to select a binary point from the associated points when the number of associated points of the target feature is greater than the target number.

Images