Information processing method, information processing system, and program

Abstract

[Problem] To provide technology capable of providing a user with a musical score in which an arbitrary arrangement is performed on an arbitrary musical piece. [Solution] According to one aspect of the present invention, provided is an information processing method in which an information processing system executes the following steps. In a musical piece acquisition step, musical piece data is acquired. In a specification acquisition step, arrangement specifications for designating arrangement for the musical piece data are acquired. In a data generation step, musical score data in which the musical piece data is arranged according to the arrangement specifications is generated on the basis of a combination of the musical piece data and the arrangement specifications and first reference information, wherein the first reference information is information pertaining to correlation between the combination of the musical piece data and the arrangement specifications and the musical score data.

Description

Information Processing Method, Information Processing System, and Program ,

[0007] ,

[0006] , ,

[0005] ,

[0001] The present invention relates to an information processing method, an information processing system, and a program.

[0002] Patent Document 1 discloses a technique for changing the difficulty level of sheet music using a machine learning model.

[0003] Japanese Patent Application Laid-Open No. 2023-77140

[0004] Among users who play musical instruments, there are users who want to play any piece of music in an arrangement suitable for themselves while referring to sheet music. However, the above-described technique does not assume the provision of sheet music with an arbitrary arrangement for an arbitrary piece of music.

[0005] In view of the above circumstances, the present invention aims to provide a technique that can provide a user with sheet music having an arbitrary arrangement for an arbitrary piece of music.

[0006] According to one aspect of the present invention, there is provided an information processing method in which an information processing system executes the following steps: In a music acquisition step, music data is acquired; in a specification acquisition step, an arrangement specification for specifying an arrangement for the music data is acquired; in a data generation step, score data obtained by arranging the music data in accordance with the arrangement specification is generated based on a combination of the music data and the arrangement specification and first reference information, and the first reference information is information regarding the correlation between the combination of the music data and the arrangement specification and the score data.

[0007] This is a diagram showing the configuration of information processing system 1. This is a block diagram showing the hardware configuration of information processing device 2. This is a block diagram showing the hardware configuration of user terminal 3. This is a block diagram showing the functions realized by information processing device 2 (processor 23). This is a diagram showing an example of the data management screen MD displayed on user terminal 3. This is a diagram showing the state in which the selection of music data is accepted on the data management screen MD. This is a diagram showing an example of the level selection screen LD displayed on user terminal 3. This is a diagram showing an example of the score display screen SD displayed on user terminal 3. This is a diagram showing an example of the score editing screen ED displayed on user terminal 3. This is a diagram showing an example of the judgment screen JD displayed on user terminal 3. This is a diagram showing an example of the judgment screen JD when the recording of the performance is complete. This is a diagram showing an example of the result display screen OD displayed on user terminal 3. This is a diagram showing an example of the feedback screen FD displayed on user terminal 3. This is a diagram showing an example of score data in which the performance level has been changed as an example of arrangement according to proficiency level. This is an example of a flowchart of the processing performed by information processing device 2. This is an activity diagram showing the flow of the first form of information processing method. This is an activity diagram showing the flow of the second form of information processing method. This is an activity diagram showing the flow of the third form of information processing method. This is an activity diagram showing the flow of information processing when the processor 33 of the user terminal 3 executes each step of the information processing system 1. This is a diagram showing the configuration of the information processing system 100. This is a flowchart showing an example of the process of generating an arranged musical score by the user terminal 110. This is a diagram showing an example of the musical score generation application screen displayed on the user terminal 110. This is a flowchart showing an example of the process of teaching an arranged musical score by the user terminal 110. This is a flowchart showing an example of the process of purchasing an arranged song by the server 120.

[0008] Incidentally, the program for realizing the software appearing in one embodiment may be provided as a computer-readable non-transitor-readable medium, or it may be provided so that it can be downloaded from an external server, or it may be provided so that the program is launched on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0009] Furthermore, in various information processing according to one embodiment, an input and an output corresponding to the input can be realized. Here, as long as an output is obtained as a result of the input, the form of the information referenced in such information processing (hereinafter referred to as "reference information") is not limited. The reference information may be, for example, rule-based information such as a database, a lookup table, or a predetermined function (including a decision formula such as a regression equation constructed by a statistical method), or a trained model that has been pre-trained to learn the correlation between input and output, or a generative AI such as a large-scale language model that can output a desired result by inputting a prompt (these models include parameters that construct the correlation relationship between input and output) or a visual language model.

[0010] Furthermore, in one embodiment, "part" may include, for example, hardware resources implemented by a circuit in a broad sense, and the information processing of software that can be specifically realized by these hardware resources. Also, in one embodiment, various types of information are handled, and this information can be represented, for example, by the physical values ​​of signal values ​​representing voltage and current, the high or low values ​​of signal values ​​as a set of binary bits composed of 0s or 1s, or by quantum superposition (so-called qubits), and communication and calculations can be performed on a circuit in a broad sense.

[0011] Furthermore, a circuit in a broad sense is a circuit realized by combining at least an appropriate combination of circuits, circuits, processors, and memory. The processor may be a general-purpose processor or a dedicated circuit. In other words, this includes application-specific integrated circuits (ASICs), programmable logic devices (for example, simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.

[0012] 1. Hardware Configuration This section describes the hardware configuration.

[0013] <Information Processing System 1> Figure 1 is a configuration diagram representing the information processing system 1. The information processing system 1 comprises at least one information processing device 2 and at least one user terminal 3. The information processing device 2 and the user terminal 3 are configured to communicate with each other via a telecommunications line. In one embodiment, the information processing system 1 consists of one or more devices or components. For example, if the information processing system 1 consists only of the information processing device 2, then the information processing system 1 can be the information processing device 2. These components will be described below.

[0014] <Information Processing Device 2> Figure 2 is a block diagram showing the hardware configuration of the information processing device 2. The information processing device 2 comprises a communication bus 20, a communication unit 21, a storage unit 22, and a processor 23. The communication unit 21, the storage unit 22, and the processor 23 are electrically connected within the information processing device 2, for example, via the communication bus 20. The communication bus 20 connecting the communication unit 21, the storage unit 22, and the processor 23 may be common or independent of each other. Furthermore, the communication unit 21, the storage unit 22, etc., may be located inside the processor 23.

[0015] <Communication Unit 21> The communication unit 21 preferably uses wired communication means such as USB, IEEE 1394, Thunderbolt®, and wired LAN network communication. However, the communication unit 21 may also include wireless LAN network communication, mobile communication such as 3G / LTE / 5G, and Bluetooth® communication as needed. The communication unit 21 may be implemented as a collection of these multiple communication means. That is, the information processing device 2 may communicate various information from the outside via the communication unit 21 and the network.

[0016] <Storage Unit 22> The storage unit 22 stores various types of information as defined above. This can be implemented, for example, as a storage device such as a solid-state drive (SSD) that stores various programs related to the information processing device 2 executed by the processor 23, or as a memory such as a random-access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to program calculations. The storage unit 22 stores various programs and variables related to the information processing device 2 executed by the processor 23.

[0017] <Processor 23> The processor 23 performs processing and control of the overall operation related to the information processing device 2. The processor 23 is, for example, a Central Processing Unit (CPU). The processor 23 realizes various functions related to the information processing device 2 by reading predetermined programs stored in the storage unit 22. That is, information processing by software stored in the storage unit 22 is concretely realized by the processor 23, which is an example of hardware, and can be executed as each functional unit included in the processor 23. In other words, the processor 23 is configured to read programs and execute each functional unit. These will be described in more detail in the next section. Note that the processor 23 is not limited to being a single unit, and the information processing device 2 may have multiple processors 23 for each function.

[0018] The information processing device 2 may be a server having a processor 23. Furthermore, the information processing device 2 may be on-premise or cloud-based. A cloud-based information processing device 2 may provide the above-mentioned functions and processing in the form of, for example, SaaS (Software as a Service) or cloud computing.

[0019] <User Terminal 3> Figure 3 is a block diagram showing the hardware configuration of user terminal 3. User terminal 3 can access the information processing device 2, which acts as a server. User terminal 3 comprises a communication bus 30, a communication unit 31, a storage unit 32, a processor 33, a display unit 34, an input unit 35, and a sound output unit 36. The communication unit 31, storage unit 32, processor 33, display unit 34, input unit 35, and sound output unit 36 ​​are electrically connected within user terminal 3, for example, via the communication bus 30. The communication bus 30 connecting the communication unit 31, storage unit 32, processor 33, display unit 34, input unit 35, and sound output unit 36 ​​may be common or independent of each other. Also, the communication unit 31, storage unit 32, etc. may be located inside the processor 33. The explanation of the communication unit 31, storage unit 32, and processor 33 is the same as the explanation of each part in the information processing device 2, so it is omitted.

[0020] <Display Unit 34> The display unit 34 displays a graphical user interface (GUI) screen that can be operated by the user. The display unit 34 may be included in the housing of the user terminal 3 or it may be an external device. Specifically, the display unit 34 may be implemented as a display device such as a CRT display, liquid crystal display, organic EL display, or plasma display. It is preferable that these display devices be used in accordance with the type of user terminal 3.

[0021] <Input Unit 35> The input unit 35 receives operation inputs made by the user. The operation inputs are transmitted to the processor 33 via the communication bus 30 as command signals. The processor 33 can perform predetermined controls and calculations based on the transmitted command signals as needed. The input unit 35 may be included in the casing of the user terminal 3 or it may be externally attached. For example, the input unit 35 may be implemented as a touch panel integrated with the display unit 34. When the input unit 35 is implemented as a touch panel, the user can input tap operations, swipe operations, etc. to the input unit 35. Instead of a touch panel, the input unit 35 can be a switch button, mouse, trackpad, QWERTY keyboard, etc.

[0022] <Sound output unit 36> The sound output unit 36 ​​outputs sound (voice). The sound output unit 36 ​​may be a speaker included in the housing of the user terminal 3, or it may be an external speaker.

[0023] 2. Functional Configuration This section describes the functional configuration of this embodiment. Information processing by software stored in the memory unit 22 is concretely realized by the processor 23, which is an example of hardware, and can be executed as each functional unit included in the processor 23. The information processing system 1 functions as a system that generates musical scores by arranging arbitrary musical pieces.

[0024] Figure 4 is a block diagram showing the functions realized by the information processing device 2 (processor 23). Specifically, the information processing device 2 (processor 23) includes a music acquisition unit 231, a specification acquisition unit 232, a data generation unit 233, an editing reception unit 234, a performance acquisition unit 235, a performance judgment unit 236, a data adjustment unit 237, and a data management unit 238.

[0025] <Music Acquisition Unit 231> The music acquisition unit 231 is configured to acquire music data. "Music data" is data recorded (recorded or photographed) of the target song (the song to be arranged) that serves as the basis for the song (hereinafter referred to as "arranged song") from which musical score data is generated by the data generation unit 233 described later. Music data includes, for example, waveform data, sequence data, musical score information data, video data, etc.

[0026] "Waveform data" is audio data that records the sounds used in a musical piece (instrumental sounds and / or vocal sounds) as waveforms. Waveform data is played directly on playback devices. Examples of waveform data formats include wav, mp3, aac, wma, and FLAC.

[0027] "Sequence data" is data that includes symbols representing a series of notes (pitch and duration), timbre, pitch changes, volume changes, etc., contained in the sounds played in a musical piece. Sequence data is control data that causes a playback device (sequencer) or software (sequencing software) capable of playing sequence data to play the musical piece. Examples of sequence data include MIDI data (data in formats such as mid, seq, sng, rcp, etc.), MML data, note token sequences, etc.

[0028] "Musical score information data" is data that contains information about the musical score of a piece of music. Musical score information data may be in a standardized format such as Music XML (Extensible Markup Language) format, or it may be in any format that contains information about the musical score (image or text).

[0029] "Video data" refers to data that includes music played along with a video (i.e., includes audio data of the performance sounds used in the music).

[0030] Music data may be data for an entire song, or data for a part of a song (for example, data for sections such as verse 1, verse 2, chorus, bridge, etc., or data for measures). Music data may include the sound of a single instrument playing (singing), or it may include the sound of multiple instruments playing (ensemble).

[0031] Music data is stored in databases such as the storage unit 32 of the user terminal 3, the storage unit 22 of the information processing device 2, or other storage (cloud server) besides the information processing device 2. Furthermore, the music data may be data created by the user (for example, data of music composed by the user, data of performances by the user or others, etc.), or data purchased by the user from the rights holder or their representative. Additionally, the music data may be provided in streaming format.

[0032] For example, the music acquisition unit 231 may receive input of data information such as the file name and storage location of the music data from the user terminal 3, and acquire the music data by downloading it from storage based on that data information. Alternatively, the music acquisition unit 231 may acquire music data by receiving an upload of music data from the user terminal 3.

[0033] The music acquisition unit 231 may generate recording data by recording the playback or performance sound of the target music, and acquire said recording data as music data. With this configuration, it is possible to generate musical score data even for music for which music data cannot be easily acquired.

[0034] The music acquisition unit 231 acquires the playback or performance sound of the target music as a signal, for example, via an audio input device (e.g., a microphone) on the user terminal 3, or an audio input device connected to the user terminal 3, and generates recording data. The recording data may be waveform data or sequence data.

[0035] The music acquisition unit 231 may acquire music data by applying sound quality correction to the recorded data. With this configuration, the accuracy of generating musical score data based on the recorded data of the target music (target music) can be improved.

[0036] "Sound quality correction" includes, for example, the removal of noise contained in the recording data, and the reinforcement (increase in volume) of specific frequency ranges (e.g., low frequencies, high frequencies, etc.) in the recording data. These sound quality corrections are performed in the musical score data generated by the data generation unit 233, which will be described later, with the aim of suppressing the transcription of unnecessary notes and the omission of necessary notes.

[0037] The music acquisition unit 231 may accept input or editing of music information, such as song titles, for recording data or music data obtained from recording data from the user terminal 3. The music acquisition unit 231 registers the music information input or edited from the user terminal 3, linking it to the music data.

[0038] <Specification Acquisition Unit 232> The specification acquisition unit 232 is configured to acquire arrangement specifications that specify arrangements for the music data acquired by the music acquisition unit 231.

[0039] "Arrangement" refers to adjusting the structure or characteristics of a target song that is played back by the music data (i.e., recorded as music data). This adjustment may include, for example, the genre of the target song (e.g., pop, rock, jazz), key, rhythm, tempo, chords, scales, main melody, accompaniment, instruments played, parts played, and performance techniques. The instruments played may also include vocals (singing).

[0040] "Arrangement specifications" are information that describes the purpose, target, content, means, and combinations thereof of arrangements to the music data. Specific examples of arrangement specifications include changing the performance level of the target song (raising or lowering it), changing the genre of the target song, changing the instruments or groups of instruments used in the target song, adjusting the main melody, and extracting the accompaniment.

[0041] The arrangement specifications may be entered or selected from the user terminal 3, or they may be automatically acquired (set) based on the user's or the attributes of the target song. For example, the specification acquisition unit 232 may automatically acquire an arrangement specification such as "change genre to jazz" based on the user's favorite genre (e.g., jazz) included in the user's registration information.

[0042] "Performance level" refers to the difficulty of performing the song in question (the level of performance skill required). The performance level depends on the song's components, such as the tempo, rhythm, chords or chord progressions, scales, performance techniques, note density (number of notes per unit time), and polyphony. Therefore, changing the performance level is achieved by adjusting at least one of these song components through arrangement. For example, to increase the performance level (i.e., to make the performance level of the arranged song higher than that of the original song), adjustments are made such as increasing the tempo, complexity of the rhythm, chord progression or scale, sophistication of the performance techniques, or increase in note density or polyphony. To decrease the performance level (i.e., to make the performance level of the arranged song lower than that of the original song), the opposite adjustments are made.

[0043] In addition, "change of musical instrument or group of musical instruments" includes, for example, changing the musical instrument used in the target piece to another instrument, changing from a plurality of musical instruments (group of musical instruments) used in the target piece to a single instrument, changing from a single musical instrument used in the target piece to a plurality of instruments, etc. "Change to a single instrument" may be extraction of a single instrument (e.g., guitar) from the group of musical instruments (e.g., band arrangement) used in the target piece, or change to a single instrument not used in the target piece (e.g., change from a band arrangement without piano to piano). "Change to a plurality of instruments" may be addition of musical instruments used in the target piece (e.g., change from a piano solo piece to an ensemble piece), addition of a new instrument (e.g., guitar) to a musical instrument (e.g., piano) used in the target piece, or change to a plurality of instruments not used in the target piece.

[0044] In addition, "change of group of musical instruments" may include changing from the group of musical instruments used in the target piece to a different group of musical instruments. Such change may be changing from the group of musical instruments (e.g., band arrangement) used in the target piece to a group of musical instruments with fewer instruments than the number of instruments in the said group of musical instruments (e.g., piano and vocal). Also, such change may be changing from the group of musical instruments (e.g., piano and vocal) used in the target piece to a group of musical instruments with more instruments than the number of instruments in the said group of musical instruments (e.g., band arrangement). Also, such change may be changing from the group of musical instruments used in the target piece to a group of musical instruments with the same number of instruments as the said group of musical instruments (e.g., changing the guitar in a band arrangement to a keyboard).

[0045] "Adjustment of the main melody" may include, for example, expanding or contracting the vocal range that bears the main melody as a musical instrument. Also, the arrangement specifications may include adjustment of the main melody according to the range of the instrument after the change, accompanying the change of the musical instrument.

[0046] "Accompaniment extraction" may include, for example, specifying musical instruments not to be included in the accompaniment or specifying musical instruments to be included in the accompaniment. For example, when the vocal is specified as a musical instrument not to be included in the accompaniment, an arrangement for the backing part excluding the vocal is intended.

[0047] The specification acquisition unit 232 may acquire an arrangement specification including at least the musical instruments for arrangement. The "musical instruments for arrangement" are musical instruments for which performance using score data is performed (that is, musical instruments used in the performance of the arranged music). Note that the musical instruments for arrangement may be the musical instruments used in the target music piece. Also, the musical instruments for arrangement may include vocals (singing).

[0048] The specification acquisition unit 232 may acquire an arrangement specification including at least a group of musical instruments for arrangement. The "group of musical instruments for arrangement" is a plurality of musical instruments for which an ensemble using score data is performed (that is, a group of musical instruments used in the ensemble of the arranged music). Note that the group of musical instruments for arrangement may be the same as the group of musical instruments used in the target music piece, or may be a part of the group of musical instruments extracted, or may be a combination of musical instruments included in the group of musical instruments and new musical instruments not included in the group of musical instruments.

[0049] For example, the specification acquisition unit 232 may receive an input (including selection from options) of the musical instruments for arrangement or the group of musical instruments for arrangement (instrumentation) from the user terminal 3, and acquire the input musical instruments for arrangement or the group of musical instruments for arrangement as an arrangement specification. Also, the specification acquisition unit 232 may acquire (automatically set) musical instruments, instrumentations, etc. pre-registered by the user as the musical instruments for arrangement. Further, the specification acquisition unit 232 may determine the musical instruments used in the target music piece by analyzing the music piece data, and acquire the determined musical instruments as the musical instruments for arrangement.

[0050] Furthermore, the specification acquisition unit 232 may acquire a pre-configured instrument or group of instruments for arrangement (hereinafter referred to as "preset instrument") as the arrangement specification. The preset instrument is configured for each application provided by the information processing system 1, for example. For example, if it is a sheet music generation application for piano, the piano is configured as the preset instrument, and the specification acquisition unit 232 acquires the piano as the arrangement specification.

[0051] The specification acquisition unit 232 may acquire an arrangement specification that includes at least the performance level. For example, the specification acquisition unit 232 may receive input of a desired performance level (including selection from options) from the user terminal 3 and acquire the input performance level as an arrangement specification.

[0052] Furthermore, the specification acquisition unit 232 may acquire the performance level (a performance level corresponding to the user's performance skill) determined by the performance judgment unit 236, which will be described later, as an arrangement specification.

[0053] Furthermore, the specification acquisition unit 232 may acquire an arrangement specification that includes at least the instrument or group of instruments for arrangement and the performance level. For example, the specification acquisition unit 232 may receive input from the user terminal 3 of the instrument or group of instruments for arrangement (instrument arrangement) (including selection from options) and the desired performance level (including selection from options), and acquire the input instrument or group of instruments for arrangement and the performance level as an arrangement specification. Alternatively, the specification acquisition unit 232 may acquire the instrument or group of instruments for arrangement input from the user terminal 3 and the performance level determined by the performance determination unit 236 as an arrangement specification.

[0054] The specification acquisition unit 232 may accept the selection of an arrangement from among multiple arrangement candidates and acquire the arrangement specifications corresponding to the selected arrangement. With this configuration, the user can easily select any arrangement from the presented arrangement candidates.

[0055] The specification acquisition unit 232, for example, displays multiple arrangement candidates on the user terminal 3 and accepts the selection of the arrangement to be adopted from the user terminal 3.

[0056] The arrangement options presented to the user may include categories such as performance level, instruments for arrangement, genre, rhythm, and tempo. The options presented to the user may be for a single category (for example, only options for performance level or type of instrument for arrangement) or for multiple categories (for example, options for both performance level and type of instrument for arrangement).

[0057] The specification acquisition unit 232 may accept a provisional selection of an arrangement from the arrangement candidates from the user terminal 3 and play a demo sound of the song data to which the provisionally selected arrangement has been applied on the user terminal 3. For example, the sequence data generated by the data generation unit 233, which will be described later, can be used as the "demo sound". Alternatively, the demo sound may be a sample sound (a sound that has been pre-arranged from a sample song different from the target song) prepared in advance for each arrangement candidate.

[0058] The specification acquisition unit 232 may acquire multiple types of arrangement specifications. Here, "multiple types of arrangement specifications" refers to arrangement specifications that specify different arrangements (for example, different performance levels, instruments used for arrangement, etc.) for a single song data. In this case, the data generation unit 233, described later, generates multiple musical score data corresponding to the multiple types of arrangement specifications for a single song data.

[0059] Figure 5 shows an example of the data management screen MD displayed on the user terminal 3. The data management screen MD is a screen for the user to manage and create musical score data. The data management screen MD includes a keyword input field KF, a data display area DA, and a create button B10.

[0060] The keyword input field KF accepts keywords for searching for existing sheet music data (arranged music) based on song information such as song title, performer name (artist name), and author (composer, lyricist, arranger, etc.). When a keyword is entered in the keyword input field KF, sheet music data containing that keyword as song information is searched and displayed in the data display area DA. Alternatively, instead of sheet music data, the system may search for song data containing keywords entered by the user as song information.

[0061] The data display area DA displays information about sheet music data created by the user (sheet music data information SI). The sheet music data information SI includes song information (song title, performer name, etc.) and tags indicating the arrangement. The tags represent the content of the arrangement (e.g., performance level, genre, instruments used for arrangement, etc.). Multiple tags may be displayed in a single sheet music data information SI. The sheet music data information SI is also associated with a selection object SO and a deletion object DO. The selection object SO accepts instructions to display the sheet music based on the sheet music data. The deletion object DO accepts instructions to delete the sheet music data.

[0062] The create button B10 receives an instruction to execute the generation of musical score data by the data generation unit 233, which will be described later.

[0063] Figure 6 shows the state in the data management screen MD where song data selection is accepted. In the data management screen MD of Figure 6, the create button B10 in Figure 5 is replaced by the purchase button B11, the file selection button B12, and the cancel button B13. These buttons are displayed when an input operation is performed on the create button B10 in Figure 5.

[0064] The purchase button B11 accepts instructions to purchase music data. When an input operation is performed on the purchase button B11, the user terminal 3 performs actions such as launching the application for the linked music data purchase service or displaying the website of the said service.

[0065] The file selection button B12 accepts the user's selection of music data. When an input operation is performed on the file selection button B12, a screen for selecting the file to be acquired as music data is displayed.

[0066] The Cancel button B13 accepts the instruction to cancel the creation of a new musical score data. When an input operation is performed on the Cancel button B13, the Create button B10 shown in Figure 5 reappears in place of the Purchase button B11, File Selection button B12, and Cancel button B13.

[0067] Figure 7 shows an example of the level selection screen LD displayed on the user terminal 3. The level selection screen LD is a screen for receiving input of the performance level as an arrangement specification. The level selection screen LD is displayed, for example, after the selection (purchase) of music data on the screen displayed by the purchase button B11 or file selection button B12 in Figure 6. The level selection screen LD displays the first level button B21, the second level button B22, the third level button B23, the fourth level button B24, the start generation button B25, and the back button B26.

[0068] The first level button B21, second level button B22, third level button B23, and fourth level button B24 accept selection of the performance level as an arrangement specification. In the example in Figure 7, the first level button B21 corresponds to the lowest level, "Beginner," the second level button B22 corresponds to the second lowest level, "Elementary," the third level button B23 corresponds to the second highest level, "Intermediate," and the fourth level button B24 corresponds to the highest level, "Advanced." Furthermore, the first level button B21, second level button B22, third level button B23, and fourth level button B24 accept selection exclusively. In other words, the user can select only one of the four buttons. Note that Figure 7 illustrates the state in which the second level button B22 is selected.

[0069] The generation start button B25 receives an instruction from the data generation unit 233, which will be described later, to generate musical score data. When an input operation is performed on the generation start button B25, the performance level corresponding to the selected button from the first level button B21, second level button B22, third level button B23, and fourth level button B24 is acquired as the arrangement specification.

[0070] The back button B26 accepts an instruction to cancel the generation of musical score data. When an input operation is performed on the back button B26, the level selection screen LD is closed, and for example, the data management screen MD shown in Figure 5 is displayed on the user terminal 3.

[0071] <Data Generation Unit 233> The data generation unit 233 is configured to generate musical score data by arranging the musical data according to the arrangement specifications, based on a combination of the musical data acquired by the musical data acquisition unit 231 and the arrangement specifications acquired by the specification acquisition unit 232, and the first reference information.

[0072] "Sheet music data" is not particularly limited as long as it is sheet music data that enables the user to perform an arranged song (a song arranged according to the arrangement specifications) (i.e., data that allows the sheet music of the arranged song to be output on the user terminal 3). The sheet music data may be, for example, data in Music XML format, or data in any format that includes sheet music information (image or text).

[0073] The musical score data may include only scores for a single instrument, or it may include scores for multiple instruments in an ensemble. Furthermore, the musical score data may include scores for some instruments or groups of instruments in an ensemble (e.g., accompaniment scores, individual parts, etc.). The data generation unit 233 may also generate multiple musical score data sets corresponding to multiple instruments from a single musical piece data set.

[0074] The content of the musical score data is set according to the instrument being used for arrangement. For example, if the instrument being used for arrangement is a keyboard instrument, the data generation unit 233 generates musical score data corresponding to the number of keys. Also, for example, if the instrument being used for arrangement is an instrument with pedal keyboards such as an organ, the data generation unit 233 generates musical score data that includes the pedal keyboards. Furthermore, the tuning may be specified for the instrument being used for arrangement. For example, if the instrument being used for arrangement is an instrument that allows for tuning specification (e.g., a guitar) and a tuning key is specified, the data generation unit 233 generates musical score data corresponding to the specified tuning, such as "drop D tuning".

[0075] Furthermore, parts may be specified for the instruments used for arrangement. For example, if an instrument used for arrangement has multiple parts (e.g., left hand, right hand, and feet (pedal) for a keyboard instrument, or each drum and each cymbal for a drum set), and those parts are specified, the data generation unit 233 generates musical score data corresponding to the specified part, such as "left hand only". In addition, the data generation unit 233 may generate musical score data with different performance levels for each part of the instrument used for arrangement. In other words, the arrangement specifications may include the performance level for each part of the instrument used for arrangement.

[0076] The first reference information is information relating to the correlation between combinations of song data and arrangement specifications and musical score data. The first reference information is stored, for example, in the storage unit 22 of the information processing device 2. The first reference information may include, for example, tables, functions, simple algorithms, etc., that show the correlation between combinations of song data and arrangement specifications and musical score data. Alternatively, the first reference information may be constructed, for example, by statistically analyzing data recorded by combining multiple combinations of song data and arrangement specifications with multiple musical score data corresponding to each of these combinations.

[0077] The first reference information may include a music score generation model that has been pre-trained to take a combination of song data and arrangement specifications as input and output music score data. In this case, the data generation unit 233 inputs the combination of song data and arrangement specifications into the music score generation model and causes the music score generation model to output music score data. The music score generation model includes at least one of a dedicated pre-trained model for generating music score data and a general-purpose pre-trained model not limited to music score data generation.

[0078] Specific machine learning algorithms for pre-trained models include, for example, nearest neighbors, naive Bayes, decision trees, support vector machines, deep learning using neural networks, and regression models. Furthermore, machine learning encompasses supervised learning, unsupervised learning, and self-supervised learning. The pre-trained model can undergo additional training using new data acquired from the information processing device 2, user terminal 3, etc., through transfer learning or fine-tuning.

[0079] A music notation generation model, which is a dedicated pre-trained model, is trained using, for example, combinations of multiple input data and multiple correct answers as training data. In such a music notation generation model, parameters calculated and tuned through training build a correlation between combinations of musical data and arrangement specifications and music notation data.

[0080] A general-purpose pre-trained music score generation model is, for example, a generative AI. When the music score generation model is such a general-purpose pre-trained model, the data generation unit 233 inputs a prompt to the music score generation model that includes an instruction to generate music score data from a combination of musical data and arrangement specifications as input information, and causes the music score generation model to output the music score data. Here, the parameters that construct the generative AI and the prompt that includes an instruction to output music score data corresponding to the combination of musical data and arrangement specifications establish a correlation between the combination of musical data and arrangement specifications and the music score data. The generative AI includes, for example, large-scale language models (LLMs) and visual language models (VLMs).

[0081] If the arrangement specifications include instruments for arrangement, the data generation unit 233 generates sheet music data arranged for the instruments included in the arrangement specifications, based on the combination of the song data and the arrangement specifications, and the first reference information. With this configuration, sheet music data for a specific instrument can be generated from any song.

[0082] If the arrangement specifications include a group of instruments for arrangement, the data generation unit 233 generates musical score data that arranges the musical score data for each instrument in the group of instruments included in the arrangement specifications, based on the combination of the musical score data and the arrangement specifications, and the first reference information. With this configuration, musical score data for ensemble performance can be generated from any musical score.

[0083] If the arrangement specifications include a performance level, the data generation unit 233 generates sheet music data arranged according to the performance level included in the arrangement specifications, based on the combination of the song data and the arrangement specifications, and the first reference information. With this configuration, sheet music data corresponding to the user's performance level can be generated from any song.

[0084] If the arrangement specifications include instruments for arrangement and performance levels, the data generation unit 233 generates sheet music data arranged for the instruments included in the arrangement specifications and according to the performance level, based on the combination of the song data and the arrangement specifications, and the first reference information. With this configuration, sheet music data for a specific instrument can be generated from any song, according to the user's performance skill.

[0085] If the music data is waveform data, the data generation unit 233 may convert the music data into sequence data and then generate musical score data based on that sequence data.

[0086] The data generation unit 233 may further generate sequence data (hereinafter referred to as "arranged sequence data") of an arranged song to be played based on the musical score data. With this configuration, the user can confirm the performance image of the arranged song by playing back the arranged sequence data corresponding to the musical score data.

[0087] The arrangement sequence data is created by arranging the music data acquired by the music acquisition unit 231 according to the arrangement specifications. Playing back the arrangement sequence data outputs the sound of the arranged music performed according to the musical score data. In other words, the arrangement sequence data is data that makes it possible to play back the arranged music while reproducing musical parameters such as key, rhythm, chord progression, dynamics, articulation, and playing style contained in the musical score data.

[0088] The data generation unit 233 may generate arrangement sequence data independently of the musical score data based on a combination of musical score data and arrangement specifications, or it may generate arrangement sequence data and then generate musical score data based on said arrangement sequence data, or it may generate musical score data and then generate arrangement sequence data based on said musical score data.

[0089] When the data generation unit 233 generates musical score data from arrangement sequence data, the first reference information includes the second reference information and the third reference information. The data generation unit 233 also generates arrangement sequence data based on the combination of the music data and arrangement specifications and the second reference information, and further generates musical score data based on the arrangement sequence data and the third reference information. With this configuration, the accuracy of the arrangement of the music data can be improved by generating musical score data via the arrangement sequence data.

[0090] The second reference information is information relating to the correlation between combinations of song data and arrangement specifications and arrangement sequence data. The second reference information is stored, for example, in the storage unit 22 of the information processing device 2. The second reference information may include, for example, tables, functions, simple algorithms, etc., that show the correlation between combinations of song data and arrangement specifications and arrangement sequence data. Alternatively, the second reference information may be constructed, for example, by statistically analyzing data recorded by combining multiple combinations of song data and arrangement specifications with multiple arrangement sequence data corresponding to each of these combinations.

[0091] The second reference information may include a first sequence generation model that has been pre-trained to take a combination of song data and arrangement specifications as input and output arrangement sequence data. In this case, the data generation unit 233 inputs the combination of song data and arrangement specifications to the first sequence generation model and causes the first sequence generation model to output the arrangement sequence data. The first sequence generation model includes at least one of a dedicated pre-trained model for sequence data generation and a general-purpose pre-trained model not limited to sequence data generation.

[0092] The first sequence generation model, which is a dedicated pre-trained model, is trained using, for example, combinations of multiple input data for training and multiple correct answers for training as training data. In such a first sequence generation model, parameters calculated and tuned through training build a correlation between the combination of music data and arrangement specifications and the arrangement sequence data.

[0093] The first sequence generation model, which is a general-purpose pre-trained model, is, for example, a generative AI. When the first sequence generation model is such a general-purpose pre-trained model, the data generation unit 233 inputs a prompt to the first sequence generation model that includes an instruction to generate arrangement sequence data from a combination of song data and arrangement specifications as input information, and causes the first sequence generation model to output the arrangement sequence data. Here, the parameters that construct the generative AI and the prompt that includes an instruction to output arrangement sequence data corresponding to the combination of song data and arrangement specifications establish a correlation between the combination of song data and arrangement specifications and the arrangement sequence data.

[0094] The third reference information is information relating to the correlation between arrangement sequence data and musical score data. The third reference information is stored, for example, in the storage unit 22 of the information processing device 2. The third reference information may include, for example, tables, functions, simple algorithms (conversion rules), etc., that show the correlation between arrangement sequence data and musical score data. Alternatively, the third reference information may be constructed, for example, by statistically analyzing data recorded by combining multiple arrangement sequence data and multiple musical score data corresponding to each of them.

[0095] The third reference information may include a data conversion model that has been pre-trained using machine learning, capable of taking arrangement sequence data as input and outputting musical score data. In this case, the data generation unit 233 inputs the arrangement sequence data into the data conversion model and outputs the musical score data to the data conversion model. The data conversion model includes at least one of a dedicated pre-trained model for data conversion and a general-purpose pre-trained model not limited to data conversion purposes.

[0096] A data transformation model, which is a dedicated pre-trained model, is trained using, for example, combinations of multiple input data sets and multiple correct answers as training data. In such a data transformation model, parameters calculated and tuned through training build a correlation between the arrangement sequence data and the musical score data.

[0097] The data transformation model, which is a general-purpose pre-trained model, is, for example, a generative AI. When the data transformation model is such a general-purpose pre-trained model, the data generation unit 233 inputs a prompt to the data transformation model that includes an instruction to generate musical score data from the arrangement sequence data as input information, and causes the data transformation model to output the musical score data. Here, the parameters that construct the generative AI and the prompt that includes an instruction to output musical score data corresponding to the arrangement sequence data establish a correlation between the arrangement sequence data and the musical score data.

[0098] When the data generation unit 233 generates arrangement sequence data from musical score data, the data generation unit 233 generates the arrangement sequence data based, for example, on the musical score data and the seventh reference information. The seventh reference information is information relating to the correlation between the musical score data and the arrangement sequence data. The seventh reference information is stored, for example, in the storage unit 22 of the information processing device 2. The seventh reference information may include, for example, a table, a function, a simple algorithm (conversion rule), etc., that shows the correlation between the musical score data and the arrangement sequence data. Alternatively, the seventh reference information may be constructed, for example, by statistically analyzing data recorded by combining multiple musical score data and multiple arrangement sequence data corresponding to each of them.

[0099] The seventh reference information may include a second sequence generation model that has undergone machine learning in advance, capable of taking musical score data as input and outputting arrangement sequence data. In this case, the data generation unit 233 inputs the musical score data into the second sequence generation model and causes the second sequence generation model to output the arrangement sequence data. The second sequence generation model includes at least one of a dedicated pre-trained model for sequence data generation and a general-purpose pre-trained model not limited to sequence data generation.

[0100] The second sequence generation model, which is a dedicated pre-trained model, is trained using, for example, combinations of multiple input data for training and multiple correct answers for training as training data. In such a second sequence generation model, parameters calculated and tuned through training build a correlation between the musical score data and the arrangement sequence data.

[0101] The second sequence generation model, which is a general-purpose pre-trained model, is, for example, a generative AI. When the second sequence generation model is such a general-purpose pre-trained model, the data generation unit 233 inputs a prompt to the second sequence generation model that includes an instruction to generate arrangement sequence data from the musical score data as input information, and causes the second sequence generation model to output the arrangement sequence data. Here, the parameters that construct the generative AI and the prompt that includes an instruction to output arrangement sequence data corresponding to the musical score data establish a correlation between the musical score data and the arrangement sequence data.

[0102] The data generation unit 233 generates analysis data indicating the musical parameters of the music data based on the music data acquired by the music acquisition unit 231 and the fourth reference information, and may further generate musical score data including the analysis data. With this configuration, by referring to the musical score data including the analysis data, the user can reproduce a performance that is close to the original music.

[0103] "Musical parameters" are parameters in a musical score other than the symbols that represent sound (notes, rests, etc.). Musical parameters include, for example, the key, time signature, chords, dynamics, articulation, performance style, and structural information (bar lines, repeat signs, section signs, etc.).

[0104] The analysis data should preferably include at least one of the musical data: key, time signature, and bar lines. This configuration allows for the presentation of useful data to the user that contributes to musical performance.

[0105] The fourth reference information is information relating to the correlation between music data and analysis data. The fourth reference information is stored, for example, in the storage unit 22 of the information processing device 2. The fourth reference information may include, for example, tables, functions, simple algorithms, etc., that show the correlation between music data and analysis data. Alternatively, the fourth reference information may be constructed, for example, by statistically analyzing data recorded by combining multiple music data and multiple analysis data corresponding to each of them.

[0106] The fourth reference information may include an analysis model that has been pre-trained using machine learning, capable of taking music data as input and outputting analysis data. In this case, the data generation unit 233 inputs the music data into the analysis model and causes the analysis data to be output by the analysis model. The analysis model includes at least one of a dedicated pre-trained model for music analysis and a general-purpose pre-trained model not limited to music analysis.

[0107] The analysis model, which is a dedicated pre-trained model, is learned using, for example, combinations of multiple input data and multiple correct answers as training data. In such an analysis model, parameters calculated and tuned through training build a correlation between the music data and the analysis data.

[0108] The analysis model, which is a general-purpose pre-trained model, is, for example, a generative AI. When the analysis model is such a general-purpose pre-trained model, the data generation unit 233 inputs a prompt to the analysis model that includes an instruction to generate analysis data from the music data as input information, and causes the analysis model to output the analysis data. Here, the parameters that construct the generative AI and the prompt that includes an instruction to output analysis data corresponding to the music data establish a correlation between the music data and the analysis data.

[0109] Musical score data that includes analysis data is generated, for example, by incorporating the information of analysis data into musical score data that does not include analysis data (for example, by inserting key signature, time signature, bar lines, etc., into designated positions in the musical score data).

[0110] The data generation unit 233 saves the generated musical score data and arrangement sequence data to storage, outputs (exports) it to the user terminal 3, etc., based on the user's instructions.

[0111] If the specification acquisition unit 232 acquires multiple types of arrangement specifications for a single song data, the data generation unit 233 generates sheet music data and arrangement sequence data for each arrangement specification (for example, for each performance level, genre, instrument used for arrangement, etc.) based on the single song data. In this case, the data generation unit 233 may output (display or play) the generated sheet music data or arrangement sequence data to the user terminal 3 and accept the selection of sheet music data or arrangement sequence data to be saved from the user terminal 3.

[0112] Figure 8 shows an example of the sheet music display screen SD displayed on the user terminal 3. The sheet music display screen SD is a screen for displaying the generated sheet music data. The sheet music display screen SD includes a sheet music display area SA, a home button B31, a save button B32, an edit object EO, a playback object PO, and a regeneration object RO.

[0113] The sheet music display area SA displays the sheet music data of the generated score. The home button B31 accepts the instruction to switch to the home screen (initial screen). The save button B32 accepts the instruction to save the sheet music data displayed in the sheet music display area SA in a predetermined format (PDF (Portable Document Format) in Figure 8).

[0114] The editing object EO receives instructions to edit the musical score data displayed in the musical score display area SA, which are executed by the editing reception unit 234 described later. The playback object PO receives instructions to play back the performance sound based on the musical score data displayed in the musical score display area SA. When an input operation is performed on the playback object PO, the arrangement sequence data corresponding to the musical score data displayed in the musical score display area SA is played back. The regeneration object RO receives instructions to regenerate the musical score data. When an input operation is performed on the regeneration object RO, the data generation unit 233 regenerates the musical score data and arrangement sequence data using the same music data and arrangement specifications used for the musical score data displayed in the musical score display area SA.

[0115] <Editing Reception Unit 234> The editing reception unit 234 is configured to accept edits from the user to the musical score data generated by the data generation unit 233. With this configuration, the user can make further arrangements, modifications, etc. to the generated musical score data.

[0116] The editing reception unit 234, for example, displays the musical score included in the score data on the user terminal 3 and accepts editing input for the musical score from the user terminal 3. "Editing" includes, for example, adding, deleting, or changing notes, rests, musical parameters (key, time signature, bar lines, etc.).

[0117] Furthermore, the editing reception unit 234 may accept the selection of an editing target (for example, a musical note or a rest) from the user terminal 3, display the editable items for that editing target on the user terminal 3, and also accept input of editing content for the editable items from the user terminal 3.

[0118] The editing reception unit 234 may modify the arrangement sequence data in accordance with the user's edits to the musical score data. With this configuration, the musical score data edited by the user and the arrangement sequence data corresponding to that musical score data can be synchronized.

[0119] The editing reception unit 234 generates modified arrangement sequence data based, for example, the edited musical score data and the seventh reference information described above. For example, the editing reception unit 234 inputs the edited musical score data into the second sequence generation model and outputs the modified arrangement sequence data to the second sequence generation model.

[0120] Figure 9 shows an example of the score editing screen ED displayed on the user terminal 3. The score editing screen ED is the screen for editing score data. The score editing screen ED includes the score editing area EA and the save button B41.

[0121] The score editing area EA displays the score data to be edited. When a note or rest to be edited is selected in the score editing area EA, the editing window EW appears on the score editing area EA. In Figure 9, the selected note is shown surrounded by a dashed line. The editing window EW displays several parameters for the selected note (pitch, length, octave, type, display status, etc.) and accepts input for changing these parameters. The editing window EW may display parameters for the entire chord, or parameters for each individual note included in the chord. The save button B41 accepts the instruction to save the score data displayed in the score editing area EA.

[0122] <Performance Acquisition Unit 235> The performance acquisition unit 235 is configured to acquire performance data that records the user's performance. The performance data is used by the performance judgment unit 236, which will be described later, to determine the performance level, proficiency, etc. The performance data is typically waveform data or sequence data, but it does not necessarily have to be in a data format that can be played back by the playback device, as long as it is in a data format that can be judged by the performance judgment unit 236.

[0123] The user's performance recorded in the performance data may be any song, phrase, pattern, etc., or it may be a performance of a designated assigned song. The assigned song may be an arranged song (a song performed using sheet music data generated by the data generation unit 233). Furthermore, if the performance judgment unit 236 determines the user's proficiency with the sheet music data, the performance acquisition unit 235 acquires performance data that records the user's performance using the sheet music data generated by the data generation unit 233. "User performance using sheet music data" refers to the user's performance of an arranged song that follows the sheet music included in the sheet music data.

[0124] Figure 10 shows an example of the judgment screen JD displayed on the user terminal 3. The judgment screen JD is used to record the user's performance and to determine the user's level of proficiency (labeled "Level Judgment" in Figure 10). The judgment screen JD includes a task display area AA and a judgment execution button B51.

[0125] The task display area AA displays the sheet music for the assigned piece. In Figure 10, the sheet music displayed is the sheet music data generated by the data generation unit 233. The judgment execution button B51 accepts the user's instruction to start recording their performance. When an input operation is performed on the judgment execution button B51, the user terminal 3 starts recording their performance.

[0126] <Performance Judgment Unit 236> The performance judgment unit 236 is configured to determine the performance level as an arrangement specification, the user's proficiency with that performance level, etc., based on the user's performance.

[0127] Specifically, the performance determination unit 236 determines the performance level as an arrangement specification acquired by the specification acquisition unit 232 based on the performance data acquired by the performance acquisition unit 235 and the fifth reference information. With this configuration, it is possible to generate sheet music data arranged to an appropriate performance level based on the user's actual performance.

[0128] The performance level determined by the performance determination unit 236 may be a parameter that directly represents the user's level of performance skill, or it may be a categorization of the user's level of performance skill into one of the pre-defined performance levels. The performance determination unit 236 determines the performance level based, for example, on the accuracy of the pitch, rhythm, and expression of the performance in the performance data, the range of the performance data, and the playing techniques included in the performance data.

[0129] The fifth reference information is information relating to the correlation between performance data and performance levels. The fifth reference information is stored, for example, in the storage unit 22 of the information processing device 2. The fifth reference information may include, for example, tables, functions, simple algorithms, etc., that show the correlation between performance data and performance levels. Alternatively, the fifth reference information may be constructed, for example, by statistically analyzing data recorded by combining multiple performance data and multiple performance levels corresponding to each of them.

[0130] The fifth reference information may include a level determination model that has been pre-trained using machine learning, which can take performance data as input and output a performance level. In this case, the performance determination unit 236 inputs the performance data to the level determination model and has the level determination model output the performance level. The level determination model includes at least one of a dedicated pre-trained model for level determination and a general-purpose pre-trained model not limited to level determination purposes.

[0131] A level determination model, which is a dedicated pre-trained model, is learned using, for example, combinations of multiple input data and multiple correct answers as training data. In such a level determination model, parameters calculated and tuned through training build a correlation between performance data and performance level.

[0132] The level determination model, which is a general-purpose pre-trained model, is, for example, a generative AI. When the level determination model is such a general-purpose pre-trained model, the performance determination unit 236 inputs a prompt to the level determination model that includes an instruction to determine the performance level from the performance data as input information, and causes the level determination model to output the performance level. Here, the parameters that construct the generative AI and the prompt that includes an instruction to output the performance level corresponding to the performance data establish a correlation between the performance data and the performance level.

[0133] If the performance data includes a user's performance of a designated piece, the performance determination unit 236 determines the performance level based, for example, on a comparison (reproducibility) between the user's performance and an example performance of the designated piece. In this case, the performance determination unit 236 may determine the user's performance level using designated piece data (data including the sheet music of the designated piece, data recording a reference performance of the designated piece, etc.). The fifth reference information at this time is information regarding the correlation between the combination of performance data and designated piece data and the performance level. For example, the performance determination unit 236 inputs the combination of performance data and designated piece data into a level determination model and outputs the performance level to the level determination model.

[0134] The performance determination unit 236 may determine the user's proficiency level based on the combination of the musical score data generated by the data generation unit 233 and the performance data acquired by the performance acquisition unit 235, and the sixth reference information. "Proficiency level" refers to the degree to which the user has achieved performance of musical score data at a specific performance level (this is the level of reproduction of the arranged song represented by the musical score included in the musical score data, and the extent to which the user is able to perform according to the score). Proficiency level may be expressed as a numerical value (score) or as a category (advanced, intermediate, beginner, etc.).

[0135] The sixth reference information is information relating to the correlation between combinations of musical score data and performance data and proficiency levels. The sixth reference information is stored, for example, in the storage unit 22 of the information processing device 2. The sixth reference information may include, for example, tables, functions, simple algorithms, etc., that show the correlation between combinations of musical score data and performance data and proficiency levels. Alternatively, the sixth reference information may be constructed, for example, by statistically analyzing data recorded by combining multiple combinations of musical score data and performance data with multiple proficiency levels corresponding to each of these combinations.

[0136] The sixth reference information may include a proficiency assessment model that has been pre-trained using machine learning, which can take a combination of musical score data and performance data as input and output a proficiency level. In this case, the performance assessment unit 236 inputs the combination of musical score data and performance data into the proficiency assessment model and causes the proficiency assessment model to output a proficiency level. The proficiency assessment model includes at least one of a dedicated pre-trained model for proficiency assessment and a general-purpose pre-trained model not limited to proficiency assessment purposes.

[0137] A proficiency assessment model, which is a dedicated pre-trained model, is learned using, for example, combinations of multiple input data and multiple correct answers as training data. In such a proficiency assessment model, parameters calculated and tuned through learning build a correlation between the combination of musical score data and performance data and the level of proficiency.

[0138] The proficiency assessment model, which is a general-purpose pre-trained model, is, for example, a generative AI. When the proficiency assessment model is such a general-purpose pre-trained model, the performance assessment unit 236 inputs a prompt to the proficiency assessment model that includes an instruction to determine the level of proficiency from the combination of musical score data and performance data as input information, and causes the proficiency assessment model to output the level of proficiency. Here, the parameters that construct the generative AI and the prompt that includes an instruction to output the level of proficiency corresponding to the combination of musical score data and performance data establish a correlation between the combination of musical score data and performance data and the level of proficiency.

[0139] The performance evaluation unit 236 may display the determined performance level or proficiency level on the user terminal 3. The performance evaluation unit 236 may also display the reason for the evaluation (for example, the location of the mistake, the nature of the mistake, etc.) along with the evaluation result.

[0140] Figure 11 shows an example of the judgment screen JD after the performance recording is complete. When the user's performance recording (generation of performance data) is completed by inputting an operation to the judgment execution button B51 on the judgment screen JD in Figure 10, the viewing button B61 and the scoring button B62 shown in Figure 11 are displayed in place of the judgment execution button B51.

[0141] The playback button B61 receives a command to play the recorded performance data. When an input operation is performed on the playback button B61, the performance sound based on the performance data is played from the user terminal 3. The scoring button B62 receives a command to judge the level of proficiency of the recorded performance data. When an input operation is performed on the scoring button B62, the performance judgment unit 236 performs a judgment on the performance data.

[0142] Figure 12 shows an example of the results display screen OD displayed on the user terminal 3. The results display screen OD is a screen for displaying the evaluation results of the performance data. In the example in Figure 12, the results of the proficiency evaluation for performance data performed using sheet music data with a performance level of "Level 1" are displayed. The results display screen OD includes an evaluation score EP, an evaluation message EM, a first viewing button B71, a second viewing button B72, and a song selection button B73.

[0143] The evaluation score EP is a score that indicates the degree of proficiency. The evaluation message EM describes the evaluation content according to the current proficiency and performance level.

[0144] The first listening button B71 and the second listening button B72 each receive instructions to play an arranged song (specifically, an arranged sequence data corresponding to the musical score data adjusted by the data adjustment unit 237 described later) that has been rearranged at a performance level different from the current performance level. In the example in Figure 12, when an input operation is performed on the first listening button B71, the rearranged song rearranged at "level 2" is played, and when an input operation is performed on the second listening button B72, the rearranged song rearranged at "level 3" is played.

[0145] The song selection button B73 is a button that accepts instructions to change the assigned song. When an input operation is performed on the song selection button B73, a screen for selecting the assigned song (sheet music data) is displayed on the user terminal 3.

[0146] Figure 13 shows an example of the feedback screen FD displayed on the user terminal 3. The feedback screen FD is a screen for displaying feedback (reason for judgment) on performance data. The feedback screen FD includes the task display area AA, the evaluation score EP, and the evaluation comment EC.

[0147] The sheet music for the assigned piece is displayed in the task display area AA. In the task display area AA, the location of the mistake and a comment PC explaining the nature of the mistake are displayed for each section where a mistake occurred. In Figure 13, the location of the mistake is enclosed in a dashed line. The evaluation score EP is a score indicating the degree of proficiency, similar to Figure 10. The evaluation comment EC contains comments about the performance (number of mistakes, areas for improvement, etc.).

[0148] <Data Adjustment Unit 237> The data adjustment unit 237 is configured to adjust the musical score data generated by the data generation unit 233 according to the proficiency level determined by the performance judgment unit 236. With this configuration, the performance level of the musical score data can be adjusted according to the user's proficiency level, thereby improving the user's performance skills using the same song.

[0149] For example, the data adjustment unit 237 generates adjusted data (new musical score data) by arranging the musical score data based on the combination of musical score data and proficiency level, and the eighth reference information. The content of "arrangement of musical score data" includes adjustments to genre, key, rhythm, tempo, chords, scales, main melody, accompaniment, instruments played, parts played, and playing techniques, as described above.

[0150] The eighth reference information is information relating to the correlation between combinations of musical score data and proficiency levels and adjustment data. The eighth reference information is stored, for example, in the storage unit 22 of the information processing device 2. The eighth reference information may include, for example, tables, functions, simple algorithms, etc., that show the correlation between combinations of musical score data and proficiency levels and adjustment data. Alternatively, the eighth reference information may be constructed, for example, by statistically analyzing data recorded by combining multiple combinations of musical score data and proficiency levels with multiple adjustment data corresponding to each of these combinations.

[0151] The eighth reference information may include a pre-trained adjustment model that can take a combination of musical score data and proficiency level as input and output adjustment data. In this case, the data adjustment unit 237 inputs the combination of musical score data and proficiency level into the adjustment model and causes the adjustment model to output adjustment data. The adjustment model includes at least one of a dedicated pre-trained model for data adjustment and a general-purpose pre-trained model not limited to data adjustment purposes.

[0152] A specially trained adjustment model is learned, for example, by using combinations of multiple input data and multiple correct answers as training data. In such an adjustment model, parameters calculated and tuned through training build a correlation between the combination of musical score data and proficiency level and the adjustment data.

[0153] The adjustment model, which is a general-purpose pre-trained model, is, for example, a generative AI. When the adjustment model is such a general-purpose pre-trained model, the data adjustment unit 237 inputs a prompt to the adjustment model that includes an instruction to generate adjustment data from a combination of musical score data and proficiency level as input information, and causes the adjustment model to output the adjustment data. Here, the parameters that construct the generative AI and the prompt that includes an instruction to output adjustment data corresponding to the combination of musical score data and proficiency level establish a correlation between the combination of musical score data and proficiency level and the adjustment data.

[0154] Furthermore, the data adjustment unit 237 may acquire (set) arrangement specifications corresponding to the level of proficiency (for example, performance level), and generate adjustment data as musical score data based on the combination of the music data acquired by the music acquisition unit 231 and the newly acquired arrangement specifications, and the first reference information.

[0155] The data adjustment unit 237 may further generate arrangement sequence data for the arranged song to be played based on the adjustment data. The data adjustment unit 237 may generate arrangement sequence data independently of the adjustment data, similar to the arrangement sequence data generation procedure by the data generation unit 233, or it may generate arrangement sequence data and then generate adjustment data based on that arrangement sequence data, or it may generate adjustment data and then generate arrangement sequence data based on that adjustment data.

[0156] The data adjustment unit 237 may present the user with multiple sheet music data sets that have been arranged differently from one another, depending on the user's skill level. With this configuration, the user can be presented with multiple sheet music data sets with different arrangements according to their skill level, thereby increasing the user's motivation to improve their instrument playing.

[0157] "Presenting musical score data" here includes displaying the musical score data, displaying information about the musical score data (such as performance level) as shown in the result display screen OD in Figure 12, and playing back the arrangement sequence data corresponding to the musical score data.

[0158] Figure 14 shows an example of sheet music data with altered performance levels, illustrating an example of arrangements tailored to different skill levels. Figure 14 illustrates piano arrangements for the same song from Level 1 (lowest level) to Level 3 (highest level). In the examples in Figure 14, as the level increases, the note density (number of note divisions), the number of simultaneous voices (chords), and the playing range increase.

[0159] <Data Management Unit 238> The data management unit 238 is configured to manage musical score data, arrangement sequence data, etc., generated by the data generation unit 233, data adjustment unit 237, etc.

[0160] The data management unit 238 may accept user consent to provide the generated musical score data and / or arrangement sequence data (hereinafter referred to as "managed data") as reference data for improving the functions of the data generation unit 233, data adjustment unit 237, etc. For example, the data management unit 238 may add the managed data approved by the user to the training data of a musical score generation model, etc., and retrain the musical score generation model, etc. The managed data also includes data generated by the data generation unit 233, data adjustment unit 237, etc., that has been edited by the user.

[0161] Furthermore, the data management unit 238 may provide certain rewards to users who have agreed to provide the managed data. Rewards may include, for example, monetary compensation, resources consumed by the services provided by the information processing system 1 (such as tickets required to use specific functions), or coupons that can be used for paid use of external services or to purchase goods.

[0162] The data management unit 238 may provide managed data to other users. For example, as a platform service, the data management unit 238 may register managed data created by the first user in a database and accept purchases of said managed data from the second user. The data management unit 238 may also permit the first user to provide managed data to the second user based on the first user's attributes. "Attributes" here include, for example, the details of the plan that the first user has contracted for in the services provided by the information processing system 1. For example, if the first user has contracted for a specific paid plan, the data management unit 238 may permit the provision of managed data to the second user.

[0163] Furthermore, the data management unit 238 may permit the above-mentioned management (provision as training data, provision to other users, etc.) depending on the status of copyright processing. "Status of copyright processing" refers, for example, to whether or not permission has been obtained from the copyright holder for the target musical works in which the user does not hold the copyright, and the scope of that permission.

[0164] 3. Information Processing Method This section describes the information processing method of the information processing device 2. In this information processing method, the functions of each part of the information processing device 2 are executed by the information processing device 2 as steps.

[0165] The first form of the information processing method comprises a music acquisition step, a specification acquisition step, and a data generation step. In the music acquisition step, music data is acquired. In the specification acquisition step, arrangement specifications that specify arrangements for the music data are acquired. In the data generation step, based on the combination of music data and arrangement specifications and first reference information, musical score data is generated by arranging the music data according to the arrangement specifications. The first reference information is information regarding the correlation between the combination of music data and arrangement specifications and the musical score data.

[0166] Figure 15 is an example of a flowchart of the process performed by the information processing device 2. In this process, the information processing device 2 acquires the music data and arrangement specifications specified by the user (step S110). Note that the music data and arrangement specifications do not necessarily have to be acquired at the same time. After acquiring the music data and arrangement specifications, the information processing device 2 generates musical score data based on them (step S120).

[0167] Figure 16 is an activity diagram showing the flow of the first type of information processing method. Below, the information processing method will be explained in accordance with each activity in this activity diagram.

[0168] First, the user inputs information specifying the song data and arrangement specifications at the user terminal 3 (Activity A110). The information processing device 2 obtains the song data and arrangement specifications specified by the user from the user terminal 3 (Activity A120). Note that the song data and arrangement specifications do not necessarily have to be obtained at the same time.

[0169] After acquiring the song data and arrangement specifications, the information processing device 2 generates musical score data based on these (Activity A130). Subsequently, the information processing device 2 outputs the generated musical score data to the user terminal 3 (Activity A140). This makes the musical score data viewable on the user terminal 3 (Activity A150).

[0170] The second form of the information processing method comprises a music acquisition step, a specification acquisition step, and a data generation step. In the music acquisition step, recording data is generated by recording the playback or performance sound of the target music, and the recording data is acquired as music data. In the specification acquisition step, an arrangement specification that specifies the arrangement for the music data is acquired. In the data generation step, based on the combination of the music data and the arrangement specification and the first reference information, musical score data is generated by arranging the music data according to the arrangement specification, sequence data of the music to be performed based on the musical score data is further generated, analysis data indicating the musical parameters of the music data is generated based on the music data and the fourth reference information, and musical score data including the analysis data is further generated. The first reference information is information regarding the correlation between the combination of music data and the arrangement specification and the musical score data. The fourth reference information is information regarding the correlation between music data and analysis data.

[0171] Figure 17 is an activity diagram showing the flow of the second type of information processing method. Below, the information processing method will be explained in accordance with each activity in this activity diagram.

[0172] First, the user records (records) the target song on the user terminal 3 and inputs information specifying the arrangement specifications (Activity A210). The information processing device 2 obtains the song data containing the recorded target song from the user terminal 3 and also obtains the arrangement specifications specified by the user (Activity A220). Note that the song data and arrangement specifications do not necessarily have to be obtained at the same time.

[0173] After acquiring the music data and arrangement specifications, the information processing device 2 generates musical score data and arrangement sequence data based on these (Activity A230). Next, the information processing device 2 generates analysis data from the music data (Activity A240). Subsequently, the information processing device 2 outputs the musical score data, including the analysis data, and the arrangement sequence data to the user terminal 3 (Activity A250). As a result, the musical score data becomes displayable on the user terminal 3, and the arrangement sequence data becomes playable (Activity A260).

[0174] The third form of the information processing method comprises a music acquisition step, a specification acquisition step, a data generation step, a performance acquisition step, a performance judgment step, and a data adjustment step. In the music acquisition step, music data is acquired. In the specification acquisition step, arrangement specifications, including at least the performance level, are acquired. In the data generation step, based on the combination of music data and arrangement specifications, and first reference information, sheet music data is generated by arranging the music data according to the performance level included in the arrangement specifications. The first reference information is information regarding the correlation between the combination of music data and arrangement specifications and the sheet music data. In the performance acquisition step, performance data is acquired, which is a record of the user's performance using the sheet music data. In the performance judgment step, the user's proficiency is determined based on the combination of sheet music data and performance data, and sixth reference information. The sixth reference information is information regarding the correlation between the combination of sheet music data and performance data and the proficiency level. In the data adjustment step, the sheet music data is adjusted according to the proficiency level.

[0175] Figure 18 is an activity diagram showing the flow of the third type of information processing method. Below, the information processing method will be explained in accordance with each activity in this activity diagram.

[0176] First, the user inputs information specifying the song data and arrangement specifications at the user terminal 3 (Activity A310). The information processing device 2 obtains the song data and arrangement specifications specified by the user from the user terminal 3 (Activity A320). Note that the song data and arrangement specifications do not necessarily have to be obtained at the same time.

[0177] After acquiring the song data and arrangement specifications, the information processing device 2 generates musical score data based on these (Activity A330). Subsequently, the information processing device 2 outputs the generated musical score data to the user terminal 3 (Activity A340). This makes the musical score data viewable on the user terminal 3 (Activity A350).

[0178] The user performs using the displayed musical score data and records the performance on the user terminal 3 (Activity A360). The information processing device 2 acquires the performance data recorded by the user from the user terminal 3 (Activity A370). Subsequently, the information processing device 2 determines the user's proficiency level based on the musical score data and the performance data (Activity A380).

[0179] After determining the proficiency level, the information processing device 2 adjusts the musical score data based on the proficiency level (Activity A390). Subsequently, the information processing device 2 outputs the adjusted musical score data (adjusted data) to the user terminal 3 (Activity A400). This makes the adjusted musical score data viewable on the user terminal 3 (Activity A410).

[0180] This information processing method is implemented by having the information processing system 1 execute each step using a program. In other words, the program is a program that causes the computer to execute the above-mentioned music acquisition step, specification acquisition step, data generation step, editing acceptance step, performance acquisition step, performance judgment step, data adjustment step, etc.

[0181] 4. Function The function of this embodiment can be summarized as follows: In other words, it is possible to provide users with sheet music with any arrangement of any song.

[0182] Although embodiments of the present invention have been described above, the present invention is not limited thereto and can be modified as appropriate without departing from the technical spirit of the invention.

[0183] 5. In the above embodiment, the information processing device 2 performed various storage and control functions, but multiple external devices may be used instead of the information processing device 2. That is, various information and programs may be stored in a distributed manner across multiple external devices using blockchain technology or the like.

[0184] The embodiments of this model are not limited to the information processing system 1, but may also be an information processing method, a program, or a program product. The information processing method comprises each step of the information processing system 1. The program or program product causes a computer to execute each step of the information processing system 1.

[0185] The information processing system 1 may be an integrated system of an information processing device 2 and a user terminal 3. For example, the processing of each functional unit of the information processing system 1 may be executed on the user terminal 3. For example, the processor 33 of the user terminal 3 may have a music acquisition unit 231, a specification acquisition unit 232, a data generation unit 233, an editing reception unit 234, a performance acquisition unit 235, a performance judgment unit 236, a data adjustment unit 237, etc. In other words, the processor 33 of the user terminal 3 may execute each step of the information processing system 1.

[0186] Figure 19 is an activity diagram showing the flow of the information processing method when the processor 33 of the user terminal 3 executes each step of the information processing system 1. Activities A510, A520, A530, and A540 in Figure 19 correspond to activities A110, A120, A130, and A150 in Figure 16, respectively.

[0187] 6. Note: Among users who play musical instruments as a hobby, some purchase commercially available sheet music that has been arranged for solo performance, such as songs by their favorite bands, and enjoy playing solo. However, finding commercially available sheet music is time-consuming, and their favorite songs are not always available in sheet music form. Furthermore, even if their favorite songs are available in sheet music form, the sheet music may not be suitable for their playing level (skill). In addition, Patent Document 1 mentioned above does not envision a service that provides sheet music to users.

[0188] In light of the above circumstances, this disclosure may also provide a method for generating sheet music that allows users to easily obtain sheet music that suits their playing level and preferences.

[0189] Another embodiment of this disclosure will be described below with reference to the drawings. Figure 20 is a diagram showing the configuration of the information processing system 100.

[0190] The information processing system 100 includes a user terminal 110 and a server 120. The components included in the information processing system 100 are connected to each other via a network 130 so that they can communicate with one another. The user terminal 110 is a computer such as a smartphone, tablet, or PC used by a user. The server 120 is a computer that provides predetermined services and applications to the user terminal 110.

[0191] The user terminal 110 includes a control unit 111, a storage unit 112, a communication unit 113, a display unit 114, and an operation unit 115. The control unit 111 includes, for example, a CPU and performs various processes and controls according to a program. The storage unit 112 includes, for example, a storage device, ROM, RAM, etc., and stores various programs and data. The communication unit 113 includes a communication interface and communicates with other devices such as a server 120 via a network 130. The display unit 114 includes, for example, a liquid crystal display or an organic EL display, and displays a screen containing various information and images. The operation unit 115 includes, for example, physical keys or a touch panel integrated with the display unit 114, and is operated by the user.

[0192] The server 120 includes a control unit 121, a storage unit 122, and a communication unit 123. Since the components of the server 120 are the same as those of the user terminal 110, their description is omitted.

[0193] The following describes the processing of the user terminal 110. It is assumed that a music score generation application (hereinafter referred to as the "music score generation app") for generating music scores is pre-installed on the user terminal 110. The music score generation app may be downloaded to the user terminal 110 via the network 130 from, for example, an external server managing an app store or server 120, and then installed. The control unit 111 of the user terminal 110 executes the processing shown in Figures 21 and 23 according to the program contained in the music score generation app stored by the storage unit 112.

[0194] (1) Arrangement Score Generation Service Figure 21 is a flowchart showing an example of the arrangement score generation process by the user terminal 110. Figure 22 is a diagram showing an example of the score generation application screen displayed on the user terminal 110.

[0195] The control unit 111 receives the user's input to select a song to be converted into sheet music via the operation unit 115 (step S11). The user may, for example, select a song that is pre-stored (saved) in the memory unit 112, or they may select and purchase a song sold by an external music distribution service that works in conjunction with the sheet music generation application.

[0196] The control unit 111 determines the instrument (hereinafter referred to as "playing instrument") on which the song selected in step S11 will be written into sheet music and played (step S12). The control unit 111 may determine the playing instrument by accepting an operation from the user to specify the playing instrument, or it may automatically determine the playing instrument based on user information (for example, information on the user's past usage history of sheet music generation applications). The playing instrument may be a keyboard instrument including, for example, a piano, keyboard, electronic organ, or guitar, or any instrument that the user can easily enjoy playing solo. If the sheet music generation application handles only one type of playing instrument (for example, only keyboard instruments), the process in step S12 may be omitted.

[0197] The control unit 111 determines multiple arrangements to be made for the song selected in step S11 (step S13). "Arrangement" may include, for example, adjustment of difficulty level or adjustment of genre. "Multiple arrangements" may include, for example, adjusting to multiple difficulty levels (e.g., beginner, intermediate, advanced) or adjusting to multiple genres (e.g., pop, rock, jazz, etc.). The control unit 111 may accept user input to specify the number and types of arrangements and determine multiple arrangements. Alternatively, the control unit 111 may automatically determine multiple arrangements based on user information. User information may include, for example, information on the user's past usage history of the sheet music generation app, information on the user's favorite songs or genres that have been registered in advance, and information on the user's performance level (performance skill). For example, if the user has frequently played jazz sheet music previously generated by the sheet music generation app, or if the user's favorite genre is jazz, the control unit 111 may automatically decide to make at least one jazz arrangement.

[0198] The control unit 111 generates multiple musical scores based on the song data selected in step S11, the information of the instruments to be played determined in step S12, and the information of multiple arrangements (information indicating the content of the arrangements) determined in step S13. That is, the control unit 111 generates multiple musical scores for instruments, each with multiple arrangements applied to a single song (step S14). The control unit 111 may perform multiple processes in steps in step S14. Specifically, for example, the control unit 111 may perform, in steps, the process of generating a song containing only one instrument from a song containing multiple instrument parts, multiple arrangement processes for the generated song containing only one instrument, and the process of creating musical scores for the multiple arranged songs.

[0199] Furthermore, the control unit 111 may utilize any known technology in at least one of the multiple processes, for example, by using a machine learning model. For example, for the arrangement process, the control unit 111 may use a pre-trained machine learning model based on training data in which the data of the song before arrangement and arrangement information are input data, and the data of the song after arrangement is output data. The machine learning model may be trained on the server 120 and then implemented in the sheet music generation application and used on the user terminal 110, but the process in step S14 itself may be executed on the server 120 side. Also, if the instrument being played is a keyboard instrument, the control unit 111 may generate sheet music corresponding to the number of keys included in the keyboard instrument, or sheet music for an electronic organ including pedal keys, etc. Also, if the instrument being played is a guitar, the control unit 111 may generate sheet music corresponding to the guitar tuning (for example, including drop D tuning).

[0200] The control unit 111 performs a process (display process) to display the information of the multiple musical scores generated in step S14 on the display unit 114 (step S15). In the example shown in Figure 22, a list is displayed that includes information on the titles of songs that have been made into musical scores, namely "AAA," "BBB," and "CCC," and information on three musical scores that have been made into three arrangements (three levels of difficulty adjustment) of the song "AAA." Note that the list means an object that arranges multiple items (information), and its display method is not limited and may include, for example, a pull-down (drop-down) list. The information of the multiple musical scores is not limited to the example shown in Figure 22 and may further include, for example, a link to preview a part of the musical score or a link to play a demo sound of the musical score being played. The control unit 111 accepts the user's operation to select each musical score and may control the display unit 114 to display the selected musical score in full screen or output it in a downloadable PDF format.

[0201] The control unit 111 may, at any time before step S14, request the user to pay the usage fee for the sheet music generation application, and generate the sheet music once the payment is completed. Alternatively, the control unit 111 may, at a time after step S15, request the user to pay the usage fee after they have reviewed the information for multiple sheet music scores, and enable full-screen display or download of the sheet music once the payment is completed. The usage fee may be a fee per song or sheet music, or a fixed fee for a predetermined period (e.g., a monthly fee).

[0202] Subsequently, the control unit 111 acquires performance-related information, which is information related to the user's performance for each musical score displayed in step S15 (step S16). The control unit 111 may acquire performance-related information by receiving user input, or it may acquire performance-related information from data recorded or photographed by the user terminal 110 or an external device of the user's performance of the musical score.

[0203] The control unit 111 displays the performance-related information acquired in step S16 on the display unit 114 in association with the information of the musical score displayed in step S15 (step S17). In the example shown in Figure 22, the performance-related information for each difficulty level of the song "AAA" is displayed, including check marks for scores of difficulty levels that can now be played, information on the period during which the score was played (practiced), and links to play recordings of the performance of the score.

[0204] As described above, the arrangement sheet music generation service generates multiple sheet music scores with multiple arrangements for a single song, and displays information on all of these scores. Therefore, it gives users the opportunity to easily compare multiple sheet music scores with various arrangements for the same song and select the one that best suits their playing level and preferences. It also gives users the opportunity to enjoy and learn about multiple different arrangements and expressions for the same song. Furthermore, since the user's performance-related information is displayed in conjunction with the information of each sheet music score, it is easy for users to review their practice history and reflections for each score.

[0205] The control unit 111 may also allow the user to select which parts of the song to be turned into sheet music to exclude. For example, if the vocal part is selected as a part to be excluded from the sheet music, the control unit 111 may generate sheet music that does not include the vocal part, but instead provides accompaniment for the vocal part.

[0206] (2) Instructional service for arranged musical scores Figure 23 is a flowchart showing an example of the instructional process for arranged musical scores by the user terminal 110. The processes in steps S21 and S22 are the same as the processes in steps S11 and S12, so their explanation is omitted.

[0207] The control unit 111 estimates the user's performance level (performance skill) (step S23). The control unit 111 estimates the performance level by receiving, for example, the user's operation to input information about the performance level. The control unit 111 may also display several questions about the performance level on the display unit 114 and estimate the performance level by receiving the user's operation to input answers to the questions. The questions about the performance level may be, for example, questions about the user's experience playing the instrument, or questions about the user's playing period or frequency. The control unit 111 may also display a test score on the display unit 114 to check the user's performance level, acquire data of the user's performance on the displayed test score that has been recorded or photographed, and estimate the user's performance level based on the acquired data. Alternatively, the control unit 111 may estimate the user's performance level based on both the information about the performance level input by the user and the user's performance data on the test score.

[0208] The control unit 111 determines multiple arrangements (e.g., difficulty level adjustments) to be made to the song selected in step S21, based on the performance level estimated in step S23 (step S24). Then, the control unit 111 generates multiple musical scores based on the song data selected in step S21, the information on the instrument to be played determined in step S22, and the information on the multiple arrangements determined in step S24 (step S25). For example, if the user's performance level is beginner level, the control unit 111 generates multiple musical scores for the instrument to be played, each with multiple arrangements suitable for a beginner-level user. The control unit 111 may also generate multiple musical scores using a machine learning model, similar to step S14.

[0209] The control unit 111 provides the user with an AI-based instruction (lesson) service using the multiple musical scores generated in step S25 (step S26). For example, the control unit 111 acquires data of the user's performance of each musical score, and performs AI-based scoring of the performance and provides feedback to the user. The control unit 111 also suggests to the user that they practice one of the multiple musical scores, and if it determines that the user is able to play it, it suggests to the user that they practice the other musical scores included in the multiple scores. The control unit 111 does not have to generate the multiple musical scores at the same time in step S25, but may generate them sequentially at different times. For example, the control unit 111 may first generate the musical score of one of the multiple arrangements, provide an AI instruction service for that musical score, and then generate the musical score of the next arrangement when it determines that the user is able to play that musical score.

[0210] Furthermore, in step S26, the control unit 111 may return to the process in step S23 at predetermined intervals, or when it determines that the sheet music to be taught is too easy for the user (for example, the AI ​​scoring score is too high), or when it determines that the user has become able to play all of the sheet music to be taught. In this case, the control unit 111 may re-estimate the user's playing level, re-generate new sheet music based on the re-estimated playing level, and continue providing the teaching service.

[0211] As described above, the arrangement sheet music instruction service generates multiple sheet music arrangements for a single song, each based on the user's playing level, and provides instruction using these multiple arrangements. Therefore, users can naturally improve their playing level simply by playing the sheet music of their favorite song as suggested. In addition, users can easily receive AI-powered instruction at home or elsewhere at their own convenience, without having to go to a music school or other venue at a fixed time, thus improving their playing level without difficulty.

[0212] Furthermore, the control unit 111 may accept user input to set a target performance level (target level), and may display information on the display unit 114 such as the difference between the target level and the current performance level, or information showing the trajectory of the performance level's growth relative to the target level. Visualizing the target level and the current performance level can improve the user's motivation to improve their performance level.

[0213] Furthermore, at least a portion of the processes shown in Figures 21 and 23 may be executed on the server 120. In this case, the user terminal 110 does not need to have the music score generation application installed and may access the server 120 through a predetermined website.

[0214] (3) Arrangement Purchase Service In the arrangement processing of the "Arrangement Score Generation Service" and the "Arrangement Score Instruction Service" described above, it was explained that machine learning models may be used, but at least a portion of the training data for the machine learning models may be provided by the user.

[0215] Figure 24 is a flowchart showing an example of the process for purchasing arranged songs by the server 120. The control unit 121 of the server 120 executes the process shown in Figure 24 according to the program stored in the storage unit 122.

[0216] The control unit 121 of the server 120 receives data (upload) from the user terminal 110 of a song that the user has arranged in their own way (hereinafter referred to as the "arranged song") from the original song (hereinafter referred to as the "song before arrangement") (step S31). The control unit 121 may also receive sheet music data of the arranged song from the user terminal 110 along with the arranged song. The control unit 121 then acquires information about the user's arrangement (for example, adjustment of difficulty or adjustment of genre) (step S32). The control unit 121 may acquire the arrangement information by receiving an operation from the user to input the arrangement information, or it may automatically acquire the arrangement information by estimating the difficulty and genre of the arranged song. The user may also acquire the arrangement information by referring to the sheet music data of the arranged song.

[0217] The control unit 121 determines whether the data of the arranged song provided in step S31 can be used as training data for a machine learning model implemented in the sheet music generation application (step S33). The control unit 121 may make the above determination based on predetermined conditions. These predetermined conditions may include that the song data is not corrupted, that the song is not silent, and that the arrangement deserves a predetermined evaluation. The evaluation of the arrangement may be from someone other than the user who provided the song, and the control unit 121 may request evaluations of the arrangement from other users using the "arranged song purchase service". Also, for example, if arrangements of a specific type or arrangements of a specific song have been solicited in advance, the predetermined conditions may include that the song provided in step S31 or the arrangement information obtained in step S32 is included in the solicitation target. The solicitation target may be, for example, a specific type of arrangement for which training data is insufficient, or an arrangement of a specific song for which copyright processing has been completed.

[0218] If the control unit 121 determines that the data of the arranged song cannot be used as training data (step S33: NO), it terminates the process. On the other hand, if the control unit 121 determines that it can be used (step S33: YES), it grants a predetermined reward to the user who provided the arranged song in step S31 (step S34). In this case, the purchase of the arranged song from the user is considered to have been completed.

[0219] Subsequently, the control unit 121 acquires data of the original song corresponding to the arranged song provided in step S31 (step S35). Then, the control unit 121 generates training data using the data of the original song acquired in step S35 and the arrangement information acquired in step S32 as input data, and the data of the arranged song provided in step S31 as output data (step S36). Subsequently, the control unit 121 trains (or retrains) the machine learning model based on the training data generated in step S36 (step S37). The control unit 121 may execute the processes in steps S35 to S37 at different timings than steps S31 to S34, or it may execute them collectively for multiple training data sets when a certain amount of training data has been accumulated.

[0220] As described above, in the arrangement song purchase service, it is determined whether the data of the song arranged by the user can be used as training data for a machine learning model. If it can be used, the user is rewarded. This allows for rewarding users for their arrangement skills and can improve their motivation to arrange songs. Furthermore, it allows for the efficient collection of training data for the machine learning model, improving the quality of the arrangement process.

[0221] At least part of the process shown in Figure 24 may be performed on the user terminal 110 (for example, a sheet music generation application). For example, the user terminal 110 may perform the processes in steps S31 to S33, and if it determines in step S33 that "the data of the arranged song can be used as training data," it may upload the data of the arranged song to the server 120. Furthermore, the decision process in step S33 may include a first decision process performed on the user terminal 110 and a second decision process performed on the server 120 in stages, and the decision criteria for the first and second decision processes may be different. In this case, if the user terminal 110 determines in the first decision process that "the data of the arranged song can be used as training data," it may upload the data of the arranged song to the server 120, and the server 120 may perform the second decision process.

[0222] The product may be provided in any of the following embodiments.

[0223] (1) An information processing method wherein the information processing system performs the following steps: in the music acquisition step, music data is acquired; in the specification acquisition step, arrangement specifications are acquired to specify an arrangement for the music data; and in the data generation step, musical score data is generated by arranging the music data in accordance with the arrangement specifications based on the combination of the music data and the arrangement specifications and first reference information, wherein the first reference information is information relating to the correlation between the combination of the music data and the arrangement specifications and the musical score data.

[0224] With this configuration, it is possible to provide users with sheet music with any arrangement they choose for any song.

[0225] (2) The information processing method described in (1) above, wherein in the specification acquisition step, the arrangement specification is acquired which includes at least an instrument for arrangement, where the instrument for arrangement is an instrument which is played using the musical score data, and in the data generation step, the musical score data is generated which is an arrangement of the musical score data for playing the instrument for arrangement included in the arrangement specification, based on the combination of the musical score data and the arrangement specification and the first reference information.

[0226] With this configuration, it is possible to generate sheet music data for a specific instrument from any given song.

[0227] (3) An information processing method according to (2) above, wherein in the specification acquisition step, the arrangement specification is acquired, which includes at least the instrument for arrangement and the performance level; and in the data generation step, the musical score data is generated, which is arranged from the musical data to be played by the instrument for arrangement included in the arrangement specification and according to the performance level, based on the combination of the musical data and the arrangement specification and the first reference information.

[0228] With this configuration, sheet music data for a specific instrument can be generated from any given song, according to the user's playing skill level.

[0229] (4) An information processing method according to (2) or (3) above, wherein in the specification acquisition step, the arrangement specification is acquired which includes at least a group of instruments for arrangement, where the group of instruments for arrangement is a plurality of instruments that are played together using the musical score data, and in the data generation step, based on the combination of the musical score data and the arrangement specification and the first reference information, the musical score data is generated which is an arrangement of the musical score data for the performance of each instrument in the group of instruments for arrangement included in the arrangement specification.

[0230] With this configuration, it is possible to generate sheet music data for ensemble performance from any given song.

[0231] (5) An information processing method according to any one of (1) to (4) above, wherein the specification acquisition step accepts the selection of an arrangement from among a plurality of arrangement candidates and acquires the arrangement specification corresponding to the selected arrangement.

[0232] With this configuration, users can easily select any arrangement from the presented options.

[0233] (6) An information processing method according to any one of (1) to (5) above, wherein the data generation step further generates sequence data of a musical piece to be played based on the musical score data.

[0234] With this configuration, users can check the performance image of the arranged song by playing back the sequence data corresponding to the musical score data.

[0235] (7) The information processing method described in (6) above, wherein the first reference information includes second reference information and third reference information, and in the data generation step, sequence data is generated based on the combination of the music data and the arrangement specifications and the second reference information, and further, musical score data is generated based on the sequence data and the third reference information, wherein the second reference information is information relating to the correlation between the combination of the music data and the arrangement specifications and the sequence data, and the third reference information is information relating to the correlation between the sequence data and the musical score data.

[0236] With this configuration, the accuracy of the musical arrangement can be improved by generating musical score data via sequence data.

[0237] (8) An information processing method according to any one of (1) to (7) above, wherein the data generation step generates analysis data indicating the musical parameters of the musical music data based on the musical music data and the fourth reference information, and further generates musical score data including the analysis data, wherein the fourth reference information is information relating to the correlation between the musical music data and the analysis data.

[0238] With this configuration, by referring to musical score data that includes analysis data, users can reproduce a performance that closely resembles the original song.

[0239] (9) The information processing method described in (8) above, wherein the analysis data includes at least one of the key, time signature, and bar lines of the musical score.

[0240] This configuration allows us to present users with useful data that contributes to musical performance.

[0241] (10) An information processing method according to any one of (1) to (9) above, wherein in the music acquisition step, recording data is generated by recording the playback sound or performance sound of the target music, and the recording data is acquired as the music data.

[0242] With this configuration, it is possible to generate musical score data even for songs for which musical data cannot be easily obtained.

[0243] (11) An information processing method described in (10) above, wherein in the music acquisition step, data obtained by correcting the sound quality of the recording data is acquired as the music data.

[0244] This configuration allows for improved accuracy in generating musical score data based on recording data of the target song.

[0245] (12) An information processing method according to any one of (1) to (11) above, wherein the editing acceptance step accepts edits from a user to the musical score data.

[0246] With this configuration, users can make further arrangements and modifications to the generated musical score data.

[0247] (13) An information processing method according to (12) above, wherein in the data generation step, sequence data of a musical piece to be played based on the musical score data is further generated, and in the edit acceptance step, the sequence data is modified in accordance with edits made by the user to the musical score data.

[0248] With this configuration, it is possible to synchronize the sheet music data edited by the user with the sequence data corresponding to that sheet music data.

[0249] (14) An information processing method according to any one of (1) to (13) above, wherein in the performance acquisition step, performance data recording the user's performance is acquired; in the performance determination step, the performance level is determined based on the performance data and fifth reference information, where the fifth reference information is information relating to the correlation between the performance data and the performance level; in the specification acquisition step, the arrangement specification including at least the performance level is acquired; and in the data generation step, the musical score data is generated by arranging the musical data according to the performance level included in the arrangement specification based on the combination of the musical data and the arrangement specification and the first reference information.

[0250] With this configuration, it is possible to generate sheet music data from any song, tailored to the user's playing level.

[0251] (15) An information processing method according to any one of (1) to (14) above, wherein in the specification acquisition step, the arrangement specification including at least the performance level is acquired; in the data generation step, the musical score data is generated by arranging the musical score data according to the performance level included in the arrangement specification based on the combination of the musical score data and the arrangement specification and the first reference information; in the performance acquisition step, performance data is acquired that records the user's performance using the musical score data; in the performance determination step, the user's proficiency is determined based on the combination of the musical score data and the performance data and the sixth reference information, where the sixth reference information is information relating to the correlation between the combination of the musical score data and the performance data and the proficiency; and in the data adjustment step, the musical score data is adjusted according to the proficiency.

[0252] With this configuration, the performance level of the sheet music data can be adjusted according to the user's skill level, allowing users to improve their playing ability using the same song.

[0253] (16) An information processing method according to (15) above, wherein in the data adjustment step, a plurality of musical score data arranged differently from each other are presented to the user according to the level of proficiency.

[0254] This configuration allows users to be presented with multiple sheet music data sets with different arrangements tailored to their skill level, thereby increasing their motivation to improve their instrument playing.

[0255] (17) An information processing system comprising at least one processor, wherein the processor is configured to execute each step of the information processing method described in any one of (1) to (16) above by reading a program.

[0256] (18) An information processing system as described in (17) above, comprising a server having the processor and a terminal that can access the server.

[0257] (19) A program that causes a computer to perform each step of the information processing method described in any one of (1) to (16) above. Of course, this is not limited to this.

[0258] Finally, while various embodiments relating to this disclosure have been described, these are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0259] 1: Information processing system, 2: Information processing device, 20: Communication bus, 21: Communication unit, 22: Storage unit, 23: Processor, 231: Music acquisition unit, 232: Specification acquisition unit, 233: Data generation unit, 234: Editing reception unit, 235: Performance acquisition unit, 236: Performance judgment unit, 237: Data adjustment unit, 238: Data management unit, 3: User terminal, 30: Communication bus, 31: Communication unit, 32: Storage unit, 33: Processor, 34: Display unit, 35: Input unit, 36 : Sound output section, AA: Task display area, B10: Create button, B11: Purchase button, B12: File selection button, B13: Cancel button, B21: Level 1 button, B22: Level 2 button, B23: Level 3 button, B24: Level 4 button, B25: Start generation button, B26: Back button, B31: Home button, B32: Save button, B41: Save button, B51: Execute judgment button, B61: View button N, B62: Scoring button, B71: First viewing button, B72: Second viewing button, B73: Song selection button, DA: Data display area, DO: Delete object, EA: Score editing area, EC: Evaluation comment, ED: Score editing screen, EM: Evaluation message, EO: Editing object, EP: Evaluation score, EW: Editing window, FD: Feedback screen, JD: Judgment screen, KF: Keyword input field, LD: Level selection screen, MD: De Data management screen, OD: Result display screen, PO: Playback object, RO: Regeneration object, SA: Score display area, SD: Score display screen, SI: Score data information, SO: Selection object, 100: Information processing system, 110: User terminal, 111: Control unit, 112: Storage unit, 113: Communication unit, 114: Display unit, 115: Operation unit, 120: Server, 121: Control unit, 122: Storage unit, 123: Communication unit, 130: Network

Claims

1. An information processing method wherein the information processing system performs the following steps: in the music acquisition step, music data is acquired; in the specification acquisition step, arrangement specifications are acquired to specify an arrangement for the music data; and in the data generation step, musical score data is generated by arranging the music data according to the arrangement specifications based on the combination of the music data and the arrangement specifications and first reference information, wherein the first reference information is information relating to the correlation between the combination of the music data and the arrangement specifications and the musical score data.

2. An information processing method according to claim 1, wherein the specification acquisition step acquires the arrangement specification which includes at least an instrument for arrangement, where the instrument for arrangement is an instrument which is played using the musical score data, and the data generation step generates the musical score data which is an arrangement of the musical score data for playing the instrument for arrangement included in the arrangement specification, based on the combination of the musical score data and the arrangement specification and the first reference information.

3. An information processing method according to claim 2, wherein in the specification acquisition step, the arrangement specification is acquired, which includes at least the instrument for arrangement and the performance level; and in the data generation step, the musical score data is generated, which is arranged from the musical data to be played by the instrument for arrangement included in the arrangement specification and according to the performance level, based on the combination of the musical data and the arrangement specification and the first reference information.

4. An information processing method according to claim 2 or claim 3, wherein the specification acquisition step acquires the arrangement specification which includes at least a group of instruments for arrangement, where the group of instruments for arrangement is a plurality of instruments which are played together using the musical score data, and the data generation step generates the musical score data which is an arrangement of the musical score data which is an arrangement of the musical score data which is an arrangement specification which includes a combination of the musical score data and the arrangement specification and the first reference information, for the performance of each instrument in the group of instruments for arrangement included in the arrangement specification.

5. An information processing method according to any one of claims 1 to 4, wherein the specification acquisition step includes accepting the selection of an arrangement from a plurality of arrangement candidates and acquiring the arrangement specification corresponding to the selected arrangement.

6. An information processing method according to any one of claims 1 to 5, wherein the data generation step further generates sequence data of a musical piece to be played based on the musical score data.

7. An information processing method according to claim 6, wherein the first reference information includes second reference information and third reference information, and in the data generation step, sequence data is generated based on a combination of the music data and the arrangement specifications and the second reference information, and further, musical score data is generated based on the sequence data and the third reference information, wherein the second reference information is information relating to the correlation between the combination of the music data and the arrangement specifications and the sequence data, and the third reference information is information relating to the correlation between the sequence data and the musical score data.

8. An information processing method according to any one of claims 1 to 7, wherein the data generation step involves generating analysis data indicating musical parameters of the musical music data based on the musical music data and fourth reference information, and further generating musical score data including the analysis data, wherein the fourth reference information is information relating to the correlation between the musical music data and the analysis data.

9. The information processing method according to claim 8, wherein the analysis data includes at least one of the key, time signature, and bar lines of the musical data.

10. An information processing method according to any one of claims 1 to 9, wherein the music acquisition step generates recording data by recording the playback sound or performance sound of the target music, and acquires the recording data as music data.

11. An information processing method according to claim 10, wherein in the music acquisition step, data obtained by correcting the sound quality of the recording data is acquired as the music data.

12. An information processing method according to any one of claims 1 to 11, wherein the editing acceptance step includes accepting edits from a user to the musical score data.

13. An information processing method according to claim 12, wherein the data generation step further generates sequence data of a musical piece to be played based on the musical score data, and the editing acceptance step modifies the sequence data in accordance with the editing of the musical score data by the user.

14. An information processing method according to any one of claims 1 to 13, wherein in the performance acquisition step, performance data recording the user's performance is acquired; in the performance determination step, a performance level is determined based on the performance data and a fifth reference information, where the fifth reference information is information relating to the correlation between the performance data and the performance level; in the specification acquisition step, the arrangement specification including at least the performance level is acquired; and in the data generation step, the musical score data is generated by arranging the musical data according to the performance level included in the arrangement specification, based on the combination of the musical data and the arrangement specification and the first reference information.

15. An information processing method according to any one of claims 1 to 14, wherein in the specification acquisition step, an arrangement specification including at least a performance level is acquired; in the data generation step, a musical score data is generated by arranging the musical score data according to the performance level included in the arrangement specification, based on the combination of the musical score data and the arrangement specification and the first reference information; in the performance acquisition step, performance data is acquired that records a user's performance using the musical score data; in the performance determination step, the user's proficiency is determined based on the combination of the musical score data and the performance data and the sixth reference information, where the sixth reference information is information relating to the correlation between the combination of the musical score data and the performance data and the proficiency; and in the data adjustment step, the musical score data is adjusted according to the proficiency.

16. An information processing method according to claim 15, wherein in the data adjustment step, a plurality of musical score data arranged differently from each other are presented to the user according to the level of proficiency.

17. An information processing system comprising at least one processor, wherein the processor is configured to read a program and perform each step of the information processing method described in any one of claims 1 to 16.

18. An information processing system according to claim 17, comprising: a server having the processor; and a terminal capable of accessing the server.

19. A program that causes a computer to perform each step of the information processing method described in any one of claims 1 to 16.

Images