Information processing system, information processing method, and program

Abstract

According to one aspect of the present invention, there is provided an information processing system comprising at least one processor, wherein the at least one processor is configured to execute the following steps by reading a program. In an acquisition step, the at least one processor acquires actual performance data in which an actual performance is recorded. In a main part presentation step, the at least one processor generates, on the basis of the actual performance data and first reference information, main part presentation data for presenting a main part of the actual performance included in the actual performance data. The first reference information includes a correlation between the actual performance data and the main part presentation data.

Description

Information Processing System, Information Processing Method, and Program

[0001] The present invention relates to an information processing system, an information processing method, and a program.

[0002] Patent Document 1 discloses a technique for reproducing an input phrase using an electronic musical instrument.

[0003] Japanese Unexamined Patent Application Publication No. 2022-046904

[0004] With the above technique, any phrase can be reproduced, but performance data obtained by recording performance sounds (that is, recording an actual performance) cannot be provided to the user in a suitable manner.

[0005] In view of the above circumstances, the present invention aims to provide a technique that enables easy use of performance data in which a performance is recorded.

[0006] According to one aspect of the present invention, there is provided an information processing system including at least one processor, the at least one processor being configured to execute the following steps by reading a program. In an acquisition step, performance data in which a performance is recorded is acquired. In a main part presentation step, main part presentation data for presenting a main part of the performance included in the performance data is generated based on the performance data and first reference information, and the first reference information includes a correlation between the performance data and the main part.

[0007] According to the present disclosure, by referring to the main part presentation data together with the performance data or referring to the main part presentation data instead of the performance data, the user can efficiently view the recorded performance.

[0008] This is a diagram showing the configuration of information processing system 1. This is a block diagram showing the hardware configuration of information processing device 2. This is a block diagram showing the hardware configuration of user terminal 3. This is a block diagram showing the functions realized by information processing device 2 (processor 23). This is a diagram showing an example of musical score data SD displayed on user terminal 3. This is a diagram showing an example of waveform data WD displayed on user terminal 3. This is an activity diagram showing the flow of the first form of information processing method. This is an activity diagram showing the flow of the second form of information processing method. This is an activity diagram showing the flow of the third form of information processing method. This is an activity diagram showing the flow of the fourth form of information processing method.

[0009] Incidentally, the program for realizing the software appearing in one embodiment may be provided as a computer-readable non-transitor-readable medium, or it may be provided so that it can be downloaded from an external server, or it may be provided so that the program is launched on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0010] Furthermore, in various information processing according to one embodiment, an input and an output corresponding to the input can be realized. Here, as long as an output is obtained as a result of the input, the form of the information referenced in such information processing (hereinafter referred to as "reference information") is not limited. The reference information may be, for example, rule-based information such as a database, a lookup table, or a predetermined function (including a decision formula such as a regression equation constructed by a statistical method), or a pre-trained model that has learned the correlation between input and output in advance, or a generative AI such as a large-scale language model or visual language model that can output a desired result by inputting a prompt.

[0011] Furthermore, in one embodiment, "part" may include, for example, hardware resources implemented by a circuit in a broad sense, and the information processing of software that can be specifically realized by these hardware resources. Also, in one embodiment, various types of information are handled, and this information can be represented, for example, by the physical values ​​of signal values ​​representing voltage and current, the high or low values ​​of signal values ​​as a set of binary bits composed of 0s or 1s, or by quantum superposition (so-called qubits), and communication and calculations can be performed on a circuit in a broad sense.

[0012] Furthermore, a circuit in a broad sense is a circuit realized by combining at least an appropriate combination of circuits, circuits, processors, and memory. The processor may be a general-purpose processor or a dedicated circuit. In other words, this includes application-specific integrated circuits (ASICs), programmable logic devices (for example, simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.

[0013] 1. Hardware Configuration This section describes the hardware configuration.

[0014] <Information Processing System 1> Figure 1 is a configuration diagram representing Information Processing System 1. Information Processing System 1 comprises an information processing device 2 and a user terminal 3. The information processing device 2 and the user terminal 3 are configured to communicate with each other via a telecommunications line. In one embodiment, Information Processing System 1 consists of one or more devices or components. For example, if Information Processing System 1 consists only of an information processing device 2, then Information Processing System 1 can be the information processing device 2. These components will be described below.

[0015] <Information Processing Device 2> Figure 2 is a block diagram showing the hardware configuration of the information processing device 2. The information processing device 2 comprises a communication bus 20, a communication unit 21, a storage unit 22, and a processor 23. The communication unit 21, the storage unit 22, and the processor 23 are electrically connected within the information processing device 2 via the communication bus 20.

[0016] <Communication Unit 21> The communication unit 21 preferably uses wired communication means such as USB, IEEE 1394, Thunderbolt®, and wired LAN network communication. However, the communication unit 21 may also include wireless LAN network communication, mobile communication such as 3G / LTE / 5G, and Bluetooth® communication as needed. The communication unit 21 may be implemented as a collection of these multiple communication means. That is, the information processing device 2 may communicate various information from the outside via the communication unit 21 and the network.

[0017] <Storage Unit 22> The storage unit 22 stores various types of information as defined above. This can be implemented, for example, as a storage device such as a solid-state drive (SSD) that stores various programs related to the information processing device 2 executed by the processor 23, or as a memory such as a random-access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to program calculations. The storage unit 22 stores various programs and variables related to the information processing device 2 executed by the processor 23.

[0018] <Processor 23> The processor 23 performs processing and control of the overall operation related to the information processing device 2. The processor 23 is, for example, a Central Processing Unit (CPU). The processor 23 realizes various functions related to the information processing device 2 by reading predetermined programs stored in the memory unit 22. That is, information processing by software stored in the memory unit 22 is concretely realized by the processor 23, which is an example of hardware, and can be executed as each functional unit included in the processor 23. In other words, the processor 23 can read programs and execute each functional unit. These will be described in more detail in the next section. Note that the processor 23 is not limited to being a single unit, and the information processing device 2 may have multiple processors 23 for each function.

[0019] <User Terminal 3> Figure 3 is a block diagram showing the hardware configuration of user terminal 3. User terminal 3 comprises a communication bus 30, a communication unit 31, a storage unit 32, a processor 33, a display unit 34, an input unit 35, and a sound output unit 36. The communication unit 31, storage unit 32, processor 33, display unit 34, input unit 35, and sound output unit 36 ​​are electrically connected within user terminal 3 via the communication bus 30. The explanation of the communication unit 31, storage unit 32, and processor 33 is the same as the explanation of each part in the information processing device 2, so it is omitted.

[0020] <Display Unit 34> The display unit 34 displays a graphical user interface (GUI) screen that can be operated by the user. The display unit 34 may be included in the housing of the user terminal 3 or it may be an external device. Specifically, the display unit 34 may be implemented as a display device such as a CRT display, liquid crystal display, organic EL display, or plasma display. It is preferable that these display devices be used in accordance with the type of user terminal 3.

[0021] <Input Unit 35> The input unit 35 receives operation inputs made by the user. The operation inputs are transmitted to the processor 33 via the communication bus 30 as command signals. The processor 33 can perform predetermined controls and calculations based on the transmitted command signals as needed. The input unit 35 may be included in the casing of the user terminal 3 or it may be externally attached. For example, the input unit 35 may be implemented as a touch panel integrated with the display unit 34. When the input unit 35 is implemented as a touch panel, the user can input tap operations, swipe operations, etc. to the input unit 35. Instead of a touch panel, the input unit 35 can be a switch button, mouse, trackpad, QWERTY keyboard, etc.

[0022] <Sound output unit 36> The sound output unit 36 ​​outputs sound (voice). The sound output unit 36 ​​may be a speaker included in the housing of the user terminal 3, or it may be an external speaker.

[0023] The information processing device 2 may be on-premise or in a cloud-based configuration. The cloud-based information processing device 2 may provide the above-mentioned functions and processing in the form of, for example, SaaS (Software as a Service) or cloud computing.

[0024] 2. Functional Configuration This section describes the functional configuration of this embodiment. Information processing by software stored in the memory unit 22 is concretely realized by the processor 23, which is an example of hardware, and can be executed as various functional units included in the processor 23. The information processing system 1 functions as a system for editing demonstration data.

[0025] Figure 4 is a block diagram showing the functions realized by the information processing device 2 (processor 23). Specifically, the information processing device 2 (processor 23) includes an acquisition unit 231, a key part presentation unit 232, a musical score generation unit 233, an accompaniment data generation unit 234, and a playback unit 235.

[0026] <Acquisition Unit 231> The acquisition unit 231 is configured to acquire various data for generating key part presentation data in the key part presentation unit 232, which will be described later. The acquisition unit 231 acquires at least the performance data that records the performance.

[0027] "Demonstration" includes, for example, demonstrations of creative works, demonstrations of competitions, and demonstrations of skills. Demonstrations of creative works include, for example, musical performances (solo or ensemble), singing, dancing (dance, ballet, etc.), acting (drama, comedy, storytelling, magic, street performance, etc.), and recitation. Demonstrations of competitions include competitive matches, practice sessions, training, and exercises. Demonstrations of skills include performing tasks requiring skills and explaining work procedures.

[0028] "Demonstration data" refers to data recorded (video or audio) of a demonstration using user terminal 3 or other equipment. Demonstration data may be video data or audio data, as long as it is data that can play video or sound (audio) continuously.

[0029] The acquisition unit 231 may, for example, accept the selection of demonstration data from the user terminal 3 and acquire the selected demonstration data from a database (a database in the storage unit 22 or a database on an external server). The selection of demonstration data is performed, for example, by inputting the file name, file path, network address (for example, a URL (Uniform Resource Locator)) of the demonstration data. The acquisition unit 231 may also accept the upload of demonstration data from the user terminal 3 and acquire the uploaded demonstration data.

[0030] The acquisition unit 231 may further acquire a specified playback time for the main section presentation unit 232, which will be described later, to generate the digest data. The "specified playback time" is a time that serves as an estimate of the total playback time of the digest data.

[0031] The acquisition unit 231, for example, receives input of a specified playback time from the user terminal 3 and acquires the input specified playback time.

[0032] The acquisition unit 231 may acquire the specified playback time from the user's schedule data. "Schedule data" may be, for example, schedule data on the cloud specified by the user terminal 3 (with file name etc. entered), or schedule data uploaded from the user terminal 3.

[0033] The acquisition unit 231 extracts the viewing period from the acquired schedule data during which the user can view the demonstration data (digest data). The "viewing period" includes, for example, free periods when there are no scheduled events, travel periods when the user is traveling, and specific periods designated by the user. The acquisition unit 231 calculates the specified playback time from the start date and end date and time of the viewing period.

[0034] In this specification, "viewing" includes not only the simultaneous appreciation of images and sound, but also the listening of sound without viewing images (including the playback of only the sound from a video containing audio), and the viewing of images without listening to sound (including the playback of a video containing audio in a muted state).

[0035] The acquisition unit 231 may further acquire performance part information indicating the user's performance part. "Performance part" means the part that the user plays in a musical piece. For example, if a musical piece has only one type of part played by the instrument played by the user, then that instrument's part is the user's performance part. If the instrument played by the user has multiple types of parts, then the part that the user is responsible for becomes the performance part. Note that "performance part information" does not necessarily have to include information indicating each user's individual performance part, and may only include information indicating the type of instrument the user plays.

[0036] The acquisition unit 231 may further acquire playback environment information indicating the user's playback environment. "Playback environment" means whether or not the user can watch the video (whether or not they can watch the video) or whether or not it is appropriate (whether or not it is preferable to watch the video). For example, if the user is driving a vehicle such as a car or bicycle, the playback environment is considered an "environment where video viewing is not possible." Also, for example, if the user is exercising outdoors, such as running, the playback environment is considered an "environment where video viewing is not preferable."

[0037] The playback environment information may be input, for example, by selecting an option that represents the user's environment (e.g., "driving," "running," etc.) presented on the user terminal 3, or it may be generated by the acquisition unit 231 based on location information, movement speed (acceleration), ambient sound, etc. acquired by the user terminal 3.

[0038] The acquisition unit 231 may acquire playback environment information for a specified time or period from the schedule data described above. The "specified time" is, for example, the time specified by the user, the time when playback of the demonstration data or digest data is started by the playback unit 235 described later (the time when the playback instruction from the user is received, or the playback reservation time), etc. The "specified period" is, for example, the period specified by the user, the period during which the demonstration data or digest data is played back by the playback unit 235 described later (playback reservation period), etc.

[0039] For example, if the scheduled playback period schedule data is available time, the acquisition unit 231 acquires playback environment information from the schedule data indicating that "the environment is suitable for watching the video." Here, the "scheduled playback period" is either the period from a specified time until the end of playback of the demonstration data or digest data, or a specified period. Also, for example, if the scheduled playback period schedule data is the driving time of a vehicle, the acquisition unit 231 acquires playback environment information from the schedule data indicating that "the environment is unsuitable for watching the video."

[0040] <Highlight portion presenting unit 232> The highlight portion presenting unit 232 is configured to generate highlight portion presentation data for presenting to the user at the user terminal 3 the highlight portion of the demonstration data in which the demonstration is recorded.

[0041] Specifically, the highlight portion presenting unit 232 generates highlight portion presentation data for presenting the highlight portion of the demonstration included in the demonstration data based on the demonstration data acquired by the acquisition unit 231 and the first reference information.

[0042] The "highlight portion of the demonstration" is an essential part in the recorded content of the demonstration. For example, in the case of a demonstration of a creative work, the portion excluding the portion that does not correspond to the demonstration of the creative work (for example, in the case where the demonstration is a performance, the portion where operations other than the performance itself, such as instrument preparation, tuning, conversation, etc., are being performed) may be regarded as the highlight portion. In the case of a demonstration of a competition or technology, the portion excluding the portion that does not correspond to the demonstration of the competition or technology (for example, in the case where operations other than the competition or technology itself, such as instrument or tool preparation, movement, conversation, etc., are being performed) may be regarded as the highlight portion.

[0043] Further, the "highlight portion of the demonstration" may be obtained by extracting from the portion corresponding to the demonstration of a creative work, competition, technology, etc. included in the demonstration data, the important portion estimated to be important to the user. The "important portion" is determined, for example, by the user's attributes, the momentum of the demonstration, the tension of the demonstration, etc.

[0044] The "user's attributes" include, for example, the demonstration part (the piece of the demonstration target, the performance part, the role, the position, etc.) in the case of a demonstration of a creative work, the event or position in the case of a demonstration of a competition, the occupation in the case of a demonstration of a technology, etc. For example, the highlight portion presenting unit 232 may extract, as the important portion, the portion including the demonstration of the performance part registered by the user from the demonstration data.

[0045] The "momentum of the demonstration" means, for example, the degree of excitement in the demonstration. The degree of excitement is determined, for example, by the relative volume in the demonstration data of the demonstration, the amount of change or the speed of change of the performer's movements, etc., obtained by sound analysis or image analysis. For example, the highlight portion presenting unit 232 may extract, as the important portion, the portion where the performance sound is of a predetermined or greater magnitude, the portion where the width of the performance sound range is of a predetermined or greater width, etc. from the demonstration data.

[0046] "Demonstration tension" means, for example, the degree of tension in a demonstration. The degree of tension is determined, for example, by the presence or absence of mistakes in the demonstration and the number of repetitions of the demonstration, which can be obtained by sound analysis or image analysis. For example, the key part presentation unit 232 may extract, as important parts, the parts where mistakes occur from the demonstration data.

[0047] "Key part presentation data" may be reproducible data obtained by editing or processing the target demonstration data, or data in any format (such as images, text, etc.) that presents the key parts in the target demonstration data. Furthermore, "key part presentation data" may be metadata for controlling the playback of the demonstration data.

[0048] The first reference information includes the correlation between the demonstration data and the key parts. The first reference information may further include the correlation between additional information such as the user's attributes and the key parts.

[0049] The first reference information is stored, for example, in the storage unit 22. Also, the first reference information may be stored in a server outside the information processing device 2. That is, the first reference information may be an external component of the information processing system 1.

[0050] The first reference information may include, for example, a table, a function, a simple algorithm, etc., that shows the correlation between the feature amounts (and additional information as required) extracted from the demonstration data and the key parts of the demonstration data. The correlation included in the first reference information can be constructed, for example, by statistically analyzing the data recorded by combining a plurality of demonstration data and their key parts.

[0051] The first reference information may include a key part determination model that has been pre-trained using machine learning, capable of taking demonstration data (and additional information as needed) as input and outputting key part presentation data. In this case, the key part presentation unit 232 inputs the demonstration data (and additional information as needed) into the key part determination model and causes the key part determination model to output information containing the key parts (key part presentation data). The key part determination model includes at least one of a dedicated pre-trained model for generating key part presentation data and a general-purpose pre-trained model not limited to the use of generating key part presentation data. In the key part determination model, parameters calculated and tuned through learning constitute the correlation of the first reference information.

[0052] Specific machine learning algorithms for pre-trained models include, for example, nearest neighbors, naive Bayes, decision trees, support vector machines, deep learning using neural networks, and regression models. Furthermore, machine learning includes supervised learning, unsupervised learning, and self-supervised learning. The pre-trained model can undergo additional training using new data acquired from the information processing device 2, user terminal 3, etc., through transfer learning or fine-tuning.

[0053] In the essential part determination model, the dedicated pre-trained model is trained using, for example, multiple training demonstration data (and additional information as needed) and combinations of the essential parts of each of the training demonstration data as training data. With this configuration, the essential parts of the demonstration data are determined based on the training demonstration data and its essential parts, so the user can obtain essential part presentation data desired by adjusting the training demonstration data and / or its essential parts.

[0054] Here, by using performance data from a specific performer, not limited to a user, as training data for the essential part determination model, it is possible to determine and present essential parts (parts determined to be essential by each individual performer) that are customized for each performer. In other words, the first reference information may include a dedicated essential part determination model prepared for each individual performer.

[0055] Here, "performing entity" includes not only individual performers (e.g., musicians, conductors, dancers, actors, competitors, skilled individuals, etc.) but also organizations such as orchestras (orchestras, bands, etc.), theater companies, units, sports organizations, and other groups or teams. Furthermore, the key information presentation unit 232 may select a key information determination model to use according to the user's attributes, instructions from the user, etc.

[0056] The general-purpose pre-trained model in the essential part determination model is, for example, a generative AI (Artificial Intelligence) such as a Large-Scale Language Model (LLM) or a Visual Language Model (VLM). When the essential part determination model includes a general-purpose pre-trained model, the essential part presentation unit 232 inputs a prompt to the essential part determination model that includes an instruction to extract essential parts from the demonstration data as input information (or to generate essential part presentation data), causing the essential part determination model to output essential part presentation data.

[0057] The main part presentation unit 232 may generate digest data by extracting the main parts from the performance data as main part presentation data. With this configuration, the user can view only the main parts of the performance data as digest data. For example, the user can efficiently view the parts that need to be checked (for example, the parts that include the user's performance part) from performance data that records a performance such as an ensemble performance.

[0058] When the main section presentation unit 232 generates digest data, the first reference information includes the correlation between the demonstration data and the digest data as a correlation between the demonstration data and the main section. In this case, the main section determination model is a trained model that takes the demonstration data (and additional information as necessary) as input and outputs digest data.

[0059] The main section presentation unit 232 may generate digest data with a playback time within the specified playback time based on the demonstration data, the specified playback time acquired by the acquisition unit 231, and the first reference information. With this configuration, it is possible to generate digest data of a length corresponding to any period during which the user can view the digest data, so that the user can view the digest data during travel time or spare time.

[0060] When the main section presentation unit 232 generates digest data within a specified playback time, the first reference information includes the correlation between the demonstration data and the main section, specifically the correlation between the demonstration data, the specified playback time, and the digest data. In this case, the main section determination model is a trained model that takes the demonstration data (and additional information as necessary) and the specified playback time as input and outputs digest data within the specified playback time.

[0061] Digest data within a specified playback time is generated, for example, by extracting essential parts so that they fit within the specified playback time. Specifically, if the length of the essential parts extracted by the first reference information exceeds the specified playback time, the essential parts presentation unit 232 adjusts the extraction conditions to shorten the length of the essential parts and repeats the extraction of essential parts until the length of the essential parts is less than or equal to the specified playback time.

[0062] The main section presentation unit 232 may generate playback speed data as main section presentation data, which specifies the playback speed of the main section or parts other than the main section of the demonstration data. With this configuration, by increasing the playback speed of parts other than the main section of the demonstration data, the user can efficiently check the main section while watching the entire demonstration data.

[0063] The playback speed data is data that includes instructions such as setting the playback speed of the essential parts of the demonstration data to 1x speed, and the playback speed of the parts other than the essential parts to a speed greater than the playback speed of the essential parts (for example, 2x speed). The playback speed data is read by the playback unit 235, which will be described later, when the playback unit 235 plays back the demonstration data. Alternatively, the playback speed data may be embedded in the demonstration data as metadata.

[0064] Furthermore, the playback speed data may include three or more playback speeds. For example, the playback speed data may include instructions to play non-performance parts such as dialogue at double speed, non-important parts of the performance at 1.5 times the speed, and important parts at 1x speed.

[0065] Furthermore, the main section presentation unit 232 may accept the creation or editing of playback speed data from the user terminal 3. For example, the main section presentation unit 232 may accept input from the user terminal 3 of any part of the demonstration data for which playback speed is to be specified (including parts other than the main section) and the corresponding playback speed. That is, the main section presentation unit 232 may accept the specification of at least one of the playback speed of the main section of the demonstration data and the playback speed of the parts other than the main section from the user terminal 3. Also, for example, the main section presentation unit 232 may accept editing (correction of playback speed) of already created playback speed data from the user terminal 3.

[0066] The key section presentation unit 232 may generate musical score data as key section presentation data, in which information indicating key sections is added to the musical score of the piece included in the performance data. With such a configuration, the user can visually confirm the key sections of the recorded performance on the musical score.

[0067] The musical score referenced by the musical score data (a musical score to which information indicating the main parts is added) may be generated by the musical score generation unit 233 described later, or it may be an existing one (for example, a musical score on the network specified by the user terminal 3, a musical score uploaded from the user terminal 3, etc.).

[0068] "Information indicating key parts" refers to information that specifies, for example, the location of key parts and their display format when the musical score is displayed on the user terminal 3. Examples of display formats for key parts include adding text or icons to the key parts, or decorating the key parts (e.g., enclosing them in a frame, coloring them, or enlarging parts of the musical staff). The information indicating key parts may also include a graded indicator showing the importance of the key parts (e.g., climax, number of mistakes). For example, the key part presentation unit 232 may generate a heatmap that color-codes the musical score based on the degree of importance.

[0069] Figure 5 shows an example of a musical score data SD displayed on the user terminal 3. In the musical score data SD, measures that are determined to be key are displayed in a highlighted state (outlined with a dashed line in Figure 5) compared to other measures.

[0070] The key part presentation unit 232 may generate waveform data as key part presentation data, which is obtained by adding information indicating the key part to the sound waveform data included in the demonstration data.

[0071] Figure 6 shows an example of waveform data WD displayed on the user terminal 3. In the waveform data WD, the parts determined to be important are displayed in a state where they are emphasized compared to other parts (outlined with a dashed line in Figure 6).

[0072] The key part presentation unit 232 may generate key part presentation data that presents key parts corresponding to the performance part, based on the performance data, the performance part information acquired by the acquisition unit 231, and the first reference information. With such a configuration, the user can preferentially view the parts of the performance data that are related to the user's performance part.

[0073] When the key part presentation unit 232 generates key part presentation data corresponding to the performance part, the first reference information includes the correlation between the performance data and the key part, specifically the correlation between the performance data, the performance part information, and the key part. In this case, the key part determination model is a trained model that takes the performance data (and additional information as necessary) and the performance part information as input and outputs key part presentation data corresponding to the performance part.

[0074] The key data presented according to the performance part may be the digest data mentioned above, playback speed data, musical score data, waveform data, or other types of data.

[0075] For example, digest data corresponding to a performance part may include only the performance portion that contains the user's performance part. Also, for example, playback speed data corresponding to a performance part may include an instruction to increase the playback speed of portions other than the performance portion that contains the user's performance part to a speed greater than 1x.

[0076] Furthermore, for example, musical score data or waveform data corresponding to a performance part (data including only the user's performance part, or a score or waveform) may be accompanied by information indicating only the essential parts related to the user's performance part. Alternatively, musical score data or waveform data corresponding to a performance part may be accompanied by information indicating the essential parts of the entire performance (the entire ensemble, including performance parts other than the user's).

[0077] The key part presentation unit 232 may generate digest data as key part presentation data corresponding to the performance part, obtained by extracting key parts from the performance data and removing the performance sound of the user's performance part from the performance data or key parts. With such a configuration, the user can practice their own performance part in accordance with the digest data.

[0078] The main part presentation unit 232, for example, generates digest data of the performance data using the procedure described above, and then separates the performance sounds of the performance parts from the digest data.

[0079] The separation of performance sounds from the performance parts can be performed using sound source separation techniques such as independent component analysis (ICA), non-negative matrix factorization (NMF), or combinations thereof. The performance sounds of the performance parts removed from the performance data may be the user's own performance sounds or the performance sounds of other people.

[0080] The key section presentation unit 232 may generate practice data as key section presentation data corresponding to the performance part, by synthesizing the performance sounds of performance parts other than the user's performance part that are included in the repeatedly played musical sections (key sections) in the performance data. With such a configuration, the user can practice their own performance part in accordance with the practice data.

[0081] The main part presentation unit 232 generates practice data by, for example, extracting the best performance of each performance part from the repeatedly played sections of the song in the performance data and combining these performances. The "best performance" is determined, for example, by comparing a sample with a performance included in the performance data. The sample is, for example, exemplary performance data or sheet music. Alternatively, the "best performance" may be determined, for example, by comparing the repeatedly played sections in the performance data.

[0082] When the main part presentation unit 232 generates practice data corresponding to the performance part, the first reference information includes the correlation between the performance data and the main part, specifically the correlation between the performance data, the performance part information, and the main part. In this case, the main part determination model is a trained model that takes the performance data (and additional information as necessary) and the performance part information as input and outputs practice data corresponding to the performance part.

[0083] In generating various key part presentation data as described above, the key part presentation unit 232 may determine that a key part is a section of a musical piece that is repeatedly played in the performance data and satisfies at least one of the predetermined conditions regarding the number of repetitions and the length of the performance range. With this configuration, in performance data that records practice of a performance, the sections that were repeatedly practiced can be extracted as key parts, allowing the user to prioritize viewing them.

[0084] "Performance range length" refers to, for example, the number of beats or measures. For example, the key section presentation unit 232 determines a musical section as a key section if, in the performance data, a musical section containing a number of measures equal to or greater than a predetermined first threshold is repeated a number of times equal to or greater than a predetermined second threshold. In addition, when determining key sections in this way, a key section determination model developed using machine learning with performance data (and additional information as needed) and key sections extracted according to the number of repetitions or performance range length as training data may be used.

[0085] Furthermore, in generating the various key part presentation data described above, the key part presentation unit 232 may determine that the parts in the performance data in which performance errors occur are key parts. With this configuration, the user can prioritize viewing the parts in the performance data in which practice of playing has occurred.

[0086] The key part presentation unit 232 determines performance errors, for example, by comparing a sample with a performance included in the performance data. The sample is, for example, exemplary performance data or sheet music. In addition, when determining key parts in this way, a key part determination model that has been trained using machine learning with performance data (and additional information as needed) and key parts extracted based on the presence or absence of performance errors as training data may be used.

[0087] The key section presentation unit 232 may generate key section presentation data that includes the key section along with the basis for determining it as a key section (number of repetitions, length of the performance range, occurrence of performance errors, etc.). For example, the key section presentation unit 232 may generate musical score data that includes information indicating the location where a performance error occurred.

[0088] <Score Generation Unit 233> The score generation unit 233 is configured to generate a score for a musical piece that is being performed (recorded) in the performance data.

[0089] Specifically, the score generation unit 233 generates a score for the musical piece included in the performance data based on the performance data acquired by the acquisition unit 231 and the second reference information. With this configuration, the user can play the musical piece being performed in the performance data while referring to the score.

[0090] The score generation unit 233 may generate scores for each song if the performance data includes multiple songs. Furthermore, if the performance data includes repeated performances of all or part of a single song, the score generation unit 233 may generate scores for each song excluding the overlapping performance portions, or it may generate scores that chronologically transcribe the performance content, including the overlapping performance portions.

[0091] The second reference information includes the correlation between performance data and musical scores. The second reference information is stored, for example, in the memory unit 22. Alternatively, the second reference information may be stored on a server outside the information processing device 2. In other words, the second reference information may be an external component of the information processing system 1.

[0092] The second reference information may include, for example, a table, function, or simple algorithm that shows the correlation between features extracted from performance data and the musical score of said performance data. The correlation included in the second reference information can be constructed, for example, by statistically analyzing data recorded by combining multiple performance data and their musical scores. The second reference information may also include existing musical scores. In this case, the musical score generation unit 233 may determine the musical pieces included in the performance data, extract existing musical scores corresponding to the determined musical pieces, and further generate musical scores for the musical pieces included in the performance data based on those scores.

[0093] The second reference information may include a music score generation model that has been pre-trained using machine learning to take performance data as input and output a musical score. In this case, the music score generation unit 233 inputs the performance data into the music score generation model and causes the music score generation model to output a musical score. The music score generation model includes at least one of a dedicated pre-trained model for music score generation and a general-purpose pre-trained model not limited to music score generation. In the music score generation model, parameters calculated and tuned through learning constitute the correlation of the second reference information.

[0094] A dedicated pre-trained model in a music score generation model is trained using, for example, combinations of multiple training performance data sets and the corresponding music scores as training data.

[0095] In a music score generation model, a general-purpose pre-trained model is, for example, a generation AI. When the music score generation model includes a general-purpose pre-trained model, the music score generation unit 233 inputs a prompt to the music score generation model that includes an instruction to generate a musical score from the performance data as input information, and causes the music score generation model to output the musical score.

[0096] The musical score generated by the score generation unit 233 is displayed in response to playback of performance data, digest data, practice data, accompaniment data, etc., by the playback unit 235, which will be described later. Alternatively, the musical score generated by the score generation unit 233 may be displayed independently on the user terminal 3.

[0097] <Accompaniment Data Generation Unit 234> The accompaniment data generation unit 234 is configured to generate accompaniment data for playing back the performance of a musician who is not present in the ensemble (music session).

[0098] Specifically, the accompaniment data generation unit 234 generates accompaniment data that reproduces the performance of the performer included in the generation demonstration data, in accordance with the performance included in the performance data acquired by the acquisition unit 231, based on the generation demonstration data, the performance data, and the third reference information. With this configuration, it becomes possible to perform in an ensemble using accompaniment data of performers who cannot participate in the ensemble in real time.

[0099] "Accompaniment data" is data used to reproduce simulated performance sounds that mimic the performer's playing style. The format of the accompaniment data may be either audio data playable on audio equipment (sound data, video data containing sound, etc.) or sequence data playable on electronic devices (electronic musical instruments, information processing devices, etc.) (MIDI files, etc.).

[0100] "Generative performance data" is data that records performances from which the performer's playing style is extracted. Generative performance data may include only the performance of a single performer, or it may include performances of multiple performers (i.e., ensemble performances). Generative performance data is pre-registered by the user, for example, in a database.

[0101] The third reference information includes the generation demonstration data and the correlation between the demonstration data and the accompaniment data. The third reference information is stored, for example, in the storage unit 22. Alternatively, the third reference information may be stored on a server outside the information processing device 2. In other words, the third reference information may be an external component of the information processing system 1.

[0102] The third reference information may include, for example, a table, function, simple algorithm, etc., that shows the correlation between the features extracted from the generation demonstration data, the features extracted from the demonstration data acquired by the acquisition unit 231, and the accompaniment data. The correlations included in the third reference information can be constructed, for example, by statistically analyzing data recorded by combining multiple generation demonstration data, the demonstration data, and the accompaniment data.

[0103] The third reference information may include an accompaniment generation model that has been pre-trained to take the generation demonstration data and the demonstration data as inputs and output accompaniment data. In this case, the accompaniment data generation unit 234 inputs the generation demonstration data and the demonstration data into the accompaniment generation model and causes the accompaniment generation model to output accompaniment data. The accompaniment generation model includes at least one of a dedicated pre-trained model for generating accompaniment data and a general-purpose pre-trained model not limited to the use of generating accompaniment demonstration data. In the accompaniment generation model, parameters calculated and tuned through learning constitute the correlation of the third reference information.

[0104] In the accompaniment generation model, a dedicated pre-trained model is trained using, for example, training data for generation, training data for performance, and accompaniment data corresponding to these combinations as training data. Here, by using the performance data for generation of a specific performer (performer, etc.) in training the accompaniment generation model, it is possible to reproduce performances (accompaniments) that are tailored to the individuality of each performer. In other words, the third reference information may include a dedicated accompaniment generation model prepared for each individual performer. In this case, the accompaniment data generation unit 234 may select an accompaniment generation model corresponding to a specified performer and output accompaniment data by inputting performance data into the accompaniment generation model. When using an accompaniment generation model prepared (trained) for each performer, the performance data for generation does not need to be input into the accompaniment generation model. In other words, the accompaniment data generation unit 234 may output accompaniment data by inputting only the performance data acquired by the acquisition unit 231 into the accompaniment generation model.

[0105] In the accompaniment generation model, the general-purpose pre-trained model is, for example, a generative AI. When the accompaniment generation model includes a general-purpose pre-trained model, the accompaniment data generation unit 234 inputs a prompt to the accompaniment generation model that includes an instruction to generate accompaniment data from the generation demonstration data as input information and the demonstration data acquired by the acquisition unit 231, causing the accompaniment generation model to output the accompaniment data.

[0106] The accompaniment data generation unit 234 may generate accompaniment data that reproduces the performance of the performer included in the performance data, in accordance with the performance included in the main section data, based on the performance data for generation, the main section presentation data (digest data, practice data, etc.), and the third reference information.

[0107] <Playback Unit 235> The playback unit 235 is configured to play back the performance data acquired by the acquisition unit 231, the key point presentation data (digest data, practice data, etc.) generated by the key point presentation unit 232, and the accompaniment data generated by the accompaniment data generation unit 234. The data played back by the playback unit 235 is output as video and / or sound by the display unit 34 and / or sound output unit 36 ​​of the user terminal 3.

[0108] If playback speed data has been generated by the main part presentation unit 232, the playback unit 235 plays each part of the demonstration data at the playback speed specified in the playback speed data.

[0109] The playback unit 235 may display the sheet music of the song being played on the user terminal 3 when playing back performance data, including performances, on the user terminal 3. The sheet music displayed by the playback unit 235 may be, for example, sheet music included in the sheet music data generated by the key part presentation unit 232 (i.e., sheet music with key parts presented), or sheet music generated by the sheet music generation unit 233 (i.e., plain sheet music without key parts presented). The playback unit 235 may also determine the song included in the performance data, extract the sheet music for that song from the sheet music stored in the database, etc., and display it on the user terminal 3.

[0110] The playback unit 235 may display the playback position of performance data, etc., on the musical score displayed on the user terminal 3. For example, the playback unit 235 may move a marker indicating the playback position on the musical score in synchronization with the playback of performance data, etc., or it may advance the score (switch the displayed page of the musical score).

[0111] The playback unit 235 may, upon receiving instructions from the user terminal 3, switch the performance part for which the musical score is displayed on the user terminal 3. For example, the playback unit 235 may switch between displaying a musical score that includes all performance parts (a score for ensemble performance) and displaying a musical score that includes only a specific performance part (for example, the user's performance part) (a score for part practice).

[0112] The playback unit 235 may, when playing back the accompaniment data on the user terminal 3, display or output to the user terminal 3 key presentation data (for example, musical score data) that was generated based on the performance data referenced when the accompaniment data was generated.

[0113] The playback unit 235 may set the playback position of the performance data to the performance time of the performance data corresponding to the performance position selected by the user via the user terminal 3 on the musical score. With this configuration, the user can visually specify the part of the recorded performance they want to listen to from the musical score.

[0114] Specifically, when the performance data is played back on the user terminal 3, the playback unit 235 presents the musical score included in the musical score data generated by the key section presentation unit 232, or the musical score generated by the musical score generation unit 233, to the user terminal 3 and accepts input from the user regarding the musical score. The playback unit 235 plays back the performance data from the performance position entered on the musical score. For example, if the key section (the enclosed part) is selected on the user terminal 3 for the musical score data SD in Figure 5, the performance data will be played back from the first measure of this key section.

[0115] Furthermore, the playback unit 235 may set the playback position of the performance data to the playback time of the performance data corresponding to the playback position selected by the user terminal 3 on the waveform data generated by the main part presentation unit 232. For example, if the main part (the enclosed portion) is selected by the user terminal 3 for the waveform data WD in Figure 6, the performance data will be played back from the beginning of this main part.

[0116] If the demonstration data is a video, the playback unit 235 may determine, based on the playback environment information, whether to output the demonstration data to the user terminal 3 as sound only or as a video. With this configuration, for example, if the demonstration data is a video, only the sound will be played if the user is unable to view the video, so that the user can listen to the demonstration content without being interrupted while working.

[0117] Specifically, if the playback environment information at the time of playback of the demonstration data (the time when the playback instruction is received from the user, or the playback reservation time) indicates an environment where video can be viewed, the playback unit 235 outputs the demonstration data as a video from the user terminal 3. On the other hand, if the playback environment information at the time of playback of the demonstration data indicates an environment other than an environment where video can be viewed, the playback unit 235 outputs only the sound contained in the demonstration data from the user terminal 3.

[0118] Furthermore, the playback unit 235 may also determine, based on the playback environment information, whether to output video data other than the performance data (digest data, practice data, etc.) to the user terminal 3 as sound only or as video to the user terminal 3.

[0119] 3. Information Processing Method This section describes the information processing method of the information processing device 2. In this information processing method, the functions of each part of the information processing device 2 are executed by the information processing device 2 as steps.

[0120] The first form of the information processing method comprises an acquisition step, a key part presentation step, and a playback step. In the acquisition step, performance data recording a performance is acquired. In the key part presentation step, based on the performance data and first reference information, digest data is generated as key part presentation data that presents the key parts of the performance contained in the performance data. In the playback step, the digest data is played back.

[0121] Figure 7 is an activity diagram showing the flow of the first type of information processing method. Below, the information processing method will be explained in accordance with each activity in this activity diagram.

[0122] First, the user terminal 3 selects the demonstration data for which digest data will be generated (Activity A110). The information processing device 2 acquires the demonstration data selected by the user (Activity A120). Next, the information processing device 2 generates digest data from the demonstration data (Activity A130). After generating the digest data, the information processing device 2 sends a notification to the user terminal 3 indicating that the digest data generation is complete (Activity A140).

[0123] After receiving the completion notification, the user instructs the user terminal 3 to play the digest data (Activity A150). The information processing device 2 receives the playback instruction and transmits the digest data to the user terminal 3 (Activity A160). As a result, the digest data is played on the user terminal 3 (Activity A170). The transmission and playback of the digest data may be in streaming format or download format.

[0124] The second form of the information processing method comprises an acquisition step, a key part presentation step, and a playback step. In the acquisition step, performance data recording the performance is acquired. In the key part presentation step, playback speed data is generated as key part presentation data that presents the key parts of the performance contained in the performance data, based on the performance data and first reference information. In the playback step, the performance data is played back while referring to the playback speed data.

[0125] Figure 8 is an activity diagram showing the flow of the second type of information processing method. Below, the information processing method will be explained in accordance with each activity in this activity diagram.

[0126] First, the user terminal 3 selects the demonstration data for which playback speed data will be generated (Activity A210). The information processing device 2 acquires the demonstration data selected by the user (Activity A220). Next, the information processing device 2 generates playback speed data that specifies the playback speed of the essential or non-essential parts of the demonstration data (Activity A230). After generating the playback speed data, the information processing device 2 sends a notification to the user terminal 3 indicating the completion of the playback speed data generation (Activity A240).

[0127] After receiving the completion notification, the user instructs the user terminal 3 to play the demonstration data for which playback speed data has been generated (Activity A250). The information processing device 2 receives the playback instruction and transmits the playback speed data to the user terminal 3 (Activity A260). As a result, the demonstration data is played back on the user terminal 3 according to the playback speed included in the playback speed data (Activity A270). Alternatively, the information processing device 2 may transmit demonstration data incorporating the playback speed data to the user terminal 3.

[0128] The third form of the information processing method comprises an acquisition step, a score generation step, a key section presentation step, and a playback step. In the acquisition step, performance data recording a performance is acquired. In the score generation step, a score of the musical piece included in the performance data is generated based on the performance data and second reference information. In the key section presentation step, score data is generated as key section presentation data that presents the key parts of the performance included in the performance data, based on the performance data and first reference information. In the playback step, the performance data is played back while the score data is displayed.

[0129] Figure 9 is an activity diagram showing the flow of the third type of information processing method. Below, the information processing method will be explained in accordance with each activity in this activity diagram.

[0130] First, the user terminal 3 selects the performance data to which musical score data will be generated (Activity A310). The information processing device 2 acquires the performance data selected by the user (Activity A320). Next, the information processing device 2 generates a musical score based on the performance data, and then generates musical score data with the essential parts indicated based on the performance data and this musical score (Activity A330). After the musical score data is generated, the information processing device 2 sends a notification to the user terminal 3 that the generation of the musical score data has been completed (Activity A340).

[0131] After receiving the completion notification, the user instructs the user terminal 3 to display the musical score data (or play the performance data that generated the musical score data) (Activity A350). The information processing device 2 receives the display instruction (or playback instruction) and transmits the musical score data to the user terminal 3 (Activity A360). As a result, the musical score data is displayed on the user terminal 3, and the performance data is played in conjunction with it (Activity A370). The information processing device 2 may also transmit performance data incorporating the musical score data to the user terminal 3.

[0132] The fourth form of the information processing method comprises an acquisition step, a key part presentation step, an accompaniment data generation step, and a playback step. In the acquisition step, performance data recording a performance is acquired. In the key part presentation step, key part presentation data is generated that presents the key parts of the performance contained in the performance data, based on the performance data and first reference information. In the accompaniment data generation step, accompaniment data is generated that reproduces the performance of the performer included in the performance data, matching the performance contained in the performance data, based on the performance data for generation (which is the performance data for data generation), the performance data, and third reference information. In the playback step, the accompaniment data is played back.

[0133] Figure 10 is an activity diagram showing the flow of the fourth type of information processing method. Below, the information processing method will be explained in accordance with each activity in this activity diagram.

[0134] First, the user terminal 3 selects the performance data for which accompaniment data will be generated (Activity A410). The information processing device 2 acquires the performance data selected by the user (Activity A420). Next, the information processing device 2 generates accompaniment data based on the pre-prepared performance data for generation and the selected performance data (Activity A430). The accompaniment data may also be generated based on the key part presentation data. After generating the accompaniment data, the information processing device 2 sends a notification to the user terminal 3 indicating the completion of the accompaniment data generation (Activity A440).

[0135] After receiving the completion notification, the user instructs the user terminal 3 to play the accompaniment data (Activity A450). The information processing device 2 receives the playback instruction and transmits the accompaniment data to the user terminal 3 (Activity A460). As a result, the accompaniment data is played on the user terminal 3 (Activity A470). In addition, the main information presentation data may be displayed or played on the user terminal 3 along with the accompaniment data.

[0136] This information processing method is implemented by having the information processing system 1 execute each step using a program. In other words, the program is a program that causes a computer to execute the acquisition step, the essential part presentation step, etc., as described above.

[0137] 4. Function The function of this embodiment can be summarized as follows: The user can efficiently view the recorded demonstration by referring to the key points presentation data along with the demonstration data, or by referring to the key points presentation data instead of the demonstration data.

[0138] Although embodiments of the present invention have been described above, the present invention is not limited thereto and can be modified as appropriate without departing from the technical spirit of the invention.

[0139] 5. In the above embodiment, the information processing device 2 performed various storage and control functions, but multiple external devices may be used instead of the information processing device 2. That is, various information and programs may be stored in a distributed manner across multiple external devices using blockchain technology or the like.

[0140] The embodiments of this model are not limited to the information processing system 1, but may also be an information processing method or a program. The information processing method comprises the steps of the information processing device 2. The program causes the computer to function as the information processing device 2.

[0141] The information processing system 1 may be an integrated system consisting of an information processing device 2 and a user terminal 3. For example, all processing, including the generation of essential information data, may be performed on the user terminal 3.

[0142] The information processing system 1 does not necessarily have to include a score generation unit 233 and an accompaniment data generation unit 234. In other words, the function of generating a score based on performance data and the function of generating accompaniment data based on performance data are not essential functions of the information processing system 1.

[0143] Information processing system 1 may acquire video data or audio data other than performance data as data to be extracted, and generate key part presentation data (for example, digest data with extracted key parts, playback speed data, etc.) that presents the key parts of the data to be extracted. Examples of data to be extracted other than performance data include recording data of meetings, interviews, etc., and voice memos. Key parts of this recording data are, for example, parts where important matters are presented, parts where important people are speaking, parts where keywords related to the theme (agenda) appear, and parts with a large amount of conversation.

[0144] The generation of accompaniment data does not necessarily have to be performed in conjunction with the generation of key section presentation data. In other words, accompaniment data may be generated for performance data for which no key section presentation data is generated. To put it another way, the information processing system 1 may have only the accompaniment data generation unit 234 and not the key section presentation unit 232. Furthermore, the accompaniment data generation unit 234 may accept a musical score as input instead of the performance data acquired by the acquisition unit 231 and output accompaniment data that reproduces the performer's performance of that musical score.

[0145] The product may be provided in any of the following embodiments.

[0146] (1) An information processing system comprising at least one processor, wherein the at least one processor is configured to perform the following steps by reading a program, the acquisition step of acquiring performance data recording a performance, and the main part presentation step of generating main part presentation data that presents the main parts of a performance contained in the performance data, based on the performance data and first reference information, wherein the first reference information includes a correlation between the performance data and the main parts.

[0147] With this configuration, users can efficiently view recorded demonstrations by referring to key point summaries along with the demonstration data, or by referring to key point summaries instead of the demonstration data.

[0148] (2) An information processing system as described in (1) above, wherein in the main part presentation step, the information processing system generates digest data obtained by extracting the main parts from the demonstration data as the main part presentation data.

[0149] This configuration allows users to view only the essential parts of the performance data as a digest. Therefore, for example, users can efficiently view the parts they need to check from performance data that has been recorded, such as an ensemble performance.

[0150] (3) An information processing system as described in (2) above, wherein in the acquisition step, a specified playback time is further acquired, the first reference information includes a correlation between the demonstration data and the main part, specifically the correlation between the demonstration data and the specified playback time and the digest data, and in the main part presentation step, the digest data is generated based on the demonstration data, the specified playback time and the first reference information such that the playback time is within the specified playback time.

[0151] With this configuration, it is possible to generate digest data of a length that corresponds to any period during which the user can view the digest data, allowing users to view the digest data during travel time or spare moments.

[0152] (4) An information processing system according to any one of (1) to (3) above, wherein in the main part presentation step, the information processing system generates playback speed data that specifies the playback speed of the main part or a part other than the main part of the demonstration data as the main part presentation data.

[0153] With this configuration, by increasing the playback speed of parts of the demonstration data other than the essential parts, users can efficiently check the essential parts while watching the entire demonstration data.

[0154] (5) An information processing system according to any one of (1) to (4) above, wherein the processor is configured to further perform the following steps, in the score generation step, generate a score of a musical piece included in the performance data based on the performance data and second reference information, wherein the second reference information includes a correlation between the performance data and the score.

[0155] With this configuration, users can play the music being performed in the performance data themselves while referring to the sheet music.

[0156] (6) An information processing system according to any one of (1) to (5) above, wherein in the essential part presentation step, the information processing system generates musical score data as essential part presentation data, in which information indicating the essential part is added to the musical score of the musical piece included in the performance data.

[0157] With this configuration, users can visually confirm key parts of a recorded performance on the musical score.

[0158] (7) An information processing system as described in (5) or (6) above, wherein the processor is configured to perform the following steps in the playback step, wherein in the playback step, the playback position of the performance data is set to the playback time of the performance data corresponding to the performance position selected by the user on the musical score.

[0159] With this configuration, users can visually select the section of the recorded performance they want to listen to from the musical score.

[0160] (8) An information processing system according to any one of (1) to (7) above, wherein the first reference information includes a key part determination model that has been trained using a combination of a plurality of learning demonstration data, which are learning demonstration data, and the key parts of each of the plurality of learning demonstration data as training data, and in the key part presentation step, the demonstration data is input to the key part determination model, and the key part determination model outputs information including the key parts.

[0161] With this configuration, the essential parts of the demonstration data are determined based on the demonstration data used for training and its essential parts. Therefore, by adjusting the demonstration data used for training and / or its essential parts, the user can obtain the essential parts presentation data they desire.

[0162] (9) An information processing system according to any one of (1) to (8) above, wherein in the essential part presentation step, the system determines that the essential part is one of the musical sections that are repeatedly played in the performance included in the performance data, in which at least one of the number of repetitions and the length of the performance range satisfies a predetermined condition.

[0163] With this configuration, by extracting key sections from the performance data that records the practice of a musical piece, users can prioritize viewing those sections.

[0164] (10) An information processing system according to any one of (1) to (9) above, wherein in the essential part presentation step, the system determines that the part in the performance included in the performance data in which a performance error has occurred is the essential part.

[0165] With this configuration, users can prioritize viewing the parts of the performance data recorded during practice that contain mistakes.

[0166] (11) An information processing system according to any one of (1) to (10) above, wherein in the acquisition step, performance part information indicating the user's performance part is further acquired, the first reference information includes the correlation between the performance data and the essential part as the correlation between the performance data and the essential part, and in the essential part presentation step, essential part presentation data is generated that presents the essential part corresponding to the performance part based on the performance data, the performance part information and the first reference information.

[0167] This configuration allows users to prioritize viewing the parts of the performance data that are relevant to their own performance.

[0168] (12) An information processing system as described in (11) above, wherein in the main part presentation step, the information processing system generates digest data as main part presentation data, obtained by extracting the main part from the performance data and removing the performance sound of the performance part from the performance data or the main part.

[0169] With this configuration, users can practice their own instrumental parts in conjunction with the digest data.

[0170] (13) An information processing system according to any one of (1) to (12) above, wherein the demonstration data is a video, the acquisition step further acquires playback environment information indicating the user's playback environment, the processor is configured to perform the next step further, and the playback step determines, based on the playback environment information, whether to output the demonstration data to the user's terminal as sound only or as a video to the terminal.

[0171] With this configuration, for example, if the demonstration data is a video, only the audio will be played if the user is unable to watch the video, allowing the user to listen to the demonstration content without interrupting their work.

[0172] (14) An information processing system according to any one of (1) to (13) above, wherein the processor is configured to further perform the following steps, in the accompaniment data generation step, generate accompaniment data that reproduces the performance of the performer who is performing the performance included in the generation demonstration data in accordance with the performance included in the performance data, based on the generation demonstration data which is performance data for data generation, the performance data and third reference information, wherein the third reference information includes the correlation between the generation demonstration data, the performance data and the accompaniment data.

[0173] This configuration makes it possible to perform an ensemble using accompaniment data from musicians who cannot participate in the ensemble in real time.

[0174] (15) An information processing method comprising each step of the information processing system described in any one of (1) to (14) above.

[0175] (16) A program that causes a computer to perform each step of the information processing system described in any one of (1) to (14) above. Of course, this is not limited to this.

[0176] Finally, while various embodiments relating to this disclosure have been described, these are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0177] 1: Information processing system, 2: Information processing device, 3: User terminal, 20: Communication bus, 21: Communication unit, 22: Memory unit, 23: Processor, 30: Communication bus, 31: Communication unit, 32: Memory unit, 33: Processor, 34: Display unit, 35: Input unit, 36: Sound output unit, 231: Acquisition unit, 232: Key part presentation unit, 233: Score generation unit, 234: Accompaniment data generation unit, 235: Playback unit, SD: Score data, WD: Waveform data

Claims

1. An information processing system comprising at least one processor, the at least one processor configured to perform the following steps by reading a program, the acquisition step of acquiring performance data recording a performance, and the key part presentation step of generating key part presentation data that presents the key parts of a performance contained in the performance data, based on the performance data and first reference information, wherein the first reference information includes a correlation between the performance data and the key parts.

2. An information processing system according to claim 1, wherein in the main part presentation step, the information processing system generates digest data obtained by extracting the main parts from the demonstration data as the main part presentation data.

3. An information processing system according to claim 2, wherein the acquisition step further acquires a specified playback time, the first reference information includes a correlation between the demonstration data and the main part, specifically the correlation between the demonstration data, the specified playback time, and the digest data, and the main part presentation step generates the digest data, with the playback time being within the specified playback time, based on the demonstration data, the specified playback time, and the first reference information.

4. An information processing system according to any one of claims 1 to 3, wherein in the main part presentation step, the system generates playback speed data specifying the playback speed of the main part or a part other than the main part of the demonstration data as the main part presentation data.

5. An information processing system according to any one of claims 1 to 4, wherein the processor is configured to further perform the following steps: in the score generation step, generate a score of a musical piece included in the performance data based on the performance data and second reference information, wherein the second reference information includes a correlation between the performance data and the score.

6. An information processing system according to any one of claims 1 to 5, wherein in the essential part presentation step, the information processing system generates musical score data as essential part presentation data, in which information indicating the essential part is added to the musical score of the musical piece included in the performance data.

7. An information processing system according to claim 5 or claim 6, wherein the processor is configured to further perform the following steps: in the playback step, the playback position of the performance data is set to the playback time of the performance data corresponding to the performance position selected by the user on the musical score.

8. An information processing system according to any one of claims 1 to 7, wherein the first reference information includes a main part determination model that has been trained using a combination of a plurality of learning demonstration data, which are learning demonstration data, and the main parts of each of the plurality of learning demonstration data as training data, and in the main part presentation step, the demonstration data is input to the main part determination model, and the main part determination model outputs information including the main parts.

9. An information processing system according to any one of claims 1 to 8, wherein in the essential part presentation step, the system determines that the essential part is one of the musical sections that are repeatedly played in the performance included in the performance data, and in which at least one of the number of repetitions and the length of the performance range satisfies a predetermined condition.

10. An information processing system according to any one of claims 1 to 9, wherein in the essential part presentation step, the system determines that the part in the performance included in the performance data in which a performance error has occurred is the essential part.

11. An information processing system according to any one of claims 1 to 10, wherein the acquisition step further acquires performance part information indicating the user's performance part, the first reference information includes a correlation between the performance data and the essential part, and the essential part presentation step generates essential part presentation data that presents the essential part corresponding to the performance part based on the performance data, the performance part information and the first reference information.

12. An information processing system according to claim 11, wherein in the main part presentation step, the information processing system generates digest data obtained by extracting the main part from the performance data and removing the performance sound of the performance part from the performance data or the main part.

13. An information processing system according to any one of claims 1 to 12, wherein the demonstration data is a video, the acquisition step further acquires playback environment information indicating the user's playback environment, the processor is configured to further perform the following steps, and the playback step determines, based on the playback environment information, whether to output the demonstration data to the user's terminal as sound only or as a video.

14. An information processing system according to any one of claims 1 to 13, wherein the processor is configured to further perform the following steps: in the accompaniment data generation step, the processor generates accompaniment data that reproduces the performance of a performer included in the generation demonstration data in accordance with the performance included in the performance data, based on the generation demonstration data which is performance data for data generation, the performance data and third reference information, wherein the third reference information includes the correlation between the generation demonstration data, the performance data and the accompaniment data.

15. An information processing method comprising each step of the information processing system described in any one of claims 1 to 14.

16. A program that causes a computer to perform each step of the information processing system described in any one of claims 1 to 14.

Images