Music Document Auto-Scrolling via Real-Time Audio Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing music reading technologies on mobile devices, such as auto-scroll and audio-to-score alignment, are inadequate for musicians playing from chord sheets or lead sheets, as they require steady tempo, detailed notation, and continuous play, failing to handle pauses, jumps, and background noise effectively.
Innovation Solution
A method that estimates a player's position in a music document using real-time audio signals to automatically scroll the display, based on activity, tonality, and tempo, allowing for intuitive scrolling during pauses and jumps, even with chord-based music without detailed notation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If auto-scroll functionality is used to automatically scroll the music document, then the user's hands are freed for playing, but the scrolling continues uninterrupted even during pauses or jumps backward, causing the display to lose synchronization with the performance
Solution Approach 1:
The system continuously monitors the audio signal during performance and uses it as feedback to dynamically adjust the scrolling behavior. By analyzing the audio input in real-time, the system detects when the performer pauses or jumps backward and automatically adjusts the scroll position to maintain synchronization, resolving the contradiction between hands-free operation and display synchronization.
2Extent of automation
If audio-to-score alignment is used to track performer position, then automatic scrolling can be achieved, but the method fails when only chord charts or lead sheets are available without detailed notation
Solution Approach 1:
The system changes the analysis parameters based on the type of music document. When detailed notation is available, it uses traditional pitch and timing analysis. When only chord charts or lead sheets are available, it switches to analyzing harmonic content and chord progression patterns, allowing automatic scrolling to work effectively with different music document types.
3Device complexity
If the music document scrolls at a constant speed based on tempo, then the scrolling is simple to implement, but it cannot adapt when the performer varies the tempo or rehearses difficult portions at lower speed
Solution Approach 1:
The scrolling speed is made dynamic by continuously estimating the tempo from the audio signal and adjusting the scroll speed accordingly. This allows the system to adapt to tempo variations, slower rehearsal speeds, and rubato expressions while maintaining a relatively simple implementation based on real-time audio analysis.
4Measurement precision
If detailed note-by-note notation is used for score following, then the performer position can be accurately tracked, but chord charts and lead sheets cannot be supported
Solution Approach 1:
The system changes the analysis parameters based on the type of music document. When detailed notation is available, it uses traditional pitch and timing analysis. When only chord charts or lead sheets are available, it switches to analyzing harmonic content and chord progression patterns, allowing automatic scrolling to work effectively with different music document types.
Data Source
AI summary
Indicating what should be played in a piece of music with a music document, including: displaying a part of the music document when a user plays the piece; receiving a real-time audio signal of the playing; automatically determining a playing position within the piece of music based on the real-time audio signal; automatically scrolling the music document on a display depending on the playing position; estimating at least the following from the real-time audio signal: activity; tonality; and tempo used in automatically determining the playing position determined from playing speed of the user. The estimating of the activity includes detecting whether the user is producing any sounding notes. The estimating of the tonality is based on an array of chord models that represent different chords that appear in the music document and allow calculating the probability that the corresponding chord is being played in various real-time audio signal segments.