Virtual Performance Audio Synchronization Using Motion Speed Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In virtual music live shows, performance sounds of performers cannot be synchronized due to communication delays, leading to a feeling of strangeness among performers and the audience, which impairs the unity and live feeling of the performance.
Innovation Solution
An information processing device that estimates performance operation speed for each performer based on motion capture data and selects performance sound data accordingly, outputting synchronized sound data to maintain unity and enhance the live feeling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If performance sounds of performers are distributed as they are collected with microphones, then live feeling is improved, but synchronization between performance sounds is impaired due to communication delay
Solution Approach 1:
The system performs preliminary actions by estimating performance operation speeds and selecting appropriate performance sound data before actual performance. Multiple performance sound data are prepared in advance for different operation speeds, and the system selects and outputs the most appropriate one, preventing synchronization issues rather than correcting them after they occur.
Solution Approach 2:
The system uses pre-recorded performance sound data that copies actual performance characteristics at different speeds. Instead of transmitting raw microphone signals directly, the system selects from pre-captured performance sounds that match the estimated operation speed, providing synchronized audio that replicates the live performance feel without communication delay issues.
2Productivity
If performance operation speed is increased for arranged performance, then live feeling is improved, but synchronization with normal performance sound data is impaired
Solution Approach 1:
The system dynamically adjusts performance sound selection based on real-time performance operation speed estimation. Instead of using fixed-speed sound data, the system continuously monitors operation speed and selects from multiple pre-prepared sound data sets with different speeds, allowing the performance to maintain synchronization while adapting to varying tempo requirements.
Solution Approach 2:
The system changes the parameter of performance sound data selection based on operation speed. By preparing performance sound data for multiple operation speeds and selecting the appropriate one based on real-time estimation, the system maintains synchronization while accommodating variations in performance tempo, including both normal and arranged performance speeds.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An information processing device according to the present technology includes: an operation estimation unit that estimates a performance operation speed for each of a plurality of performers whose avatars are arranged in common virtual space based on pieces of motion capture data on the performers; and an output sound control unit that selects performance sound data in accordance with the performance operation speed estimated by the operation estimation unit among pieces of performance sound data prepared for each of performance operation speeds for each of the performers, and performs control to output selected pieces of performance sound data for each of the performers in synchronization with each other.