Self-Personalized Voice Synthesis for Editable Audio Content
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The barriers to creating audio-based content items include the lack of recording and editing equipment, varying audio quality, discomfort with hearing one's own voice, and the difficulty in editing audio clips compared to text.
Innovation Solution
An online system allows users to generate audio-based content using a voice synthesis model trained on their voice, enabling them to modify the synthetic audio stream and compose voice-based content items, with mood-specific models and editing tools.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If users record audio clips using personal equipment, then they can create audio-based content items, but the audio quality varies greatly and requires professional recording and editing equipment
Solution Approach 1:
The system creates a voice synthesis model that copies the user's voice characteristics by training on multiple voice samples. This allows the system to generate audio content that mimics the user's voice without requiring them to record actual audio clips, thereby maintaining consistent audio quality while eliminating the need for professional recording equipment
Solution Approach 2:
The voice synthesis model acts as an intermediary between the user's text input and the final audio output. Instead of directly recording and editing user audio, the system uses the synthesized voice model to convert text to speech, ensuring consistent audio quality while preserving the user's voice identity
2Adaptability or versatility
If users re-record audio clips to make changes, then they can modify the content, but editing audio is harder than editing text and requires re-recording
Solution Approach 1:
The system replaces the mechanical process of audio recording and editing with a computational text-based editing process. Users can modify the synthesized audio content by editing the underlying text script, which is then re-synthesized using the voice model. This substitution makes editing as easy as text manipulation rather than requiring audio editing skills
Solution Approach 2:
The system performs preliminary voice synthesis based on a text script before the user needs to make changes. When modifications are needed, users simply edit the text and trigger re-synthesis, rather than having to re-record from scratch. This preliminary action of creating editable text-based content enables easy modifications
3Reliability
If users listen to their own voice recordings, then they can review the content, but they feel uncomfortable because the recorded voice sounds different from how they perceive their own voice
Solution Approach 1:
The voice synthesis model is trained to replicate the acoustic parameters of the user's voice, including pitch, timbre, and spectral characteristics. By carefully controlling and matching these parameters, the system generates audio that sounds like the user's natural voice, reducing the discomfort associated with hearing distorted or unnatural voice recordings
Data Source
Figure 1~2
Figure 3
Figure 4A~4B
AI summary
An online system receives, from a client device of a posting user, a script for a voice-based content item. The online system retrieves a voice synthesis model stored in the user profile of the posting user and generates a synthetic audio stream using the retrieved voice synthesis model and based on the received script. The online system presents the generated synthetic audio stream to the posting user and receives instructions for modifying the synthetic audio stream. The online system generates a second audio stream based on the received instructions and composes the voice-based content item based on the generated second audio stream. The voice-based content item is then presented to a viewing user.