Parameter-Sharing Decoder Pair for Compact Transformer Auto Composition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based language models have a large number of parameters, making them impractical for storage-limited applications due to their size and computational demands, which hinders their deployment in devices with limited resources.
Innovation Solution
A parameter sharing decoder pair (PSDP) is introduced, where two decoders share parameters across layers, reducing the overall parameter count while maintaining the capability to generate understandable compositions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a transformer-based language model with multiple decoder layers is used to generate coherent text compositions, then the quality of text generation is improved, but the number of parameters increases significantly, making the model impractical for storage-limited applications
Solution Approach 1:
The patent merges the parameter sets of multiple decoder layers by sharing parameters across layers. Specifically, the first decoder and second decoder each have their parameters shared across all their respective layers, and further parameter sharing is established between the first and second decoders. This merging reduces the total parameter count while preserving the multi-layer decoding architecture needed for coherent text generation.
Solution Approach 2:
The patent implements universality by making decoder parameters applicable across multiple layers and both decoders. The shared parameters serve multiple functions: they participate in the attention mechanisms of different layers, enable both decoders to generate coherent text, and reduce storage requirements. This multi-functional use of parameters allows the model to maintain generation quality with fewer parameters.
2Adaptability or versatility
If the number of decoder layers is increased to improve text composition capability, then the model's understanding and generation ability is improved, but the storage requirements and learning effort increase
Solution Approach 1:
The patent combines the parameter spaces of multiple decoders and layers through systematic parameter sharing. The first decoder's parameters are shared across all its layers, the second decoder's parameters are shared across all its layers, and parameters are further shared between the two decoders. This merging maintains the adaptability benefits of multiple layers while reducing model size complexity.
Solution Approach 2:
The patent changes the parameter configuration from having independent parameters for each layer and decoder to having shared parameters across layers and decoders. This parameter sharing strategy reduces the total number of parameters while preserving the architectural depth needed for sophisticated text composition, thereby reducing device complexity.
3Manufacturing precision
If a full transformer model with independent parameters for each decoder layer is used, then the model can learn complex patterns, but the learning effort and training time increase significantly
Solution Approach 1:
The patent merges parameter learning across multiple decoders and layers by implementing parameter sharing. Instead of independently learning parameters for each layer and decoder, the model learns a shared set of parameters that serve multiple functions. This reduces the total learning burden while maintaining the capacity to recognize complex patterns through the multi-layer architecture.
Solution Approach 2:
The shared parameters in the patent serve multiple functions across different decoders and layers, allowing the model to learn complex patterns more efficiently. The universal parameters participate in attention mechanisms, sequence generation, and pattern recognition across both decoders, reducing the total learning effort required compared to having independent parameters for each component.
Data Source
AI summary
Techniques for auto composing using a transformer-based language model having a parameter sharing decoder pair (PSDP) that reduces a number of parameters of the model and maintains the capability of generating understandable and reasonable compositions. In one particular aspect, a method is provided that includes obtaining a full encoder sequence, and inputting the full encoder sequence into a transformer model having a PSDP. The PSDP includes: a first decoder having parameters that are shared across all N layers of the first decoder; and a second decoder having parameters that are shared across all N layers of the first decoder. The parameters of the first decoder are different from the parameters of the second decoder. The method further includes using the transformer model to predict sequence elements based on the full encoder sequence, generate an output sequence comprising the sequence elements, and output an output sequence different from the full encoder sequence.