Codon-Level Transformers for Long-Range mRNA Property Prediction

The patent applies preliminary action by implementing a pre-training phase before the main prediction task. The transformer model is first pre-trained on masked language modeling and homology sequence prediction tasks using large amounts of unlabeled mRNA sequence data. This preliminary training enables the model to learn fundamental codon representations, positional relationships, and long-range interactions before being fine-tuned for specific property prediction tasks, thereby achieving high accuracy without excessive complexity during the actual prediction phase.

The patent segments the mRNA sequence into codon units rather than treating it as a continuous nucleotide sequence. Each codon is processed as a discrete token, allowing the model to independently capture position and order information for each codon. This segmentation enables the transformer model to effectively attend to specific codons and their relationships, improving the prediction accuracy by properly accounting for codon-level structural and functional elements.

The pre-training phase performs preliminary action by learning general mRNA sequence patterns and codon representations from large amounts of unlabeled data. This upfront investment in computational resources during pre-training enables the model to achieve high prediction accuracy with fewer labeled examples required for fine-tuning, ultimately reducing the total computational cost compared to training from scratch for each specific prediction task.

The patent implements universality by designing a single transformer model architecture that can perform multiple functions: masked language modeling, homology sequence prediction, and various mRNA property predictions. This multi-functional model learns transferable representations that can be applied across different prediction tasks, reducing the need to train separate models for each property and thereby lowering overall computational resource consumption.

The patent uses the transformer model's self-attention mechanism as an intermediary that captures long-range interactions between codons. The attention mechanism computes relationships between all pairs of codons in the sequence, allowing the model to directly model long-range dependencies without requiring intermediate processing steps. This intermediary mechanism enables the model to adaptively weigh the importance of different codon positions and their interactions, achieving high versatility in capturing various interaction patterns.

The patent transitions from traditional sequence analysis to a dimensional space where codons are represented as high-dimensional vectors. This dimensionality change allows the model to encode complex information about codon position, order, and interactions in a continuous vector space, enabling the capture of long-range interactions through geometric relationships in this high-dimensional space rather than through explicit sequential processing.