Neural Network Model Fragmentation for Memory-Constrained Serving

The patent segments neural network models into hierarchical fragments organized in a tree structure, where common prefix layers are shared across multiple model variants. This segmentation allows the system to serve multiple model variants without loading complete copies of each model in memory, thereby improving model serving capability while reducing memory usage.

The patent creates a universal base model fragment that serves multiple functions across different model variants. The shared prefix layers are designed to be universally applicable to multiple downstream tasks and model variants, allowing a single memory-resident fragment to support inference for multiple models simultaneously.

The patent divides complete models into modular fragments organized in a hierarchical tree structure. Each fragment represents a functional segment of the model that can be independently deployed and managed. This segmentation maintains model accuracy by preserving the complete computational graph when needed while simplifying deployment through modular organization.

The patent implements a nested fragment structure where smaller model fragments are nested within larger parent fragments, forming a hierarchical tree. This nesting allows the system to deploy only the necessary portions of models for each variant while sharing common ancestor fragments, thereby maintaining accuracy for each variant while reducing overall deployment complexity.

The patent segments large models into smaller hierarchical fragments that can be selectively loaded into accelerator memory based on the specific inference task. This segmentation enables the system to handle large model architectures by loading only the necessary fragment subsets, thereby improving model performance while keeping memory requirements within available accelerator capacity.

The patent introduces a hierarchical dimension to model organization, creating a tree structure of fragments across multiple levels. This dimensional organization allows the system to manage large models by navigating through hierarchical levels and loading only the relevant fragments for each inference request, effectively scaling model performance without linearly increasing memory requirements.

The patent merges multiple model variants into a unified hierarchical fragment tree structure, where common prefix layers are consolidated into shared memory-resident fragments. This merging eliminates redundant memory access overhead by ensuring that common computations are performed once and shared across multiple tasks, thereby maintaining task versatility while reducing inference latency.

The patent designs the shared base fragments to be universally functional across multiple downstream tasks and model variants. This universality allows a single memory-resident fragment to serve multiple inference requests simultaneously without requiring separate memory copies, thereby achieving task versatility while minimizing memory access overhead and inference latency.