Dynamic Batch Assembly for Neural Network Embedding Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine intelligence systems, particularly those employing artificial neural networks (ANNs), are computationally and energy intensive due to the large data structures and immense number of discrete computations required, making them resource-hungry and difficult to parallelize.
Innovation Solution
The proposed solution involves a method and system for accelerating the execution of directed graphs, specifically by generating and rearranging batches of indices to access embeddings efficiently, using a network of computational nodes, including a multicore processor, to optimize data access and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional ANNs are executed with fully connected layers, then computational accuracy is maintained, but computational resources and execution time increase with power law relationship to layer size
Solution Approach 1:
The patent segments the fully connected layer computations into batched operations with dynamic batch assembly. Instead of processing all elements simultaneously, the system divides computations into manageable batches that can be processed in parallel, reducing the power law growth of computational resources while maintaining accuracy through progressive batch processing.
Solution Approach 2:
The patent implements dynamic batch assembly where the batch size and composition are adjusted during execution based on data access patterns and computational requirements. This dynamic adaptation allows the system to optimize resource usage by varying batch dimensions to match the actual computational needs of each layer, preventing excessive resource consumption.
2Productivity
If traditional ANNs process all elements simultaneously, then computational completeness is ensured, but parallelization difficulty increases due to layer dependencies
Solution Approach 1:
The patent segments the sequential layer processing into parallelizable batch operations. By dividing the computation into batches that can be processed independently and in parallel, the system maintains computational completeness while reducing parallelization complexity through systematic batch organization and memory access optimization.
Solution Approach 2:
The patent performs preliminary batch assembly and memory access optimization before executing computations. By pre-organizing data into batches and preparing memory access patterns in advance, the system enables parallel processing without increasing complexity, as the batch structure is established beforehand to facilitate parallel execution.
3Adaptability or versatility
If deep learning ANNs with recursive relationships are implemented, then model capability is enhanced, but computational intensity and resource requirements increase
Solution Approach 1:
The patent segments the recursive computations in deep learning models into batched operations that can be processed more efficiently. By dividing the computational graph into manageable batches and optimizing memory access patterns, the system reduces the computational intensity required to achieve the same model capability through recursive relationships.
Solution Approach 2:
The patent changes key computational parameters including batch size, memory access patterns, and data organization to optimize the execution of deep learning models. These parameter changes enable the system to handle recursive relationships with reduced computational intensity by adjusting how data is processed and accessed during model execution.
4Ease of operation
If embeddings are accessed sequentially, then memory access simplicity is maintained, but execution latency increases due to sequential processing constraints
Solution Approach 1:
The patent implements dynamic batch assembly that adapts memory access patterns based on the actual data and computational requirements. Instead of fixed sequential access, the system dynamically organizes batches to optimize memory access, reducing latency while maintaining operational simplicity through automated batch management.
Solution Approach 2:
The patent performs preliminary batch organization and memory access planning before executing embedding lookups. By pre-assembling batches and preparing access patterns in advance, the system reduces execution latency without complicating the memory access process, as the optimization is performed beforehand rather than during execution.
Data Source
AI summary
Methods and systems for the accelerated execution of a directed graph are disclosed. The execution can involve the generation of an inference from a set of inputs provided to an artificial neural network. In a specific example, a method for executing a directed graph includes receiving at least two batches of indices. The batches of indices, when used to access a set of embeddings, provide at least two batches of embedding outputs and execute a layer of the directed graph. The method further includes accessing the set of embeddings using the at least two batches of indices. The method further includes rearranging, based on a set of latencies for the accessing step, the at least two batches of embedding outputs into at least two batches of rearranged embeddings. The method further includes providing the at least two batches of rearranged embeddings to a subsequent layer of the directed graph.