Tensor Automatic Differentiation via Contraction Gradient Calculator
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing tensor gradients for complex tensor models, such as tensor networks and tensor decompositions, is challenging due to high time and memory complexities, particularly when using traditional automatic differentiation methods that rely on computing and storing Jacobians or partial derivatives.
Innovation Solution
The introduction of a tensor contraction gradient calculator (TCGC) and a tensor automatic differentiation (TAD) method that computes tensor gradients without the need for explicit partial derivatives or Jacobians, using transformations and auxiliary gradients to efficiently propagate gradients through tensor contractions, reducing time and memory complexities to asymptotically equal those of the evaluation mode.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional automatic differentiation methods are used to compute tensor gradients, then gradient computation is achieved, but time and memory complexities become excessively high
Solution Approach 1:
The patent segments the tensor contraction operation into multiple components: identifying contracting tensors, determining contraction dimensions, computing transformations, and calculating auxiliary gradients. This segmentation allows each step to be optimized independently, reducing overall computational complexity while maintaining gradient accuracy.
Solution Approach 2:
The patent introduces transformations and auxiliary tensor gradients as intermediary elements. Instead of directly computing the full Jacobian matrix, the method uses these intermediaries to propagate gradients through the tensor contraction, significantly reducing memory requirements and computation time while preserving the exact gradient values.
2Measurement precision
If Jacobians or partial derivatives are explicitly computed and stored, then tensor gradients can be updated through differentiation tape or graph, but memory complexity increases significantly
Solution Approach 1:
The patent extracts only the necessary gradient information from the full Jacobian computation. By computing transformations and auxiliary gradients selectively, the method obtains the required tensor gradients without storing the complete Jacobian matrix, thus dramatically reducing memory consumption while maintaining computational accuracy.
Solution Approach 2:
The patent changes the computational parameters from storing full Jacobian matrices to storing only transformations and auxiliary gradients. This parameter change reduces the memory footprint from O(n²) to O(n) while preserving the ability to compute exact tensor gradients through the differentiation process.
3Adaptability or versatility
If tensor contractions with multiple contracting dimensions are performed, then complex tensor models can be evaluated, but computation becomes formidable due to excessive time and memory complexities
Solution Approach 1:
The patent segments the complex tensor contraction into identifiable components: contracting tensors, contraction dimensions, and non-contracting dimensions. This segmentation enables the gradient calculation to proceed through structured steps (computing transformations, then auxiliary gradients) that scale efficiently with tensor complexity, making previously intractable models computationally feasible.
Solution Approach 2:
The patent introduces dynamic computation where transformations and auxiliary gradients are computed on-the-fly during the backward pass rather than being pre-computed and stored. This dynamic approach allows the system to handle complex tensor models with adaptable memory usage, reducing the fixed computational overhead associated with static Jacobian storage.
Data Source
AI summary
Methods and systems are disclosed to reduce the time and memory complexities associated with automatic differentiation of tensor models. The disclosed embodiment consists of a tensor contraction gradient calculator (TCGC) method, a tensor automatic differentiation (TAD) method and a TAD system. The disclosed embodiment eliminates the need to compute partial derivatives or Jacobians for computing tensor gradients of tensor contractions and tensor models. The disclosed embodiment computes tensor gradients of any arbitrary tensor model automatically with both memory and time complexities asymptotically equal to those of the evaluation of tensor models that are theoretically the lowest achievable complexities.