Streaming Compiler for Automatic Adjoint Differentiation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for automatic differentiation, such as code transformation and operator overloading, face challenges in efficiently computing derivatives of complex functions due to truncation errors, high execution times, and inflexibility, especially when dealing with large numbers of input variables.
Innovation Solution
A computer-implemented method that generates computer code for automatic adjoint differentiation by determining a linearized computational map of the target function, allowing for parallel formation of forward and reverse functions without observing the entire forward function, and includes features like hardcoded static data and overloaded operators for efficient multithreaded execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If code transformation techniques are used for automatic differentiation, then manufacturing precision of derivatives is improved, but device complexity increases due to the complexity of high level computer code languages
Solution Approach 1:
The patent introduces an intermediary representation called the linearized computational map (LCM) that serves as a bridge between high-level source code and optimized derivative code. The LCM breaks down complex functions into elementary operations with explicit data dependencies, making the transformation process more manageable and less error-prone while maintaining derivative accuracy
2Adaptability or versatility
If operator overloading techniques are used for automatic differentiation, then adaptability to arbitrary functions is improved, but productivity decreases due to inferior performance and harder compile-time optimization
Solution Approach 1:
The patent segments the derivative computation process into two distinct phases: forward function evaluation and reverse adjoint computation. This segmentation allows the system to maintain flexibility in handling arbitrary functions through the LCM representation while achieving improved productivity through optimized reverse-mode automatic differentiation that computes all partial derivatives in a single pass
Solution Approach 2:
The patent transitions from traditional single-mode differentiation to a two-dimensional approach combining forward evaluation and reverse adjoint computation. By adding the temporal dimension of computation phases and the structural dimension of the LCM representation, the system achieves both adaptability to arbitrary functions and high performance through compile-time optimization
3Measurement precision
If traditional automatic differentiation is used for functions with large numbers of input variables, then measurement precision of derivatives is improved, but loss of time increases due to execution times proportional to the number of input variables
Solution Approach 1:
The patent performs preliminary action by constructing the linearized computational map and analyzing data dependencies before the actual derivative computation. This upfront preparation enables efficient memory management and allows the reverse adjoint computation to proceed without redundant calculations, significantly reducing computation time for functions with large numbers of input variables while maintaining derivative accuracy
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for operating on a target function to provide computer code instructions configured to implement automatic adjoint differentiation of the target function. The method comprises: determining, based on the target function, a linearized computational map (100), LCM, of the target function wherein each node of the LCM (100) comprises an elementary operation; for each node of the LCM (100) forming computer code instructions configured to: (i) compute intermediate data associated with a forward function of an automatic adjoint differentiation algorithm; and, (ii) increment, according to the automatic adjoint differentiation algorithm, adjoint variables of the preceding connected nodes of the each node in dependence on intermediate data; wherein forming computer code instructions for both step (i) and step (ii) for each node is performed prior to performing said steps for a subsequent node of the LCM (100).