JIT Compiler Dynamic Data Format Conversion for AI Kernels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The inefficiencies in data format conversions within artificial intelligence (AI) networks, particularly due to the complex and voluminous nature of data structures, lead to costly operations and reduced implementation efficiency, necessitating more effective systems and methods for handling these conversions.
Innovation Solution
A computer-implemented system utilizing a just-in-time (JIT) compiler to dynamically determine and convert data formats between kernels, eliminating the need for intermediary data structures and optimizing end-to-end execution by embedding postprocessor or preprocessor modules within kernels to ensure efficient input and output data formats based on hardware-specific requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data format conversion is performed using traditional methods in AI networks, then compatibility between different kernels is achieved, but processing efficiency deteriorates due to costly conversion operations
Solution Approach 1:
The patent applies preliminary action by determining the optimal data format for each kernel in advance during the compilation phase, before the actual neural network execution. The JIT compiler analyzes the computational graph and pre-determines which data formats (e.g., NHWC, NCHW) will be most efficient for each kernel operation, eliminating the need for runtime format conversion decisions and reducing overhead during execution.
Solution Approach 2:
The patent introduces an intermediary mechanism in the form of a postprocessor module that acts as a bridge between kernels. This postprocessor dynamically converts data formats between kernels during execution, allowing each kernel to operate in its optimal format while maintaining compatibility in the overall system. The intermediary handles the conversion overhead without affecting the core computation efficiency of individual kernels.
2Ease of manufacture
If static data format conversion is used in AI network compilation, then implementation simplicity is maintained, but runtime efficiency deteriorates due to inability to adapt to hardware-specific requirements
Solution Approach 1:
The patent applies dynamics by transitioning from static data format conversion (determined at compile time) to dynamic format conversion (determined at runtime). The JIT compiler enables the system to adaptively select and switch between different data formats based on runtime conditions, hardware capabilities, and specific computational requirements, allowing the neural network execution to optimize performance for different hardware platforms dynamically.
3Adaptability or versatility
If data format conversion operations are performed frequently in AI networks, then compatibility across different hardware platforms is improved, but processing time increases due to conversion overhead
Solution Approach 1:
The patent applies local quality by allowing different parts of the neural network (different kernels) to use different data formats optimized for their specific computational tasks and hardware requirements. Instead of forcing a uniform data format throughout the entire network, each kernel can operate in its locally optimal format (e.g., some kernels use NHWC while others use NCHW), minimizing the need for format conversions and reducing overall processing time.
Data Source
AI summary
Disclosed are systems and methods for efficiently processing artificial intelligence networks. In some embodiments, a just-in-time (JIT) compiler can determine an efficient input data format for a second kernel or microkernel, which is to digest the output of a first kernel or microkernel. The JIT compiler can generate the first kernel or microkernel with an embedded postprocessor, which can dynamically generate a data structure in the efficient data format as the first kernel or microkernel generates its output. The embedded postprocessor can receive the output stream of the first kernel, individually or in portions and can dynamically populate a data structure in the determined efficient data format to be digested by the second kernel or microkernel.