Hybrid Vertex Stream Compression With Dynamic Codec Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for transferring data stored in vertices are inefficient due to the lack of accurate identification of the most effective compression method, leading to wasted time and resources, as they fail to determine whether row-by-row or column-by-column compression is more efficient based on the data type and resource usage.
Innovation Solution
A hybrid intermediate stream format is implemented, where a vertex is compressed into both row-by-row and column-by-column data blocks, with a codec estimating the size of each block to select the more cost-effective method for final compression and transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is compressed using a single fixed method (row-by-row or column-by-column), then the compression process is simple, but the compression ratio may be suboptimal and resources are wasted
Solution Approach 1:
The patent implements dynamic compression method selection by evaluating both row-by-row and column-by-column compression approaches and choosing the optimal method based on actual data characteristics. The system transitions from a static single-method approach to a dynamic multi-method evaluation system that adapts to different data patterns, thereby improving compression ratios without excessive complexity
Solution Approach 2:
The patent changes the parameter of compression method selection from fixed to variable. By introducing evaluation metrics that assess data patterns and resource usage, the system dynamically adjusts which compression method (row-by-row or column-by-column) is applied, optimizing the compression ratio based on actual conditions rather than using a predetermined approach
2Productivity
If compression method is selected without evaluation, then the process is fast, but time and resources are wasted due to inefficient compression
Solution Approach 1:
The patent performs preliminary evaluation of compression methods by assessing data patterns and estimating resource requirements before committing to a full compression operation. This preliminary action involves analyzing data characteristics to predict which method (row-by-row or column-by-column) will be more efficient, thereby avoiding wasted time and resources on suboptimal compression approaches
Solution Approach 2:
The patent implements a feedback mechanism where the system evaluates the results of compression method trials and uses this information to select the optimal method. By measuring actual compression performance and resource usage, the system learns from each evaluation and makes informed decisions about which compression approach to apply, reducing overall time and resource waste
3Productivity
If both row-by-row and column-by-column compression are attempted, then the optimal method can be selected, but additional CPU overhead is incurred
Solution Approach 1:
The patent applies partial action by performing limited evaluation of both compression methods rather than fully executing both compression processes. The system conducts preliminary assessments to determine which method is likely to be more efficient, then applies only that method to the full dataset. This partial evaluation approach achieves near-optimal compression ratios while significantly reducing the CPU overhead that would result from fully processing both methods
Data Source
AI summary
Systems and methods providing a hybrid intermediate stream format are provided. The method includes compressing a vertex into a first data block via a first compression method, compressing the vertex into a second data block via a second compression method, determining a smaller file of the first data block and the second data block, finalizing compression of the vertex via a compression method, selected from the first compression method and the second compression method, corresponding to the determined smaller file of the first data block and the second data block, and transferring the compressed vertex.