Data Transform Accelerator Compression With Uniform Containers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data transform operations in file systems experience performance degradation due to fragmented memory usage and inefficient storage of data in non-uniform block sizes, leading to reduced performance and increased fragmentation.
Innovation Solution
A data transform accelerator employs uniform-sized containers for storing compressed source data, utilizing data compression engines to generate metadata and perform operations in encode and decode directions, reducing memory fragmentation and improving performance through optimized data handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If data is stored in non-uniform block sizes, then data compression can be performed, but memory fragmentation increases and performance degrades
Solution Approach 1:
The patent segments data into fixed-size containers (e.g., 4KB blocks) after compression, rather than storing compressed data in variable-size blocks. This segmentation approach maintains uniform memory allocation while preserving compression benefits, resolving the contradiction between compression efficiency and memory stability.
Solution Approach 2:
The patent introduces a two-dimensional storage structure: first dimension for compressed data (variable size) and second dimension for container allocation (fixed size). By adding the container dimension, the system achieves both compression efficiency and uniform memory block sizes, eliminating fragmentation.
2Loss of substance
If data is stored in non-uniform block sizes, then compression operations can be performed, but loading times increase
Solution Approach 1:
By segmenting compressed data into uniform fixed-size containers, the system enables efficient sequential access and parallel loading operations. The uniform structure allows memory to be pre-allocated and accessed predictably, reducing loading times while maintaining compression benefits.
3Loss of substance
If data is stored in non-uniform block sizes, then compression can be applied, but indexing efficiency decreases
Solution Approach 1:
The uniform container segmentation enables creation of efficient index structures where each container has a predictable position and size. Indexing can be performed using simple arithmetic based on container number and fixed size, dramatically improving indexing efficiency compared to variable-size block structures.
Data Source
AI summary
A method includes obtaining, by a data transform accelerator, a first command and first source data associated with the first command. The method also includes generating, by the data transform accelerator, one or more containers using the first command. Each of the one or more containers may have a container size and may be operable to store compressed source data. The method further includes obtaining a mode of operation for a data compression operation. The method also includes performing the data compression operation to a first portion of the first source data using the mode of operation to obtain the compressed source data. The method further includes storing the compressed source data in a first container of the one or more containers.