Indirect Data Classification in Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing devices face challenges in efficiently managing large volumes of data generated by internal components, requiring significant resources and struggling with data redundancy and compliance with regulatory standards.
Innovation Solution
A method involving erasure coding and deduplication of data, combined with classification mapping, to store data chunks and parity chunks across fault domains, ensuring efficient storage and compliance by associating classification attributes with data chunks without analyzing the data contents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored without deduplication, then storage capacity is maintained for all data copies, but storage resources are wasted due to redundancy
Solution Approach 1:
The data is divided into fixed-size chunks before deduplication processing. Each chunk is independently hashed and compared against existing chunks, allowing precise identification and elimination of redundant segments while preserving unique data portions.
Solution Approach 2:
The system changes the parameter of data representation by converting raw data chunks into hash values for comparison purposes. This transformation enables efficient redundancy detection without requiring direct byte-by-byte comparison of the original data.
2Measurement precision
If data classification is performed by analyzing data contents, then accurate classification is achieved, but processing time and computational resources increase significantly
Solution Approach 1:
Classification labels are assigned to data chunks during the initial deduplication process rather than performing separate content analysis later. The classification information is obtained from metadata or external sources beforehand, allowing it to be integrated into the storage workflow without additional processing delays.
Solution Approach 2:
The system uses an intermediary classification ID to represent data categories instead of directly analyzing data contents. This intermediary identifier links data chunks to classification information stored separately, enabling accurate classification without computational analysis of the actual data.
3Reliability
If erasure coding is applied to all data, then data protection and fault tolerance are improved, but storage efficiency decreases due to redundant parity chunks
Solution Approach 1:
Erasure coding is applied to segmented data chunks rather than entire data objects. This segmentation allows the system to apply protection selectively to unique data portions while managing parity chunk generation more efficiently at a granular level.
Solution Approach 2:
The system applies erasure coding partially by first deduplicating data to identify unique chunks, then applying erasure coding only to the deduplicated set. This partial application of erasure coding reduces the total number of parity chunks needed compared to applying it to all original data copies.
4Ease of operation
If metadata is stored with each data chunk, then data retrieval and management are facilitated, but storage overhead increases
Solution Approach 1:
Metadata for multiple data chunks is merged into consolidated metadata structures rather than storing separate metadata entries for each chunk. The deduplication process enables metadata consolidation by referencing unique chunks multiple times through the same metadata entry with different chunk references.
Solution Approach 2:
Instead of copying full metadata for each data chunk, the system creates lightweight references or pointers to shared metadata entries. This copying approach maintains ease of data management through metadata while minimizing the actual storage overhead by avoiding redundant metadata duplication.
Data Source
AI summary
A method for managing data includes obtaining data from a host, wherein the data is associated with an object identifier (ID), initiating a classification mapping update to obtain a classification entry, applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk, deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks, generating storage metadata associated with the plurality of deduplicated data chunks and the at least one parity chunk, generating an object entry associated with the plurality of data chunks, and the at least one parity chunk, wherein the object entry comprises the object ID and a classification ID, storing the storage metadata and the object entry in an accelerator pool, and storing the plurality of deduplicated data chunks and the at least one parity chunk.