Storage Controller Pattern Metadata Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As data storage systems grow in size and complexity, they face challenges in efficiently managing large amounts of data, leading to increased storage costs and potential performance issues in distributed storage systems.
Innovation Solution
A data storage subsystem that uses a mapping table to identify and store information about repeating patterns instead of storing multiple instances of the pattern, allowing for efficient reconstruction of data during read operations, thereby reducing storage requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in a distributed storage system, then data availability and functionality are maintained, but storage costs increase as the amount of data grows
Solution Approach 1:
The patent extracts repeating patterns from the data stream and stores them separately in a pattern database, while only storing references to these patterns in the main data structure. This separation allows the system to maintain full data availability while reducing the actual storage footprint of repetitive data.
Solution Approach 2:
Instead of storing multiple physical copies of repeating pattern data, the system creates a single master copy in the pattern database and uses references/pointers to access it from multiple locations. This virtual copying mechanism maintains data availability without proportionally increasing storage costs.
2Adaptability or versatility
If mapping tables are used to translate logical addresses to physical locations, then storage virtualization is enabled, but the amount of metadata to maintain grows with data volume
Solution Approach 1:
The patent combines the mapping table functionality with pattern recognition and compression logic into a unified data structure. The same structure that maps logical to physical addresses also identifies repeating patterns and stores references to them, reducing the separate metadata overhead.
Solution Approach 2:
The data structure serves multiple functions simultaneously: it acts as a mapping table for address translation, a pattern database for compression, and a reference system for efficient data retrieval. This multi-functionality reduces the need for separate metadata structures.
3Reliability
If all data is stored in full to ensure data availability, then data can be reconstructed accurately, but storage requirements increase significantly
Solution Approach 1:
The system extracts repeating pattern sequences from the data and stores them as master copies in a pattern database, while the main data structure contains only references to these patterns. This ensures data can be accurately reconstructed while minimizing storage requirements.
Solution Approach 2:
The system performs preliminary analysis of incoming data to identify repeating patterns before storage, pre-processing the data to separate unique content from repetitive content. This upfront action enables efficient storage without losing the ability to reconstruct complete data accurately.
Data Source
AI summary
A system and method for efficiently storing data in a storage system. A data storage subsystem includes multiple data storage locations on multiple storage devices in addition to at least one mapping table. A data storage controller determines whether data to store in the storage subsystem has one or more patterns of data intermingled with non-pattern data within an allocated block. Rather than store the one or more pattern on the storage devices, the controller stores information in a header on the storage devices. The information includes at least an offset for the first instance of a pattern, a pattern length, and an identification of the pattern. The data may be reconstructed for a corresponding read request from the information stored in the header.