Data Stream Structure for Efficient Virtual Full Backup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems are inefficient in identifying and accessing files and objects within data streams due to the lack of effective markers for deduplication processing and the need for exhaustive traversal of data streams for read and write operations, leading to suboptimal storage management.
Innovation Solution
The implementation of a data stream structure that includes header objects, application-specific data format (ASDF) sections, data domain markers, and checksums to facilitate efficient deduplication and access, using a size tree to dynamically calculate object offsets and support virtual full construction with minimal content shifting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If current data storage formats are used without markers, then the storage system can maintain a simpler structure, but the identification and analysis of files and objects in data streams becomes inefficient
Solution Approach 1:
The data stream is segmented into distinct components including header objects, ASDF sections, data domain markers, and checksums. Each segment serves a specific function in facilitating efficient identification and deduplication processing, resolving the contradiction between efficiency and complexity by organizing the data stream into manageable, purpose-driven segments
Solution Approach 2:
Data domain markers are introduced as intermediary elements within the data stream to assist deduplication heuristics processing. These markers act as mediators that enable efficient identification of files and objects without requiring complex traversal of the entire data stream, thus improving productivity while maintaining manageable structure
2Speed
If exhaustive traversal of data stream is used to locate objects, then the storage system can maintain a simpler access mechanism, but the access efficiency for read and write operations deteriorates
Solution Approach 1:
Header objects and data domain markers are inserted into the data stream in advance to indicate the locations of files and objects. This preliminary action enables direct access to desired objects without requiring exhaustive traversal, significantly improving access speed while the structured format keeps the access mechanism manageable
Solution Approach 2:
The mechanical traversal process is replaced with a marker-based location system. Instead of sequentially scanning through the data stream to locate objects, the system uses pre-positioned markers and headers to directly identify object locations, substituting the mechanical traversal with a more efficient marker-based retrieval mechanism
3Productivity
If deduplication processing is performed without markers, then the processing logic can remain simpler, but the efficiency of deduplication heuristics deteriorates
Solution Approach 1:
Data domain markers serve as intermediaries that facilitate deduplication heuristics processing by clearly delineating data boundaries and domains. These markers enable the processing logic to efficiently identify and compare data segments without requiring complex analysis, thus improving deduplication efficiency while keeping the processing logic relatively simple
Solution Approach 2:
The data stream uses distinct markers and headers that visually (conceptually) differentiate between various data domains and objects. This marker-based differentiation enables deduplication processing to quickly identify comparable data segments, improving processing efficiency through clear visual (structural) cues rather than complex logic
4Adaptability or versatility
If locations of files and objects change over time due to stream modifications, then the data stream can remain flexible and dynamic, but the efficiency of accessing specific objects deteriorates
Solution Approach 1:
The data stream structure is designed to be dynamic, allowing modifications and changes to be made while maintaining the header object and marker framework. This dynamic structure enables the data stream to adapt to changes while the persistent markers continue to provide efficient location information, resolving the contradiction between flexibility and access speed
Solution Approach 2:
The header objects and data domain markers provide continuous location information that feedbacks the system about object positions within the data stream. Even when modifications cause location changes, the markers are updated to reflect new positions, providing accurate feedback that maintains efficient access speed while allowing the data stream to remain flexible and adaptive
Data Source
AI summary
Systems and methods are described for backing up files and directories using a common backup format. The files and directories may be represented by objects within a data stream constructed using the common backup format. The data streams may be traversed and updated using a size tree such that modifications are made to individual objects within the data streams without complete traversal. This process results in efficient management of storage systems as read and write operations are not dependent on exhaustive traversal of data streams.