Data Stream Structure with Size Tree for Backup Object Location
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems are inefficient in identifying and accessing files and objects within data streams due to the lack of effective markers for deduplication processing and the need for exhaustive traversal of data streams for read and write operations, leading to suboptimal management and resource utilization.
Innovation Solution
The implementation of a data stream structure that includes header, application-specific data format (ASDF) sections, data domain markers, and checksums, along with a size tree for efficient object location and virtual full construction, allowing for efficient deduplication and reduced resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If exhaustive traversal of the data stream is used to locate desired objects, then all objects can be accessed, but the time and computational resources required increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing metadata about data stream objects including their positions, sizes, and hierarchical relationships in a catalog structure. This allows the system to locate desired objects directly using the pre-computed metadata without performing exhaustive traversal of the entire data stream, thereby reducing object location time while maintaining complete access capability
Solution Approach 2:
The patent introduces a metadata catalog as an intermediary layer between the data stream and the access operations. This catalog contains pre-computed information about object locations, sizes, and relationships, serving as a mediator that enables direct object location without traversing the entire data stream, thus resolving the contradiction between complete access and fast location
2Productivity
If markers are inserted into data streams to assist deduplication processing, then deduplication efficiency improves, but the data stream structure becomes more complex
Solution Approach 1:
The patent applies segmentation by dividing the data stream into distinct segments or chunks with clear boundaries marked by metadata. Each segment represents a manageable unit for deduplication processing, allowing the system to efficiently identify and process duplicate segments without analyzing the entire data stream as one large unit, thereby improving deduplication efficiency while keeping the structure manageable through systematic division
Solution Approach 2:
The patent introduces structured metadata markers as intermediaries within the data stream that provide organizational information without fundamentally altering the underlying data. These markers serve as guides for deduplication processing, enabling efficient identification of data segments while maintaining a relatively simple structure through the use of lightweight metadata annotations rather than complex structural changes
3Loss of energy
If virtual full construction is performed without scanning operations, then resource consumption is reduced, but the ability to identify and analyze files and objects in the data stream is limited
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing comprehensive metadata about data stream contents including file boundaries, object locations, and hierarchical relationships during the initial data stream creation or previous processing stages. This pre-computed metadata enables virtual full construction to identify and analyze files and objects without performing costly scanning operations, thereby reducing resource consumption while maintaining complete identification capability
Solution Approach 2:
The patent uses copying by creating and maintaining a metadata catalog that replicates essential information about data stream objects including their positions, sizes, and relationships. This metadata copy enables the system to perform file and object identification and analysis operations on the metadata rather than scanning the actual data stream, significantly reducing computational resource consumption while preserving complete identification capability
Data Source
AI summary
Systems and methods are described for backing up files and directories using a common backup format. The files and directories may be represented by objects within a data stream constructed using the common backup format. The data streams may be traversed and updated using a size tree such that modifications are made to individual objects within the data streams without complete traversal. This process results in efficient management of storage systems as read and write operations are not dependent on exhaustive traversal of data streams.