Deduplicated Merged Index File System for Scalable Object Storage
Solution Overview
Problem
Conventional indexing systems face scalability issues, requiring additional storage space and server computers, and are costly to maintain due to the need for consistent data management across multiple hard disk drives and computers, with failures leading to accessibility challenges.
Innovation Solution
A deduplicated time-indexed file system using object storage, a distributed database for deduplication indices, and a merged index for namespace metadata, allowing batch processing and simultaneous access by multiple computers, thereby overcoming scalability and cost issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional indexing systems use hard disk drives to store information and create indexes, then the index can be searched, but the system becomes difficult to scale and requires additional storage space and server computers
Solution Approach 1:
The patent divides the indexing system into distributed components across multiple servers, where each server maintains a portion of the index. This segmentation allows the system to scale by adding more servers without requiring a single monolithic system to handle all data, thereby resolving the scalability issue while maintaining index search capability.
Solution Approach 2:
The patent transitions from traditional hard disk drive storage to cloud-based object storage, adding a dimensional shift in how data is stored and accessed. This enables the system to leverage networked storage infrastructure rather than being constrained by local disk capacity, facilitating easier scaling.
2Quantity of substance
If additional duplicative computers are added to manage the index, then the system can handle more data, but the cost to manage consistency among computers increases
Solution Approach 1:
The patent merges the indexing functionality into a unified cloud-based object storage system that can be accessed by multiple computers simultaneously. This consolidation reduces the need for complex inter-computer consistency management while still enabling multiple servers to work together on data storage and retrieval operations.
Solution Approach 2:
The patent introduces a cloud-based object storage system as an intermediary between multiple computers and the data/index. This intermediary layer handles the consistency and coordination requirements, allowing multiple computers to access and manage data without direct complex interactions between them.
3Quantity of substance
If more disk drives and computers are added to the system, then storage capacity increases, but the operational complexity of keeping systems up and running increases
Solution Approach 1:
The patent implements a cloud-based object storage system that provides self-service capabilities for data storage and retrieval. The system automatically handles data distribution, consistency, and availability without requiring manual intervention to manage the complexity of multiple disk drives and computers, thereby reducing operational maintenance complexity.
Solution Approach 2:
The patent changes the fundamental parameters of the storage system by moving from local hard disk drives to cloud-based object storage. This parameter change fundamentally alters how storage capacity is managed and accessed, simplifying operational complexity while enabling scalable storage capacity through networked infrastructure.
4Reliability
If hard disk drives and computers remain operational for index updates and searches, then the index is accessible, but the cost to run the system continuously increases
Solution Approach 1:
The patent enables index updates and searches to be performed periodically or on-demand through the cloud-based object storage system, rather than requiring continuous operation of all hardware components. The system can maintain accessibility while allowing for more flexible operational patterns that reduce continuous energy consumption and operational costs.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A file system may include an object storage, a merged index, and a distributed database. When a file is stored in the file system, the file may be converted to an object and be stored in the object storage. The deduplication index of the file may be stored in the distributed database. The namespace metadata of the file may be stored in the merged index. The merged index generates namespace entries of the file when the file is created, deleted, and/or modified. A namespace entry may be associated with a specific file and may include a creation version and a deletion version. When a file is deleted or modified, instead of modifying the existing namespace entries, new entries associated with different versions and including different creation or deletion versions are created. The status of a file may be monitored by one or more entries associated with a file.