Quasi-Listing Table Entries for Distributed Object Storage Listing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In geographically distributed data storage systems like Dell EMC's ECS, querying for object listings across remote zones is inefficient due to scattered metadata, leading to slow returns and increased workload.
Innovation Solution
The implementation of quasi-listing table entries, which are temporary and converted to normal entries upon object creation or deletion, allows for efficient object listing by reducing the need for queries across remote zones, with most listing work performed by the zone owning the bucket.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata is distributed across multiple zones for data protection and storage efficiency, then reliability and storage efficiency are improved, but object listing performance deteriorates due to remote queries
Solution Approach 1:
The patent segments the listing table into local and remote parts, maintaining metadata distribution across zones while optimizing the listing operation to first query local segments before remote segments. This segmentation allows the system to preserve data protection through distribution while improving listing speed by minimizing remote queries.
Solution Approach 2:
The patent implements preliminary action by pre-loading frequently accessed metadata into local cache structures before actual listing operations occur. This allows the system to prepare data in advance, reducing the need for remote queries during listing operations and thereby improving speed while maintaining distributed reliability.
2Quantity of substance
If metadata is scattered across directory tables in different zones, then storage efficiency is improved, but query efficiency deteriorates
Solution Approach 1:
The patent merges the listing table with the object table in a way that allows simultaneous access to both structures during listing operations. By combining these tables, the system can retrieve object information and metadata in a single operation, improving query efficiency while maintaining the distributed storage architecture that enables high storage capacity.
Solution Approach 2:
The patent introduces an intermediary caching layer between the distributed directory tables and the listing operation. This intermediary cache stores frequently accessed metadata locally, acting as a mediator that reduces the need to query remote directory tables directly, thereby improving query processing efficiency while preserving the distributed storage structure for high capacity.
3Adaptability or versatility
If asynchronous replication is used for data distribution, then storage efficiency and scalability are improved, but consistency and listing accuracy may be affected
Solution Approach 1:
The patent implements feedback mechanisms that continuously monitor the replication status of metadata across zones. This feedback allows the system to detect when data is fully replicated and adjust listing operations accordingly, ensuring consistency is maintained while allowing asynchronous replication to provide scalability.
Solution Approach 2:
The patent makes the listing operation dynamic by adjusting its behavior based on the replication state of metadata. When metadata is being replicated asynchronously, the system uses special handling to ensure consistency; when replication is complete, normal efficient listing operations proceed. This dynamic approach maintains reliability while enabling scalability through asynchronous replication.
Data Source
AI summary
The described technology is generally directed towards quasi-listing table entries that facilitate the efficient return of an object list in response to a query for relevant objects in a data storage system. A quasi-listing table entry is created when an object is created, and subsequently converted to a normal listing table entry. A normal listing table entry is converted to a quasi-listing table entry when an object is deleted, with the quasi-listing table entry subsequently deleted. When an object list request is received, a normal listing table entry can be used directly to add to the object list; only if a quasi-listing table entry is encountered (before it can be converted or deleted) does the object table need to be accessed to determine if the object actually exists. This eliminates most object table accesses, including those in remote geographic zones, thereby making object listing significantly more efficient.