Hierarchical Index Entry Estimation by Random Walk Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional RAID systems face challenges with disk failures, increased maintenance costs, security issues due to data duplication, and vulnerability to natural disasters, which affect data integrity and accessibility in cloud storage networks.
Innovation Solution
A dispersed storage network (DSN) using dispersed storage units geographically distributed across multiple sites, employing error encoding and decoding techniques like Cauchy Reed-Solomon encoding to create and manage encoded data slices, ensuring data integrity and security through decentralized storage and retrieval processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is copied to multiple RAID devices for redundancy, then data loss risk is reduced, but security issues arise due to unauthorized access possibilities
Solution Approach 1:
The patent segments data into multiple slices and distributes them across different storage units. Instead of creating complete copies like RAID, the system divides data into fragments (slice 1, slice 2, slice 3, etc.) and stores them separately. This segmentation provides redundancy without creating vulnerable duplicate copies, as reconstructing the original data requires collecting sufficient slices from multiple locations.
Solution Approach 2:
The patent introduces an encoding mechanism as an intermediary between the original data and stored data. Encoding functions transform original data into encoded slices, and decoding functions reconstruct original data from these slices. This intermediary layer ensures that even if multiple slices are accessed, the original data cannot be recovered without the proper decoding process and sufficient number of slices.
2Quantity of substance
If more disks are added to RAID array for increased storage capacity, then storage capacity is improved, but disk failure probability increases leading to higher maintenance costs
Solution Approach 1:
The system segments data into multiple slices that can be distributed across numerous storage units. This segmentation allows the system to scale storage capacity by adding more storage units without proportionally increasing failure risk, as the encoding scheme is designed to tolerate a certain number of failures while maintaining data integrity.
Solution Approach 2:
The patent employs error correction coding with configurable parameters (such as minimum slices required for reconstruction vs. total slices created). By adjusting these parameters, the system can optimize the balance between storage capacity, redundancy level, and tolerance to disk failures. This allows flexible configuration based on specific reliability requirements without linearly increasing maintenance costs with capacity.
3Ease of operation
If RAID devices are co-located for ease of access, then data accessibility is improved, but vulnerability to natural disasters and site-specific failures increases
Solution Approach 1:
The patent segments data into multiple slices that are distributed across different storage units located at different geographical sites. This segmentation combined with geographical distribution ensures that a natural disaster at one location does not result in complete data loss, as slices stored at other locations remain intact and can be used for reconstruction.
Solution Approach 2:
The system adds a geographical dimension to data storage by distributing slices across multiple locations. Instead of storing all data copies at a single accessible location, the system trades some accessibility complexity for improved resilience against location-specific disasters. Data can be reconstructed from slices at any sufficient number of locations, providing both accessibility and disaster resistance.
Data Source
AI summary
Methods and systems for estimating a number of entries in a dispersed hierarchical index. The method and systems involve determining a number of random walks N to perform on the dispersed hierarchical index, conducting N walkthroughs based on the number of walkthroughs, determining a number of walk entries for each of the N random walks and averaging the number of walk entries for each of the N random walks to produce an estimated total number of entries for the dispersed hierarchical index. The determining may be based on one or more of a number of levels, a desired confidence interval, a predetermination, and interpretation of system registry information, and an interpretation of a request. Each random walk starts at a root node and ends at a leaf node through L levels of the dispersed hierarchical index.