Incremental Data Sampling in Distributed Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face inefficiencies in sampling large data sets, as existing methods require restarting the sampling process when changing the sample size, leading to computational delays and inefficient memory access.
Innovation Solution
A distributed data storage system that utilizes a sampling module to track data block accesses using identifiers, allowing for incremental sampling by combining initial and new data block samples to form cumulative results, enabling faster and more efficient data exploration without reinitiating the sampling task.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the sampling process is restarted when changing sample size, then new sampling results can be obtained, but computational delays increase and memory access becomes inefficient
Solution Approach 1:
The system performs preliminary sampling actions and stores the results in memory. When a new sample size is requested, the system leverages the pre-collected sampling results rather than starting from scratch, thus reducing computational delay while maintaining sampling accuracy.
Solution Approach 2:
The system changes the parameter of sample size by selectively combining or filtering from the existing sampling results stored in memory, rather than re-executing the entire sampling process. This allows efficient adjustment of sample size parameters while minimizing computational overhead.
2Adaptability or versatility
If the sampling process is restarted when changing sample size, then updated sampling results are achieved, but memory access efficiency deteriorates
Solution Approach 1:
The system performs preliminary sampling and stores results in memory for future use. When sample size requirements change, the system accesses and processes the pre-stored results rather than re-reading from storage, significantly improving memory access efficiency while maintaining sample size flexibility.
Solution Approach 2:
The system transitions from a single-dimension approach (restarting sampling) to a multi-dimensional approach by maintaining sampling results in memory and operating on them through filtering, combining, or selective retrieval. This adds the dimension of in-memory processing that improves efficiency.
Data Source
AI summary
A system includes a distributed data storage system disseminated across worker machines connected by a network. A distributed data storage management module has instructions executed by a processor to utilize data block identifiers to track data block accesses to the distributed data storage system. A sampling module with instructions executed by the processor receives a new sample request from a client machine connected to the network. Initial data block samples are gathered from the distributed data storage system during a first time period. A revised sample request is received from the client machine during the first time period. The initial data block samples are gathered. New data block samples are collected from the distributed data storage system. The initial data block samples and the new data block samples are combined to form cumulative data block sample results. The cumulative data block sample results are supplied to the client machine.