Data Sampling in Storage Systems Using Record Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data preparation technologies are inefficient for large datasets, requiring prohibitive costs and limiting interactive data preparation to small samples, as they fail to support interactive experiences for larger data sets.
Innovation Solution
A computer-implemented method for data sampling that scans a dataset record-by-record, determining whether each record belongs to a random sample and using a buffer to manage storage, replacing records with lower storage scores to maintain a representative subsample, allowing for both data quality issue representation and specific characteristic inclusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data preparation is performed on small samples (e.g., 10,000 records), then processing costs are reduced and interactive experience is enabled, but the sample size is limited and cannot represent larger datasets
Solution Approach 1:
The system performs preliminary actions by pre-selecting and buffering records that meet specific criteria (data quality issues, specific characteristics) before the interactive data preparation session begins. This allows the interactive environment to work with a pre-curated representative sample rather than requiring full dataset processing, enabling both interactivity and broader dataset representation.
Solution Approach 2:
The dataset is segmented into different categories: records with data quality issues, records with specific characteristics, and random samples. This segmentation allows the system to create a representative subsample by selecting from different segments rather than processing the entire dataset uniformly, resolving the contradiction between sample size and representativeness.
2Quantity of substance
If state of the art data preparation technology is used for large datasets, then complete data analysis is possible, but the cost becomes prohibitive
Solution Approach 1:
The system extracts only the necessary records from the large dataset based on specific criteria (data quality issues, specific characteristics, random sampling) and places them in a buffer. This extraction approach allows analysts to work with a smaller, cost-effective sample that still represents the full dataset, avoiding the prohibitive costs of processing complete large datasets while maintaining analytical value.
3Ease of operation
If a buffer is used to store records temporarily, then record management and replacement is enabled, but additional memory storage is required
Solution Approach 1:
The system changes the parameter of buffer size dynamically based on the specific needs of the data preparation task. The buffer is sized to hold a representative sample rather than the entire dataset, and records are replaced in the buffer based on scoring criteria. This parameter adjustment allows efficient record management with minimal memory requirements.
Data Source
AI summary
A computer-implemented method, computer program product and system for data sampling in a storage system. The storage system includes a dataset comprising records and a buffer. The dataset is scanned record-by-record to determine whether the current record belongs to a random sample. If so, then the current record may be added to a first set of records. Otherwise, at least one storage score may be calculated or determined for the current record using attribute values of the current record. Next, it may be determined whether the buffer includes available size for storing the current record. In case the buffer comprises the available size, the current record may be stored in the buffer. Otherwise, at least part of the buffer may be free up. A subsample of the dataset may be provided as a result of merging the first set of records and at least part of the buffered records.