Datastore Checkpoints for ML Training Data Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing datastores face challenges in efficiently querying and processing large datasets, particularly when ordered by specific fields, which affects performance and accuracy in generating training data for machine learning models.
Innovation Solution
The method involves generating data representing checkpoints within a datastore, where each checkpoint corresponds to a record and represents a common shared value for the field on which the datastore is ordered. This allows for efficient querying and generation of training data for machine learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional querying methods are used on large datasets ordered by specific fields, then complete data retrieval is achieved, but query efficiency deteriorates and performance decreases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing checkpoint values (aggregate statistics like min, max, count) for ordered fields before queries are executed. These checkpoints are generated in advance and stored with the dataset, allowing the query system to quickly determine whether data segments contain relevant records without scanning entire datasets, thus improving query efficiency and reducing access time.
2Measurement precision
If all records are accessed to ensure complete training data, then data accuracy is improved, but the number of records to be accessed increases leading to reduced performance
Solution Approach 1:
The patent extracts only the necessary records for training data generation by using checkpoint values to identify data segments that actually contain records meeting the query criteria. Instead of accessing all records, the system extracts only those segments where the ordered field values fall within the query range, ensuring training data accuracy while improving generation performance by avoiding unnecessary record access.
Data Source
AI summary
The present disclosure is directed to generating datastore checkpoints. In particular, the methods and systems of the present disclosure may generate, within a datastore, data representing multiple checkpoints. Each checkpoint of the checkpoints may correspond to a respective record of the datastore and may represent a common shared value for a field based at least in part on which the datastore is ordered. Based at least in part on the checkpoints, the datastore may be queried to produce one or more responsive records to one or more criteria of the query. Based at least in part on the responsive record(s), training data may be generated. The training data may be utilized for training one or more machine learning (ML) models configured to process input based at least in part on values for the field based at least in part on which the datastore is ordered.