ML-Based Data Object Reference Prediction for Batch Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face inefficiencies and increased latency due to repeated connections and queries to a datastore for dynamic object dependencies in graph structures, leading to sub-optimal performance in loading data objects for processing operations.
Innovation Solution
A machine learning engine predicts a set of data object references required for a processing operation, allowing for batch retrieval and caching of these objects, reducing the need for repeated datastore queries and improving retrieval efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data objects are loaded individually from datastore with dynamic dependency discovery, then system adaptability to evolving graph structures is maintained, but retrieval latency and processing time increase significantly
Solution Approach 1:
The system performs preliminary actions by training a machine learning model on historical data object access patterns before actual processing operations. The model learns and predicts required data objects in advance, enabling batch retrieval that eliminates repeated individual queries while maintaining adaptability to dynamic graph structures through continuous learning from historical patterns.
2Reliability
If repeated queries are made to datastore for dynamic object dependencies, then complete data objects are retrieved for processing, but connection overhead and I/O operations increase
Solution Approach 1:
The system merges multiple individual data object retrieval operations into a single batch retrieval operation. By predicting all required data objects upfront using the machine learning model, the system combines what would have been numerous separate datastore connections and queries into one efficient batch operation, reducing connection overhead while ensuring completeness of data retrieval.
3Productivity
If all required data objects are retrieved in batch operation, then retrieval efficiency improves and latency reduces, but prediction accuracy of required objects must be high
Solution Approach 1:
The system applies partial action by retrieving a predicted subset of data objects in batch operations rather than all possible objects. The machine learning model predicts only the necessary data objects based on historical patterns, achieving high retrieval efficiency without unnecessary data fetching. When prediction accuracy is imperfect, the system can fall back to individual queries for missing objects, ensuring completeness while maintaining overall efficiency.
Data Source
AI summary
In some aspects, the techniques described herein relate to a method including: receiving, as input to a machine learning engine, a processing operation identifier, and an initial data object reference, wherein the processing operation identifier identifies a processing operation; generating, as output from the machine learning engine, a set of data object references, wherein the set of data object references are predicted as required input to the processing operation by the machine learning engine; executing a batch retrieval process, wherein the batch retrieval process retrieves a set of data objects that corresponds to the set of data object references from a datastore; loading the set of data objects in a cache; and executing the processing operation using the predicted set of data objects loaded in the cache.