Adaptive Data Prefetching Using Locality-Sensitive Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database prefetching techniques are inadequate for dynamic and unpredictable workloads, often leading to inaccurate prefetching, thrashing in limited RAM environments, and lack tuning knobs for improving accuracy at the cost of efficiency.
Innovation Solution
An adaptive prefetching technique that generates data structures based on incoming workloads to determine which pages to prefetch, using locality-sensitive hashing to create signatures of co-occurring page requests, allowing for accurate predictions and tunable accuracy and sensitivity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing prefetching techniques are used, then prefetching coverage is improved, but prefetching accuracy deteriorates
Solution Approach 1:
The patent implements dynamic prefetching by continuously learning from actual query execution patterns and adapting prefetch decisions in real-time. The system maintains a sliding window of recent page access patterns and dynamically updates prefetch predictions based on changing workload characteristics, rather than relying on static prefetching rules
Solution Approach 2:
The patent replaces traditional mechanical prefetching approaches (sequential prefetching based on physical contiguity, semantic prefetching based on predefined data structures) with a machine learning-based predictive model that uses locality-sensitive hashing to identify and prefetch pages likely to be needed based on learned access patterns
2Measurement precision
If existing prefetching techniques are used, then prefetching accuracy is improved, but buffer cache utilization deteriorates
Solution Approach 1:
The patent changes the parameters of prefetching by introducing a configurable prefetch horizon and using locality-sensitive hashing with adjustable sensitivity parameters. This allows the system to tune the balance between prefetching accuracy and buffer cache utilization based on available memory and workload characteristics
3Loss of time
If prefetching is implemented, then I/O latency is reduced, but memory consumption increases
Solution Approach 1:
The patent implements partial prefetching by selectively prefetching only the most promising pages based on learned patterns, rather than aggressively prefetching all possible pages. The system uses a sliding window mechanism to limit the scope of prefetching to recently observed patterns, reducing memory consumption while maintaining I/O performance
4Measurement precision
If adaptive prefetching with learning is implemented, then prefetching accuracy for dynamic workloads is improved, but computational overhead increases
Solution Approach 1:
The patent implements self-service prefetching where the system learns from its own execution patterns and automatically adapts without external intervention. The learning mechanism operates incrementally using online learning algorithms that update predictions based on each query execution, avoiding the need for expensive periodic retraining or complex offline analysis
Data Source
AI summary
A system includes reception of a request to fetch a first dataset, the request including a first identifier associated with the first dataset, determination, in response to the request, of a first signature associated with the first dataset based on the first identifier and on respective identifiers of one or more datasets fetched consecutively with the first dataset, and determination, based on the determined first signature, of whether to update a first link associating a second dataset with the first dataset, the link indicating to prefetch the first dataset in response to a request to fetch the second dataset.