Split Index for External Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face suboptimal performance when interacting with external data storage systems, particularly for selective queries, due to the need for scanning large data quantities in conventional approaches of data import or pushdown query computation.
Innovation Solution
A split index is created locally in a primary data storage system for external data, allowing for incremental updates and enabling hybrid processing, where the index can be stale yet still used for query execution, with fallback mechanisms for new or unindexed data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional data import or pushdown query computation is used to access external data, then data can be retrieved from external storage systems, but query execution performance is suboptimal due to scanning large data quantities
Solution Approach 1:
The patent creates indexes on external data storage systems in advance, before queries are executed. These pre-built indexes enable the system to quickly locate and retrieve specific data without scanning entire data sets during query execution, thus improving query performance and reducing execution time.
Solution Approach 2:
The patent introduces an index structure as an intermediary between the query system and external data storage. This index acts as a mediator that guides query execution by providing direct paths to relevant data, eliminating the need for full data scans and significantly improving access efficiency.
2Productivity
If an index is created locally for external data to accelerate queries, then query execution is faster, but the cost of frequent index updates increases
Solution Approach 1:
The patent implements periodic or incremental index updates rather than continuous updates. The index is updated at scheduled intervals or only when necessary, allowing the system to maintain query performance while reducing the frequency and cost of update operations. This periodic approach balances the trade-off between having current indexes and minimizing update overhead.
Solution Approach 2:
The patent allows the index to be stale (out of date) with respect to external data, changing the parameter of index freshness. By accepting that the index may not reflect the most recent data state, the system reduces update frequency and cost while still providing adequate query performance for many workloads where exact real-time consistency is not critical.
3Reliability
If the index is kept up to date with every data change, then query accuracy is improved, but the overhead of continuous updates increases
Solution Approach 1:
The patent applies partial updating to the index, where only certain portions or aspects of the index are updated based on data changes, rather than performing complete re-indexing. This partial action approach maintains sufficient query accuracy for most operations while reducing the complexity and overhead associated with continuous full updates.
Solution Approach 2:
The patent changes the parameter of index freshness by allowing the index to be stale relative to external data. This parameter change accepts a trade-off where the index may not reflect the absolute latest data state, thereby reducing update complexity and overhead while maintaining adequate reliability for query operations that do not require real-time consistency.
Data Source
AI summary
A split-index can be employed for access to external data. The index can be created on a primary data storage system for data stored externally on a secondary data storage system. After creation, the index can be utilized to expedite at least query execution over the externally stored data. The index can be updated upon detection of changes to data. Further, even when the index is not completely up to date, the index can be exploited for query execution. Furthermore, hybrid execution is enabled with the index and without the index.