Transparent Data Cache for Interactive Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional SQL-on-Big Data approaches face challenges in providing interactive analytics due to high latency and the need for manual creation of reflections for pre-extracting data, which limits ad-hoc analytics and introduces performance degradation in cloud environments.
Innovation Solution
A data system that automatically caches frequently accessed data from external data sources to a local cluster of nodes, using a unit of hashing to map data objects to nodes and a unit of caching to store only frequently accessed sections, thereby reducing latency and enhancing interactive performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If SQL-on-Big Data tools are used to process data in the same cluster where data is stored, then data processing capability is improved, but latency increases and interactive analytics performance deteriorates
Solution Approach 1:
The system pre-loads frequently accessed data from external data sources into a local cache before it is needed for query processing. This preliminary action ensures that hot data is already available in memory when queries arrive, eliminating the latency associated with reading from external sources during query execution.
Solution Approach 2:
The patent introduces a cache layer as an intermediary component between the query processing engine and external data sources. This cache acts as a mediator that holds copies of frequently accessed data, allowing queries to be satisfied from the fast local cache rather than requiring direct access to slow external data sources.
2Speed
If data is pre-extracted and stored as OLAP cubes outside the data lake, then interactive speed is improved, but ad-hoc analytics capability is limited
Solution Approach 1:
The cache is designed to be dynamic rather than static, automatically adapting its contents based on query patterns and data access frequencies. The system continuously monitors which data is being accessed and adjusts the cache contents accordingly, allowing it to support both pre-defined analytics and ad-hoc queries on the full dataset without requiring manual re-extraction.
Solution Approach 2:
The cache serves multiple functions: it acts as a performance accelerator for frequently accessed data, a flexible storage layer for ad-hoc analytics, and an automatic data extraction mechanism. Unlike traditional OLAP cubes that are limited to pre-defined structures, this universal cache can hold any data from the external sources and adapt to different query types and analytics needs.
3Speed
If manual data pre-extraction is performed to create reflections, then query performance is improved, but system complexity and operational overhead increase
Solution Approach 1:
The system performs self-service by automatically identifying which data should be cached and extracting it without requiring manual intervention. The cache management system monitors query patterns, determines which external data sources and tables are frequently accessed, and automatically pre-loads that data into the cache, eliminating the need for manual reflection creation and data extraction processes.
Solution Approach 2:
The system implements feedback mechanisms that monitor query workloads and cache performance, using this information to dynamically adjust what data is cached. This feedback loop allows the system to automatically optimize itself based on actual usage patterns, reducing the complexity of manual configuration while maintaining high query performance.
4Reliability
If external data sources are accessed directly for each query, then data freshness is maintained, but query response time increases
Solution Approach 1:
The system performs preliminary data retrieval by caching copies of external data sources before they are needed for query processing. This advance preparation ensures that data is available immediately when queries are executed, maintaining data freshness while eliminating the time penalty of direct external source access during query time.
Solution Approach 2:
The patent creates and maintains copies of external data source content in the local cache. These copies allow queries to be executed against the cached data rather than requiring direct access to external sources, significantly reducing query response time while maintaining data accuracy through periodic synchronization with the external sources.
Data Source
AI summary
The disclosed embodiments include a method for caching by a data system. The method includes automatically caching a portion of a data object from an external data source to a local cluster of nodes in accordance with a unit of caching. The portion of the data object can be selected for caching based on a frequency of accessing the portion of the data object. The portion of the data object in the cache is mapped to the external data source in accordance with a unit of hashing. The method further includes, responsive to the data system receiving a query for data stored in the external data source, obtaining query results that satisfy the received query by reading the portion of the cached data object instead of reading the data object from the external data source.