Analytics Query Subgraph Reuse for Overlapping Computations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In shared analytics clusters, significant overlaps in computations performed by submitted jobs lead to wasteful resource usage, increasing operational costs, which is challenging to detect and address due to distributed user bases and complex, recurring data pipelines.
Innovation Solution
A system and method for automatically detecting and reusing overlapping computations in analytics job services by using normalized and precise signatures to identify and materialize subgraphs, with a feedback loop to reconcile compile-time and run-time statistics for efficient computation reuse.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored in multiple different data lakes for different analytics jobs, then data accessibility and job execution are improved, but data duplication and storage waste increase
Solution Approach 1:
The patent implements a universal data lakehouse that serves multiple analytics jobs simultaneously. Instead of creating separate data lakes for different jobs, the system provides a single multi-tenant platform where multiple jobs can access and analyze data together, eliminating duplication while maintaining accessibility for all jobs
Solution Approach 2:
The patent merges multiple separate data lakes into a single unified data lakehouse infrastructure. By combining storage, processing, and governance capabilities into one shared platform, the system eliminates redundant data storage across multiple lakes while enabling all analytics jobs to access the consolidated data repository
2Adaptability or versatility
If data is copied and transformed for each analytics job, then job-specific analysis requirements are met, but processing time and computational resources increase
Solution Approach 1:
The patent performs data transformation and preparation activities in advance, before analytics jobs execute. By pre-processing data into standardized formats and making it readily available in the data lakehouse, the system eliminates the need for time-consuming data copying and transformation during actual job execution
Solution Approach 2:
The patent uses virtualization and referencing mechanisms instead of physical data copying. Analytics jobs can access and analyze data through virtual copies or references to the master data in the lakehouse, allowing job-specific analysis without duplicating the actual data storage or requiring lengthy copy operations
3Ease of manufacture
If separate data lakes are used for different data types, then data organization and management are improved, but system complexity and operational overhead increase
Solution Approach 1:
The patent implements a universal data lakehouse platform that handles multiple data types (structured, semi-structured, unstructured) within a single unified system. Instead of creating separate specialized data lakes for different data types, the lakehouse provides unified storage, processing, and governance capabilities that work across all data formats
Solution Approach 2:
The patent organizes data within the unified lakehouse using logical segmentation and namespace structures. Data can be divided into different logical sections or catalogs for different data types and jobs, while physically remaining in a single consolidated storage system, thus maintaining organization without creating separate physical systems
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
Described herein is a system and method for detecting and reusing overlapping computations. Overlapping subgraphs of the query are determined using a normalized signature for a particular subgraph that identifies a particular subgraph across recurring instances of data. A normalized signature for each overlapping subgraph for the determined overlapping subgraphs of the query is provided. For each overlapping subgraph determined to be materialized: whether or not the particular subgraph has been materialized is determined using a precise signature corresponding to a normalized signature of the particular overlapping subgraph. The precise signature identifies a particular subgraph corresponding to the normalized signature within a particular recurring instance of data. When the particular subgraph has not been materialized, the subgraph is materialized and used to respond to the query. When the particular subgraph has been materialized, the materialized subgraph is used to respond to the query.