Graph-Based Dataset Partitioning for Data Warehouse Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large volumes of data in online businesses is challenging due to inefficiencies in storage and processing, as existing systems struggle to optimize storage configuration and resource utilization across multiple data storage systems.
Innovation Solution
A graph-based analysis method is employed to determine the optimal configuration of datasets across data storage systems, using a greedy breadth-first traversal algorithm to partition datasets based on storage and processing capacity, ensuring efficient use of resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If more data is stored in data warehouses, then data volume increases, but storage efficiency and processing efficiency deteriorate
Solution Approach 1:
The patent segments datasets into distinct categories (e.g., transactional data, analytical data, master data) and assigns them to different storage systems based on their specific characteristics and access patterns. This segmentation allows each storage system to be optimized for its intended purpose, maintaining high efficiency even as overall data volume increases.
Solution Approach 2:
Different storage systems are assigned different qualities and characteristics tailored to specific data types and access requirements. For example, high-speed storage is allocated for frequently accessed transactional data, while cost-effective storage is used for archival data. This local optimization ensures that efficiency is maintained despite increasing data volumes.
2Power
If more hardware and software resources are deployed, then data processing capability increases, but resource utilization efficiency deteriorates
Solution Approach 1:
The patent implements dynamic resource allocation where storage and processing resources are dynamically assigned based on real-time data access patterns and workload requirements. This dynamic adjustment ensures that resources are utilized efficiently without over-provisioning, maintaining high resource utilization efficiency even as processing capability requirements increase.
Solution Approach 2:
The system employs self-service mechanisms where data is automatically categorized, prioritized, and assigned to appropriate storage systems based on predefined policies and characteristics. This automated approach eliminates manual resource allocation inefficiencies and optimizes resource utilization without requiring additional human intervention or complex management overhead.
3Reliability
If datasets are stored on multiple data storage systems, then data availability and redundancy improve, but system complexity increases
Solution Approach 1:
The patent performs preliminary actions by pre-categorizing datasets according to their characteristics, access patterns, and redundancy requirements before deployment. This pre-planning enables automated, rule-based distribution across storage systems, reducing operational complexity while ensuring reliable data availability and redundancy through consistent, predictable placement strategies.
Data Source
AI summary
Techniques are described for employing a graph-based analysis to determine a configuration of datasets to be stored on data storage systems in a data warehouse environment. Associations between datasets may be determined based on the parsing of join statements or other types of statements in jobs that are executed on the data storage systems. A graph may be generated that describes the associations among datasets. A greedy breadth-first traversal of the graph may be performed to determine sets of associated datasets. A utilization metric describing a weight of storing the datasets may be determined and employed to identify a data storage system on which to store a set of associated datasets, given the storage and processing capacity of the data storage system.