Relocating Data Sharing for Distributed Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge lies in managing and processing distributed data across various locations and formats, which complicates analysis and processing due to the difficulty in selecting suitable data management and storage solutions that ensure current needs are met without obstructing future accessibility.
Innovation Solution
Implementing a format-independent data processing service that relocates data sharing operations for query processing, allowing data to be processed across different data processing engines, reducing processing burdens by distributing operations and results directly to appropriate nodes, thereby minimizing the need for intermediate data sharing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored or distributed across different locations in different formats, then data accessibility and storage flexibility are improved, but data processing complexity and difficulty increase
Solution Approach 1:
The patent introduces a data sharing operation relocation mechanism that acts as an intermediary between distributed data sources and processing nodes. The query planning engine identifies and relocates data sharing operations to appropriate data processing nodes, enabling seamless integration of distributed data in different formats without increasing processing complexity. This mediator approach allows the system to maintain data accessibility across diverse locations and formats while managing processing complexity centrally through automated operation relocation.
2Ease of operation
If data sharing operations are performed at intermediate nodes, then data can be distributed to multiple nodes, but processing burden and costs on intermediate nodes increase
Solution Approach 1:
The patent inverts the traditional approach by having data processing nodes proactively request and receive data sharing operations from remote data processing nodes directly, rather than having intermediate nodes distribute data. This inversion eliminates the need for intermediate nodes to perform data sharing operations, thereby reducing their processing burden while maintaining the ability to distribute data to multiple nodes as needed.
3Device complexity
If centralized data processing is used, then data management is simplified, but processing time and resource consumption increase
Solution Approach 1:
The patent segments data processing operations by relocating specific data sharing operations to appropriate data processing nodes based on query requirements. The query planning engine divides the overall query processing into segments, with some operations executed centrally and others distributed to remote nodes. This segmentation enables simplified centralized management of query planning while improving processing efficiency through distributed execution of specific operations.
Data Source
AI summary
Relocating data sharing operations for query processing may be implemented when generating plans to perform a query. A query operation that causes a node to share data obtained from a different set of nodes with other nodes in a same set of nodes may be identified. The identified query operation may be replaced with another operation that causes the different set of nodes to share the data directly with the other nodes in the same set of nodes.