Remote Data Blending via Subquery Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data merging techniques face challenges in efficiently blending data from multiple data sets residing on different systems, leading to resource-intensive processes and programming complexities due to the need for all data sets to be on the same system.
Innovation Solution
The method involves generating subqueries from a main data model blend query, sending them to remote systems, accessing results, and performing the blend at a primary system, allowing data blending across separate systems by utilizing a data provisioning agent for communication and result retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data blending is performed on a single system having all data sets, then data blending can be performed, but device complexity and programming difficulty increase
Solution Approach 1:
The patent segments the data blending process into multiple components: a primary system that coordinates the blending operation and remote systems that provide data sets. Each system handles a portion of the data blending task, allowing the operation to span multiple systems without requiring all data to reside on a single system. This segmentation reduces the complexity burden on any single system while enabling distributed data blending.
Solution Approach 2:
The patent introduces an intermediary mechanism (the primary system acting as coordinator) that manages communication between remote systems and facilitates data blending. This intermediary handles the complexity of coordinating multiple data sources, allowing remote systems to provide data without requiring complex programming at each endpoint. The intermediary abstracts the distributed nature of the system, simplifying the operation for users.
2Reliability
If all data sets are merged in a data merge process, then complete data integration is achieved, but resource consumption increases
Solution Approach 1:
The patent extracts only the necessary data from remote data sets rather than merging all data. The primary system identifies specific data requirements for the blending operation and retrieves only those portions from remote systems. This extraction approach maintains data integration completeness for the intended purpose while significantly reducing resource consumption compared to merging entire data sets.
Solution Approach 2:
The patent applies partial action by performing data blending on a subset of data from multiple sources rather than complete data merging. The primary system determines the minimum necessary data portions required for the blending operation and processes only those. This partial approach achieves sufficient data integration for analytical purposes while avoiding the excessive resource consumption of complete data merging.
3Loss of information
If data sets from multiple sources are blended, then useful correlations are discovered, but memory space and network bandwidth requirements increase
Solution Approach 1:
The patent extracts and transmits only the specific data portions required for blending from remote systems to the primary system. Rather than moving entire data sets, the system identifies and retrieves only the necessary columns and rows needed for the blending operation. This extraction minimizes network bandwidth consumption and memory space requirements while preserving the useful correlations that can be discovered through blending.
Data Source
AI summary
The present disclosure provides improved systems, methods, and computer readable media for blending data from data sets that reside on different systems. This is done by generating subqueries from a main data model blend query, sending the subqueries to respective separate systems, accessing the results to all subqueries at a primary system, and performing a blend based on the main data model blend query at the primary system.