Blended Data Operations Using Global Identifier Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data merging operations require unique values for foreign keys and process entire database tables, leading to significant memory, bandwidth, and resource expenses, while data blending lacks these constraints and offers capabilities for discovering correlations between data sets.
Innovation Solution
The method involves receiving input at a client interface to select dimensions and measures from multiple data sets, generating data structure models with local and global identifiers, and performing data blending operations on linked dimensions without requiring unique values, allowing for efficient data processing and visualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data merging operations are performed on entire database tables, then complete data integration is achieved, but memory usage and processing resources increase significantly
Solution Approach 1:
The patent extracts only the specific columns and rows needed for the blending operation from the entire database tables, rather than processing complete tables. This is achieved through selective data retrieval based on blend specifications, reducing memory consumption while maintaining data integration completeness for the required fields.
Solution Approach 2:
The patent segments the data processing into distinct phases: first retrieving only necessary columns from source tables, then blending them in memory, and finally writing back only the blended results. This segmentation allows selective processing of data subsets rather than handling entire tables simultaneously.
2Reliability
If data merging operations are performed on entire database tables, then all data is processed, but network bandwidth and data processing resources are consumed excessively
Solution Approach 1:
The system extracts and retrieves only the specific columns required for blending operations from database tables, rather than transferring entire tables over the network. This selective extraction reduces network bandwidth consumption while ensuring all necessary data for the blend operation is obtained.
3Reliability
If unique values are required for foreign keys in data merging, then data integrity is maintained, but the capability to blend data with repeated values is lost
Solution Approach 1:
The patent changes the fundamental parameter requirement for data blending by eliminating the unique value constraint on foreign keys. Instead of requiring unique identifiers for merging, the system allows blending on columns with repeated values, enabling correlation analysis between datasets while maintaining data integrity through proper join operations.
4Reliability
If the entire resulting data set is written back to the database, then data persistence is ensured, but storage space and write operations increase significantly
Solution Approach 1:
The patent extracts and writes back only the blended result columns to the database, rather than storing the entire resulting data set. This selective writing approach ensures that only the necessary blended data is persisted, reducing database storage space requirements while maintaining data persistence for the blended results.
Data Source
AI summary
Data blending operations are disclosed configured to blend data from a primary data set and a secondary data set. Primary and secondary data structure models are generated reflecting selected dimensions and measures of the primary and secondary data structure models upon which the data blending operation is to be performed. Each dimension and measure in the primary and secondary data structure models includes a local identifier. Unique global identifiers are assigned to each local identifier of the primary and secondary data structure models. Client queries are generated and communicated to a backend server to perform data blending on the selected dimensions and measures of the primary data set and the secondary data set. The client queries may comprise the global identifiers and mapping information between the global identifiers and local identifiers. The backend system retrieves a resulting data set in response to the data blending operation based on this mapping.