Heterogeneous Data Query Mediator Without Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current big data analysis systems like Apache Solr and Elastic search are limited in handling operations involving multiple data sets from different data sources, requiring resource-intensive processes such as data replication into a single database for combined analysis.
Innovation Solution
A method and device that acquire and process data elements from different data sources, including Lucene-based search engines, RDBs, and files, using JOIN and UNION operations without data replication, by converting data types and performing joint operations based on user queries, enabling output operations such as data processing and display.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data replication into a single database is performed to enable operations on data sets from different data sources, then the ability to perform joint operations on diverse data sources is improved, but resource consumption and processing time increase significantly
Solution Approach 1:
The patent introduces an intermediary system that translates and coordinates queries across multiple heterogeneous data sources without requiring data replication. This mediator enables joint operations by maintaining data in its original locations while providing unified access through translation layers, thus resolving the contradiction between operational capability and resource consumption.
Solution Approach 2:
The system implements a universal interface layer that can handle multiple types of data sources (relational databases, NoSQL databases, file systems, cloud storage) through a common protocol. This multi-functional approach allows the system to perform operations across diverse data sources without replicating data, achieving versatility while avoiding the resource overhead of replication.
2Adaptability or versatility
If data replication into a single database is performed to enable operations on data sets from different data sources, then the ability to perform joint operations on diverse data sources is improved, but processing time increases significantly
Solution Approach 1:
The system performs preliminary actions by pre-translating and pre-processing queries before execution, and by caching translation rules and data access patterns. This allows the system to quickly adapt to different data sources without the time penalty of real-time translation or data movement, resolving the contradiction between operational capability and processing time.
Solution Approach 2:
The intermediary layer pre-establishes connection protocols and translation rules for various data sources, enabling rapid execution of joint operations without the time-consuming process of data replication. The mediator optimizes query routing and data retrieval paths, significantly reducing processing time compared to replication-based approaches.
3Adaptability or versatility
If dedicated applications are implemented to perform operations on data sets from different data sources, then the functionality for handling diverse data sources is improved, but device complexity and implementation cost increase
Solution Approach 1:
The patent implements a universal data access framework that can handle multiple data source types through a single integrated system rather than separate dedicated applications. This framework provides standardized interfaces and translation mechanisms that work across relational databases, NoSQL databases, file systems, and cloud storage, significantly reducing implementation complexity while maintaining high functionality.
Solution Approach 2:
The system employs dynamic configuration and adaptive translation capabilities that allow it to automatically adjust to different data source types and operations requirements. This dynamic approach eliminates the need for static, dedicated applications for each data source combination, reducing overall system complexity while preserving full functionality.
Data Source
AI summary
A method for outputting a result of one or more operations using data sources of different types is provided. The method includes steps of: (a) when a user query is acquired, a device (i) acquiring data elements respectively from the data sources of different types by referring to the user query, and (ii) performing main joint operations on the data elements, to thereby generate data set; and (b) the device performing data processing operations and output operations on the data set, to thereby generate an answer for the user query. It has an effect of providing the method for outputting the result of the operations using the data sources of the different types by referring to each of languages corresponding to each of the data sources.