Request Proxy for Real-Time Querying of Disparate Data Sources
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Disparate data sources in non-relational formats, such as XML, JSON, and unstructured data sources like social media and sensors, are difficult to query uniformly in real-time due to their varied structures and protocols, requiring inefficient intermediate storage and specialized skills, making it cumbersome for data analysts to consume and integrate them.
Innovation Solution
A request proxy system that registers data adapters and uses listeners to route requests across multiple data protocols, emulating virtual databases to allow real-time querying of disparate data sources without an intermediate RDBMS, using agnostic request objects and centralized configuration for secure, multi-protocol communication and caching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If an intermediate RDBMS is used to host disparate data sources, then data can be stored and queried using SQL, but real-time querying is not possible and intermediate storage is required
Solution Approach 1:
The patent introduces a virtual database management system as an intermediary layer between the user and disparate data sources. This virtual DBMS translates SQL queries into protocol-specific queries for different data sources (XML, JSON, REST APIs, sensors) without requiring physical intermediate storage. The mediator converts uniform SQL syntax into source-specific query formats, enabling real-time access while maintaining data abstraction.
Solution Approach 2:
The virtual database system creates a virtual copy of the database interface that mimics traditional RDBMS behavior. Instead of physically copying and storing data in an intermediate database, the system creates a virtual representation of data sources that responds to SQL queries in real-time, eliminating the need for materialized views or cached copies while maintaining the familiar database interface.
2Ease of manufacture
If batch or near-time integration tools are used to import data into the database, then data can be consolidated, but the process is inefficient and cannot handle real-time sources
Solution Approach 1:
The system replaces discrete batch import operations with continuous real-time data access. Instead of periodically extracting, transforming, and loading data in batches, the virtual database management system continuously listens for and processes queries against all connected data sources in real-time, maintaining an ongoing state of data availability without interruption or delay.
Solution Approach 2:
The patent replaces the mechanical batch processing system (ETL tools, import utilities, scheduled jobs) with a software-based virtualization layer that handles data access through intelligent query translation. This substitution eliminates the need for physical data movement and manual integration workflows, replacing them with automated, real-time query routing and response generation.
3Adaptability or versatility
If disparate data sources in non-relational formats are queried directly, then specialized skill sets and hardware are required for each source, but uniform querying is not possible
Solution Approach 1:
The virtual database management system provides a universal interface that handles multiple data source types (XML files, JSON APIs, REST services, sensor data, relational databases) through a single SQL query language. This multi-functional system translates the same SQL syntax into appropriate query formats for each data source type, allowing users to query any supported source uniformly without learning source-specific protocols or syntax.
Solution Approach 2:
The system dynamically changes query parameters and syntax based on the target data source type. When a SQL query is received, the virtual DBMS analyzes the query and transforms it into the appropriate format for the specific data source (e.g., converting SQL JOIN operations into nested JSON queries or XML path expressions), automatically adapting to the target format without requiring user intervention or source-specific programming.
Data Source
AI summary
A system and method for querying disparate data sources in real time utilizes a plurality of listeners and at least one data adapter operated on a request proxy. A interface specific data query is received from a client account by the request proxy through a specific listener. The interface specific data query is formatted into an agnostic request object that is then forwarded to a specific adapter, wherein a specific method is determined from a plurality of methods for the specific adapter. The request proxy retrieves requested data from at least one data source through the specific adapter. The requested data is then formatted into an agnostic response object that can be cached on the request proxy. The agnostic response object is then formatted into a interface specific response stream by the request proxy and the interface specific response stream is sent to the client account through the specific listener.