In-Memory Query Engine for Hybrid Cloud Data Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing enterprise data distributed across multiple cloud platforms is challenging due to security concerns and slow application migration, leading to over- or under-retrieval of data.
Innovation Solution
Implementing an in-memory query engine that parses and converts data queries into relational trees, using tree partitioning and pruning techniques to optimize data retrieval from multiple data stores while enforcing data restrictions and adhering to a budget, and supporting schema discovery to manage data across hybrid cloud platforms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is distributed across multiple cloud platforms for security and migration flexibility, then security and adaptability are improved, but data management complexity and retrieval efficiency deteriorate
Solution Approach 1:
The patent introduces a query engine as an intermediary layer between data consumers and distributed cloud data stores. This query engine acts as a mediator that translates complex distributed data access requirements into simplified queries, automatically managing the complexity of retrieving data from multiple cloud platforms without requiring users to directly handle the underlying complexity.
Solution Approach 2:
The system segments data access operations into independent units by creating separate retrieval services for different data stores. Each data store can be accessed independently through its own retrieval service, allowing the system to manage distributed data by breaking down the complex retrieval process into manageable segments that can be executed in parallel.
2Ease of manufacture
If traditional data retrieval methods are used across multiple cloud platforms, then implementation simplicity is maintained, but data retrieval efficiency and cost control deteriorate
Solution Approach 1:
The query engine dynamically selects and executes optimal retrieval plans based on real-time conditions. It can adapt the retrieval strategy by choosing different data stores, adjusting query parameters, and selecting appropriate retrieval services based on data location, accessibility, and current system state, thereby optimizing retrieval efficiency without requiring complex manual configuration.
Solution Approach 2:
The system performs preliminary actions by pre-resolving query plans and pre-establishing retrieval strategies before actual data retrieval occurs. The query engine analyzes the full query, resolves dependencies, and creates an optimized execution plan in advance, which then guides the actual data retrieval process to improve efficiency and enable better cost control.
3Loss of information
If comprehensive data retrieval is performed to ensure completeness, then data completeness is improved, but compute resource consumption and cost deteriorate
Solution Approach 1:
The query engine applies partial action by selectively retrieving only the necessary data portions required to satisfy the query, rather than retrieving all available data. It can identify and retrieve only specific records, fields, or data subsets that match the query criteria, thereby avoiding unnecessary compute resource consumption while maintaining data completeness for the actual retrieval needs.
Solution Approach 2:
The system incorporates feedback mechanisms that allow the query engine to monitor and adjust retrieval operations based on actual data characteristics and query requirements. By receiving feedback about data distribution, accessibility, and retrieval progress, the engine can dynamically adjust its strategies to optimize the balance between data completeness and resource consumption.
Data Source
AI summary
System and methods for processing data queries in hybrid data mesh architectures are disclosed. A method for processing data queries in hybrid data mesh architectures may include an in-memory query engine: (1) receiving, from a requestor, a data query to retrieve data; (2) parsing the data query into a parse tree; (3) converting the parse tree into a relational tree, wherein the relational tree is a tree representation of relational operators used to execute the data query; and (4) executing a data retrieval method selected from the group consisting of tree partitioning by cost and pruning using subtree replacement using a plurality of retrieval services to retrieve data from data stores identified by the data retrieval method.