Data Mesh Query Routing Protocol for Decentralized Data Products
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data meshes, querying data across different data products is challenging due to decentralization, requiring users to identify the correct data product for queries and often necessitating multiple queries when data resides across multiple products, which can be inefficient and requires prior knowledge of data distribution.
Innovation Solution
A data mesh device divides data queries into subqueries based on query routing data, performing these subqueries across different data sources and aggregating responses to provide a unified query result, utilizing query routing data and user role-specific metadata layers to optimize query processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is decentralized into separate data products, then data ownership and responsibility are improved, but query complexity increases
Solution Approach 1:
The patent introduces a data mesh protocol as an intermediary layer that mediates between users and decentralized data products. This protocol automatically routes queries to the appropriate data products, manages data discovery, and coordinates multi-product queries, thereby resolving the complexity issue while preserving data decentralization and ownership benefits
2Loss of information
If users query data across multiple data products, then data completeness is improved, but number of queries increases
Solution Approach 1:
The patent merges multiple separate queries into a single unified query through the data mesh protocol. When a user queries for data that spans multiple data products, the protocol automatically combines these into one coordinated query operation, retrieving data from multiple sources simultaneously and returning aggregated results, thus maintaining data completeness while reducing the number of user-initiated queries
3Measurement precision
If users identify data products manually, then query precision is improved, but ease of operation deteriorates
Solution Approach 1:
The patent implements self-service through automated data discovery and routing mechanisms. The data mesh protocol automatically identifies which data products contain the relevant data based on the query parameters, eliminating the need for users to manually identify data products. The system serves itself by autonomously routing queries to the correct data sources while maintaining precise data retrieval
Data Source
AI summary
In one embodiment, a device of a data mesh divides a data query into a first subquery and a second subquery for different data sources in the data mesh, based on query routing data associated with the device. The device performs the first subquery in the data mesh, to obtain a first subquery response. The device also performs the second subquery in the data mesh, to obtain a second subquery response. The device provides a query response that aggregates the first subquery response and the second subquery response.