Cloud Agnostic Data Mesh Central Hub Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional tools fail to provide efficient and centralized access to vast amounts of data across organizations, leading to issues like data duplication, aging, and quality problems, which hinder faster data consumption and reduce the quality of customer experiences and increase system risks.
Innovation Solution
A cloud-agnostic data mesh module is implemented, featuring a data mesh architecture with a central hub that connects data producers and consumers, allowing for seamless data access, metadata tagging, and controlled access management to prevent data duplication and improve data utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional tools are used for data access and management, then data can be stored in distributed locations, but data duplication, aging, and quality problems occur, leading to inefficient data consumption
Solution Approach 1:
A centralized data mesh module is introduced as an intermediary between distributed data sources and data consumers. This module provides unified data access, eliminates data duplication by serving as a single source of truth, and ensures data quality through centralized management and validation mechanisms.
Solution Approach 2:
The data mesh module serves multiple functions: it acts as a centralized data repository, provides data access control, manages metadata, and coordinates data requests across the organization. This multi-functional approach consolidates previously scattered data management tasks into a single universal system.
2Ease of operation
If data is stored in distributed locations across the organization, then data can be maintained at source, but centralized seamless access becomes difficult and time-consuming
Solution Approach 1:
The centralized data mesh module serves as an intermediary that abstracts the complexity of distributed data storage from end users. Data consumers interact with a single unified interface, and the module automatically routes requests to the appropriate data sources, eliminating the need for users to manually navigate distributed systems.
Solution Approach 2:
The system segments data management responsibilities: the centralized mesh module handles access coordination, metadata management, and request routing, while distributed data sources maintain their data. This segmentation allows centralized access control without requiring physical centralization of data.
3Quantity of substance
If conventional data management approaches are used, then data can be stored locally, but data duplication and processing complexities increase
Solution Approach 1:
The data mesh module acts as an intermediary that manages data requests centrally, preventing duplicate data storage by serving as a single source of truth. It coordinates access to distributed data sources, eliminating the need for multiple copies of the same data and reducing processing complexity through centralized request management.
4Adaptability or versatility
If data access is allowed without centralized control, then data consumers can access data freely, but access security and controlled permissions become problematic
Solution Approach 1:
The centralized data mesh module serves as a security intermediary that mediates all data access requests. It implements permission checks, validates user credentials, and controls data flow between consumers and sources. This maintains flexible access for authorized users while ensuring security through centralized authentication and authorization mechanisms.
Data Source
AI summary
Various methods, apparatuses/systems, and media for providing centralized seamless data access are disclosed. A processor builds a data mesh architecture in a cloud environment. The data mesh architecture includes a single central hub account provided between a plurality of data producer accounts and a plurality of data consumer accounts. The processor causes the plurality of data producer accounts to publish different types of data received from a plurality of data sources onto the single central hub account along with corresponding metadata. The processor also causes the single central hub account to: incorporate the metadata into an application; receive a request from the plurality of data consumer accounts to access the published different types of data from the single central hub account; and control who from the plurality of data consumer accounts can access the published different types of data from the single central hub account based on the metadata.