Data Processing Metadata Mapping for Storage-Compute Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In storage-compute separation architectures, the management layer's fixed metadata and permission models limit the scalability of connecting computing engines due to adaptation difficulties, leading to data inconsistency and redundancy.
Innovation Solution
A data processing system with a management apparatus that maps adaptable metadata and permission models to computing engines, allowing seamless integration and scalability by establishing mapping relationships between these models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a management layer with fixed metadata model and permission model is added to manage data access, then data consistency and centralized control are improved, but adaptability to different computing engines deteriorates
Solution Approach 1:
The patent introduces a metadata model adapter as an intermediary component between the management layer's fixed metadata model and the computing engine's required metadata model. This adapter translates and converts metadata structures, allowing the management layer to maintain a unified fixed model while accommodating various computing engine requirements through the translation layer.
Solution Approach 2:
The patent implements dynamic parameter changes in the metadata model by allowing configuration of mapping relationships between the fixed metadata model and different computing engine metadata models. This enables the system to adapt metadata parameters (such as field names, data types, and structures) according to the specific computing engine being accessed.
2Ease of operation
If metadata is copied to multiple computing engines to enable data sharing, then data accessibility is improved, but data redundancy and inconsistency problems worsen
Solution Approach 1:
The management layer serves as a centralized intermediary that manages metadata for all computing engines. Instead of copying metadata to multiple engines, the system maintains a single source of truth in the management layer and provides metadata to computing engines through controlled access mechanisms, eliminating redundancy and ensuring consistency.
Solution Approach 2:
The management layer provides universal metadata management functionality that serves multiple computing engines simultaneously. The fixed metadata model in the management layer acts as a universal interface that can serve different computing engine types through the adapter mechanism, eliminating the need for separate metadata copies for each engine.
3Extent of automation
If computing engines must adapt to fixed metadata models in the management layer, then centralized control is improved, but scalability of connecting new computing engines deteriorates
Solution Approach 1:
The metadata model adapter acts as a mediator that enables new computing engines to connect to the centralized management layer without requiring changes to the fixed metadata model. The adapter handles the adaptation logic, allowing the system to scale by simply adding new engine adapters rather than modifying the centralized management layer.
Solution Approach 2:
The patent segments the metadata management functionality into separate components: the fixed metadata model in the management layer, the adaptive metadata model in computing engines, and the translator/adapter layer in between. This segmentation allows independent evolution of each component and simplifies adding new computing engines.
Data Source
AI summary
The present disclosure relates to data processing methods, apparatuses, and systems. In one example method, a management apparatus in a data processing system receives an access request that is for metadata of target data stored in a storage apparatus and that is sent by a computing engine, and determines, in response to the access request, the metadata of the target data based on a first mapping relationship between a second metadata model adapted to the computing engine and a first metadata model built in the management apparatus. In addition, the management apparatus authenticates the access request based on a second mapping relationship between a second permission model adapted to the computing engine and a first permission model built in the management apparatus, to send the metadata to the computing engine after the access request passes the authentication.