Layered Data Manager for Scalable Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems fail to provide central administration while allowing for local modifications and merging of data across multiple levels, especially in distributed environments, and lack scalability to handle both small and large data volumes.
Innovation Solution
A layered data management system with a layered data manager (LDM) at each level, where each LDM has a local data store and can request data from a superordinate LDM, merging data transiently without persistence, allowing for local changes to be maintained independently without propagating to superordinate stores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is centrally managed in a client/server network, then data consistency and centralized control are improved, but the ability to allow local customization and modifications at user level deteriorates
Solution Approach 1:
The system segments data management into multiple hierarchical layers (enterprise level, business unit level, local level) with a Layered Data Manager at each level. Each layer can independently manage and customize data within its scope while maintaining consistency with upper layers through the layered architecture.
Solution Approach 2:
The Layered Data Manager merges data from multiple sources including centrally managed data from superordinate layers and locally customized data from subordinate layers. The merge function combines these data sets while maintaining data consistency and allowing local customizations to coexist with central data management.
2Adaptability or versatility
If data management is distributed along an organizational hierarchy, then local adaptability and user customization are improved, but management complexity deteriorates
Solution Approach 1:
The Layered Data Manager is designed as a universal component that can operate at any hierarchical level in the organization. Each LDM performs the same core functions (data storage, data merging, data management) regardless of its position in the hierarchy, which simplifies implementation and reduces management complexity despite the distributed nature of the system.
Solution Approach 2:
The system implements a nested hierarchical structure where Layered Data Managers at different levels are organized in a tree-like hierarchy. Each LDM contains or manages data relevant to its level while being nested within the broader organizational hierarchy, allowing local adaptability while maintaining overall system coherence through the nested structure.
3Adaptability or versatility
If data from different storage technologies is integrated, then platform independence and data accessibility are improved, but integration complexity deteriorates
Solution Approach 1:
The Layered Data Manager acts as an intermediary layer between different storage technologies and data sources. It provides a unified interface and abstraction layer that handles the complexity of integrating diverse storage technologies, allowing platform independence without requiring direct integration between different storage systems.
4Productivity
If the data management system is made highly scalable, then ability to handle both small and massive data volumes is improved, but system complexity deteriorates
Solution Approach 1:
The layered data management system is dynamically scalable, allowing the addition or removal of Layered Data Managers at different hierarchical levels based on organizational needs. The system can adapt its structure and capacity dynamically to handle varying data volumes from small to massive scales without requiring a complete system redesign.
Data Source
AI summary
A distributed system for administration and storage of data includes a data manager to access a local data store. Several data managers can be chained to form a layered hierarchical structure. All data layers are merged top down to form the resultant data records. Fields can be modified or added in each layer. The system is self organizing in that each data layer only knows about its superordinate data layer. The system provides scalable persistent data storage with centralized data management and the option to locally extend and modify data.