Multi-Node Object Storage for Parallel Data Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Big data analysis is hindered by the need for multiple file storage apparatuses due to the upper limit of file system size, requiring object storage systems with multiple nodes, which leads to inefficient data loading times, especially when data needs to be updated or rearranged across nodes.
Innovation Solution
A storage system with multiple nodes that provides tenant spaces for data units, allowing for multiplexed data storage and parallel loading to Hadoop systems, reducing data loading time by distributing data across multiple nodes and utilizing a common space for efficient data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in HDFS for big data analysis, then data storage capacity is improved, but data loading time increases due to sequential copying from bridge storage system
Solution Approach 1:
The patent divides the storage system into multiple independent nodes (N nodes) that can operate in parallel. Each node stores portions of the data set and can independently serve data loading requests, transforming the sequential copying process into parallel operations that reduce overall data loading time while maintaining total storage capacity.
2Ease of operation
If data is migrated between storage spaces for rearrangement, then data distribution is improved, but data consistency is impaired due to update conflicts
Solution Approach 1:
The patent performs data rearrangement and distribution before the analysis process begins. By pre-distributing data across multiple nodes in an optimized configuration, the system avoids the need for data migration during active analysis, thereby maintaining data consistency while achieving improved data distribution and parallel access capabilities.
3Quantity of substance
If multiple file storage apparatuses are used to exceed file system size limit, then storage capacity is improved, but system complexity increases
Solution Approach 1:
The patent merges multiple storage nodes into a unified object storage system that presents a single logical interface to the Hadoop system. This integration allows the system to exceed file system size limits by combining the capacity of multiple nodes while maintaining simplified management and access through a unified namespace and interface layer.
4Reliability
If data is copied instead of migrated for rearrangement, then data consistency is maintained, but storage efficiency decreases due to duplicate data
Solution Approach 1:
The patent creates multiple copies of the data set across different nodes in the storage system. Each node stores an identical copy of the data, enabling parallel read operations from multiple sources simultaneously. This approach maintains data consistency across all copies while improving storage efficiency by utilizing distributed parallel access rather than requiring single-point sequential access.
Data Source
AI summary
A storage system includes N nodes (N is an integer, equal to or larger than 2) coupled to at least one data processing system. A plurality of tenant spaces, based on the N nodes, contain Y1 tenant spaces (Y1 is an integer, equal to or larger than 2 and equal to or smaller than N) for a first tenant. The Y1 tenant spaces are based on Y1 node groups in the N nodes. Each of the Y1 node groups contains Q1 or more nodes (Q1 is an integer, equal to or larger than 1). For the first tenant, the storage system provides, to each of the Y1 tenant spaces, data units multiplexed Y1 times. From a first load-source tenant space from among the Y1 tenant spaces, the storage system loads a first data unit group containing one or more data units to a first load-destination data processing system.