Nested Subarray Data Lake Storage for Parallel Read Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data lake systems face significant performance issues with read speed, particularly as file sizes increase, due to the conversion of file streams into single-field type arrays, which limits parallel processing and storage capacity.
Innovation Solution
A data storage method that converts file streams into nested subarrays matching the number of CPU cores, segments large arrays into multiple rows, and employs a polling mechanism for automatic storage, enhancing read performance and storage efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If file streams are converted into single-field type arrays in existing data lake systems, then the storage structure is simple, but the read performance deteriorates significantly as file size increases
Solution Approach 1:
The patent segments the file stream conversion process by dividing data into multiple subarrays corresponding to different field types, rather than using a single-field type array. This segmentation enables parallel processing during read operations, significantly improving read performance while maintaining manageable storage structure complexity through organized data categorization.
Solution Approach 2:
The patent transitions from a one-dimensional single-field type array structure to a multi-dimensional nested array structure where data is organized by field types across multiple dimensions. This dimensional change allows for parallel access patterns and improves read performance by enabling simultaneous processing of different field types without increasing overall structural complexity.
2Ease of operation
If all fields correspond to one field type in the array conversion process, then the field structure is uniform and simple, but the parallel processing capability is limited
Solution Approach 1:
The patent applies local quality by assigning different field types to different subarrays within the overall array structure. Each subarray is optimized for its specific field type, allowing parallel processing of heterogeneous data while maintaining uniformity within each subarray. This resolves the contradiction by achieving both local uniformity and global parallel processing capability.
Solution Approach 2:
The patent creates a composite array structure that combines multiple subarrays of different field types into a unified data structure. This composite structure maintains the simplicity of uniform field handling within each subarray while enabling parallel processing across different field types, effectively resolving the contradiction between structural simplicity and processing capability.
3Quantity of substance
If the array size exceeds memory capacity, then more data can be stored, but memory errors occur and processing fails
Solution Approach 1:
The patent segments large arrays into smaller subarrays organized by field types, allowing each subarray to be processed and stored within memory capacity limits. This segmentation prevents memory overflow errors while maintaining the ability to store and process large quantities of data through distributed subarray management and iterative processing.
Solution Approach 2:
The patent implements partial processing by handling data in manageable chunks across multiple iterations rather than loading entire large arrays into memory at once. This approach allows storage of data exceeding memory capacity by processing portions sequentially, preventing memory errors while achieving effective storage of large data volumes.
Data Source
AI summary
The present disclosure provides a high-performance data lake system and a data storage method. The data storage method includes the following steps: S1: converting a file into a file stream; S2: converting the file stream into an array in which multiple subarrays are nested; and S3: converting the array into a resilient distributed dataset (RDD), and storing the RDD to a storage layer of a data lake. The present disclosure provides a nested field structure, which lays the foundation for parallel processing in reading, and effectively improves read performance. Furthermore, the present disclosure flexibly generates a number of nested subarrays according to hardware cores, such that the data lake achieves better extension performance, and can keep optimal writing efficiency for different users.