Multi-Tier Storage for IoT Small File Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face performance bottlenecks and scalability issues when processing large numbers of small files from IoT devices, leading to degraded performance and processing inefficiencies.
Innovation Solution
A multi-tier storage system is implemented with a front-end storage tier comprising a file system and a key-value store, where small files are combined into objects and stored in a back-end storage tier, with data mover modules performing tar and compression operations to optimize data management and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple small files are processed individually in a typical multi-tier storage system, then file-by-file processing can be performed, but processing bottlenecks occur leading to degraded performance
Solution Approach 1:
The patent combines multiple small files into a single archive file (e.g., TAR format) before storage operations. This merging approach reduces the number of individual file processing operations from N (number of small files) to 1 (archive file), thereby eliminating processing bottlenecks and improving throughput while maintaining manageable system complexity
Solution Approach 2:
The system performs preliminary archiving and compression operations on small files before they are stored in the multi-tier storage system. By consolidating files in advance, the storage system handles fewer, larger objects rather than numerous small files, preventing downstream processing bottlenecks and improving overall productivity
2Adaptability or versatility
If small files are stored individually in the storage system, then each file can be accessed independently, but scalability is adversely impacted when processing large numbers of files
Solution Approach 1:
Multiple small files are merged into a single archive object for storage in the multi-tier system. This approach improves scalability by reducing the total number of objects the system must manage, while data access performance is maintained through the ability to extract individual files from the archive when needed
Solution Approach 2:
The archive file format (e.g., TAR) acts as an intermediary structure between the original small files and the storage system. This intermediary allows the system to handle fewer objects at the storage layer while preserving the logical independence and accessibility of individual files through extraction operations
3Device complexity
If small files are processed without compression, then processing operations are simpler, but storage space efficiency and transfer performance are reduced
Solution Approach 1:
The system merges multiple small files into a single archive and simultaneously applies compression. This combined approach achieves storage efficiency through compression while the merging process itself manages the complexity of handling multiple files, resulting in both space efficiency and operational feasibility
Data Source
AI summary
An apparatus in one embodiment comprises a multi-tier storage system having at least a front-end storage tier, a back-end storage tier and a data mover module configured to control movement of data between the storage tiers. The multi-tier storage system is configured to receive a plurality of files, such as relatively small size files containing data from respective Internet of Things (IoT) devices. The front-end storage tier comprises a file system and at least one key-value store. The multi-tier storage system is configured to combine multiple ones of the received files written to at least one specified directory of the file system into an object for storage in association with a key in the key-value store. The object comprises multiple sub-keys identifying respective ones of the multiple files within the object. The data mover module may be configured to copy the object from the key-value store to the directory.