Hardware Offloading Device for Object Storage File Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In object-based storage systems, massive amounts of small files result in a high number of object file accesses, leading to increased request fees and data access costs.
Innovation Solution
A data access method and system that involves a hardware offloading device caching to-be-written files and merging them into a new object file before writing it into the object-based storage system, reducing the number of object file accesses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If each small file is written as a separate object file into the object-based storage system, then data persistence and availability are maintained, but the number of object file accesses increases significantly, leading to higher request fees and data access costs
Solution Approach 1:
The patent merges multiple small files into a single object file before writing to the object-based storage system. The hardware offloading device aggregates to-be-written files from the electronic device, combines them into one consolidated object file, and then performs a single write operation to the storage system. This merging approach maintains data persistence while dramatically reducing the number of access operations and associated request fees.
Solution Approach 2:
The patent performs preliminary file merging in the hardware offloading device before the actual write operation to the object-based storage system. By consolidating multiple small files into a single object file in advance, the system prepares the data in an optimized format that reduces subsequent access operations. This preliminary action eliminates the need for multiple separate write operations, thereby reducing request fees while maintaining data integrity.
2Productivity
If multiple small files are cached and merged into a single object file before writing, then the number of object file accesses is reduced, but the cache storage requirement and processing complexity increase
Solution Approach 1:
The patent introduces a hardware offloading device as an intermediary between the electronic device and the object-based storage system. This intermediary component performs the file merging and caching operations, offloading the processing complexity from the main electronic device. The hardware offloading device maintains a cache for temporary file storage and handles the consolidation logic, thereby improving data access efficiency without significantly increasing the complexity of the primary system components.
3Object-generated harmful factors
If file merging is performed on cached files, then the number of object file accesses to the storage system is reduced, but the time required for file merging processing increases
Solution Approach 1:
The patent implements continuous file merging operations in the hardware offloading device. Instead of performing merging only when the cache is full, the system continuously aggregates to-be-written files as they arrive in the cache. This continuous action allows the system to maintain an ongoing consolidation process, reducing the overall waiting time and enabling faster write operations to the object-based storage system, thereby lowering data access costs without excessive delay.
Data Source
AI summary
Embodiments of the present application provide a data access method and system, a hardware offloading device, an electronic device, and a medium. In the embodiments of the present application, on one hand, a data access task for an object-based storage system is transferred from an electronic device to a hardware offloading device for execution, which can reduce processing pressure on the electronic device, improve processing performance of the electronic device, and enhance data access performance. On the other hand, after every time receiving a to-be-written file from an application, the hardware offloading device does not directly write the to-be-written file into the object-based storage system, but first caches it locally, and after a plurality of to-be-written files are cached, the hardware offloading device merges the plurality of to-be-written files into a new object file to write it into the object-based storage system.