Archive Server Small File Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
File systems struggle to efficiently manage and store large numbers of small files, leading to poor performance due to exhaustion of random Input/Output resources and inefficient storage volume, as they are typically optimized for handling larger files.
Innovation Solution
An Archive Server mediates between client workloads and file system performance by identifying and managing duplicate file occurrences, archiving files into larger units, and communicating with the file system in archive units, thereby relieving the file system of managing small files and improving storage efficiency through deduplication and purging operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a file system stores each small file individually, then file retrieval is straightforward, but storage volume efficiency deteriorates and random I/O resources are exhausted
Solution Approach 1:
The patent combines multiple small files into archive units for storage. Instead of storing each small file individually in the file system, the archiving server groups related small files into archives, which are then stored as larger units. This merging approach improves storage volume efficiency by reducing metadata overhead and improving space utilization, while the file system still provides straightforward retrieval through the archive interface.
2Reliability
If duplicate files are stored separately, then each file occurrence is preserved, but storage efficiency deteriorates due to redundant data
Solution Approach 1:
The patent changes the storage parameter from individual file copies to deduplicated archive units. When duplicate files are detected, the system stores only one copy in the archive and maintains references to it for all duplicate occurrences. This parameter change preserves all file occurrences for reliability while dramatically improving storage efficiency by eliminating redundant data storage.
3Device complexity
If the file system manages small files directly, then no additional complexity is introduced, but performance deteriorates due to resource exhaustion
Solution Approach 1:
The patent introduces an archiving server as an intermediary between clients and the file system. This mediator receives small files from clients, archives them into larger units, and manages storage in the file system. The file system's performance improves because it no longer needs to handle numerous small file operations, while the additional archiving layer manages complexity without affecting the file system's core functionality.
4Loss of information
If archives retain all file occurrences, then complete history is maintained, but storage space is wasted due to duplicates
Solution Approach 1:
The patent uses a reference copying mechanism where duplicate file occurrences are represented by references to the original archive entry rather than storing actual duplicate copies. The catalog maintains references to all file occurrences, preserving complete file history and enabling retrieval of any occurrence, while the actual storage space is used only once for the unique file content, optimizing space utilization.
Data Source
AI summary
Methods and systems for storing and managing large numbers of small files. A data processing system includes clients that generate large numbers be stored on a storage device managed by a File System (FS). An Archive Server (AS) receives multiple files from the client, archives the files in larger archives, and sends the archives to the FS for storage. When requested to read a file, the AS retrieves the archive in which the file is stored, extracts the file and sends it to the requesting client. In other words, the AS communicates with the clients in individual file units, and with the storage device in archive units. The AS is typically constructed as an add-on layer on top of a conventional FS, which enables the FS to handle small files efficiently without modification.