Backup File List Segmentation for Memory Preservation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face challenges in preserving memory resources during large-scale data backup, as they require significant memory and time, and previous solutions like breaking up logical file systems or journaling systems have limitations such as compatibility issues and unreliable accuracy.
Innovation Solution
A system and method that includes a generation module to create an unordered, randomly accessible list of backup files and attributes, a processing module to update backup files one at a time, and optional modules for marking matches and expiring unmarked files, allowing efficient data backup without relying on extensive memory resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If file lists from local and central repositories are compared to identify changed files, then data backup accuracy is improved, but memory consumption increases significantly
Solution Approach 1:
The patent divides the file comparison process into segments by creating separate file list files for different directories or file sets. Instead of loading all files into memory simultaneously, the system processes file lists in manageable segments, comparing them in pairs rather than as one large comprehensive comparison. This segmentation approach maintains backup accuracy while reducing peak memory consumption to levels proportional to individual segment sizes rather than the total file system size.
2Reliability
If file attribute information is stored in a central repository for comparison, then backup reliability is improved, but the time required to scan and compare files increases
Solution Approach 1:
The patent implements preliminary action by pre-generating and storing file list files that contain file attribute information (timestamps, sizes, checksums) before the actual backup comparison operation. These pre-prepared file lists are stored in a efficient format on disk or in cached memory. During backup, the system directly loads these pre-computed lists rather than scanning the file system in real-time, significantly reducing the time required for file comparison while maintaining complete and accurate file attribute data for reliable change detection.
3Quantity of substance
If virtual memory is extended to handle large file lists, then memory capacity is improved, but the problem of insufficient memory is only postponed
Solution Approach 1:
The patent extracts the file list data from volatile memory and stores it in persistent storage format optimized for sequential and random access. File list files are written to disk in a compact binary format that preserves all necessary file attribute information. During backup operations, only the portions of file lists currently being processed are loaded into memory, with the remainder remaining on disk. This extraction approach eliminates the need for extensive virtual memory extensions while maintaining the ability to handle arbitrarily large file systems.
4Quantity of substance
If logical file systems are broken into smaller parts, then memory requirements are reduced, but compatibility with inherited large file systems deteriorates
Solution Approach 1:
The patent introduces a new dimensional organization by creating a hierarchical structure of file list files that mirrors the directory structure of the file system. Each directory or logical unit generates its own file list file, and the backup system navigates this hierarchical organization to process files in manageable units. This dimensional approach allows the system to handle any size of file system by simply traversing the hierarchy deeper or broader, without requiring actual partitioning of the logical file system itself, thereby maintaining full compatibility with inherited file systems while reducing memory requirements through controlled-unit processing.
Data Source
AI summary
A system and method to efficiently back up files while preserving memory resources. The system includes a generation module to generate an unordered list of backup files and associated attributes, and a processing module to back up local files one at a time by modifying the backup files to reflect changes to the local files. The processing module may include a retrieval module to retrieve local file attribute information from the hard disk, a comparator module to compare the local file attributes to corresponding backup file information on the list, and an update module to update the corresponding backup file to reflect any differences. In this manner, the present invention avoids processing limitations caused by insufficient memory and/or excessive quantities of local files.