File Processing Method for JVM Memory Overflow Prevention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file processing methods encounter memory overflow issues when dealing with large numbers of small files, as they fill up the JVM memory by loading all file objects at once, leading to unavailability of the file system.
Innovation Solution
A file processing method that generates a folder stream and traverses it sequentially, creating file objects only as needed, and deletes them once their size is acquired and utilized, avoiding the creation of multiple file objects at once and ensuring timely memory clearance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all files are traversed at once and file objects are loaded into memory to acquire total size, then the total size can be obtained, but memory overflow occurs causing file system unavailability
Solution Approach 1:
The patent segments the file traversal process into iterative batches using a folder stream iterator. Instead of loading all file objects simultaneously, the system processes files in sequential batches, releasing memory after each batch is processed. This segmentation prevents memory overflow while maintaining the ability to calculate total file size accurately.
Solution Approach 2:
The patent implements discarding and recovering by explicitly clearing file objects from memory after their size information is extracted. The folder stream iterator processes each file, retrieves its size, adds it to the total, then discards the file object reference to allow garbage collection. This cycle continues until all files are processed, recovering memory resources while completing the measurement task.
2Productivity
If file objects are created for all files simultaneously, then size statistics can be performed, but memory consumption increases causing overflow
Solution Approach 1:
The patent divides the file processing workload into sequential segments using the folder stream iterator. Each iteration processes a subset of files rather than all files at once, creating file objects only for the current batch. This segmentation maintains processing productivity while controlling memory consumption by limiting the number of simultaneous file objects in memory.
Solution Approach 2:
The patent employs periodic action through iterative processing with explicit memory clearance between iterations. The folder stream iterator cycles through files in batches, performing size statistics periodically while clearing memory resources after each cycle. This periodic approach sustains productivity over time while preventing cumulative memory consumption that would lead to overflow.
Data Source
AI summary
Provided are a file processing method, apparatus and device, and a readable storage medium. The method includes: judging whether a received file access path exists; if so, judging whether the file access path corresponds to a folder; if so, generating a folder stream, traversing the folder stream, and generating a file object corresponding to the folder stream; storing the file object in a memory, acquiring the size of a corresponding file according to the file object, detecting whether there is a file in the folder, and a size of the file is known and the file has been traversed and used, and if so, deleting, from the memory, the file object corresponding to the file; and acquiring the total size of files in the folder according to the size of each file in the folder.