Decompression Overflow Handling with Cache Splicing and Re-Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In decompression processes, data overflow occurs when the decompressed file is larger than expected, leading to inefficiencies and task suspension, as traditional methods fail to accurately manage cache resources, resulting in data loss and repeated decompression.
Innovation Solution
A method and device that detect data overflow events, store decompressed data in a target memory, close the decoding engine's read-in port, calculate the position of decompressed data, re-decompress incomplete data, and splice it with other data to achieve complete decompression, ensuring efficient data handling and avoiding repeated decompression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional approach suspends current task to avoid overflow, then internal memory leakage is avoided, but decompression efficiency deteriorates due to task suspension and repeated decompression
Solution Approach 1:
The decompression process is segmented into multiple stages: initial decompression into host cache, detection of overflow condition, and subsequent handling of remaining data. The host cache is segmented as a temporary buffer region that can be safely overflowed without affecting internal memory, allowing the system to process data in manageable chunks rather than requiring complete decompression upfront.
Solution Approach 2:
The host cache acts as an intermediary buffer between the decompression circuit and the final storage location. It temporarily holds decompressed data that exceeds the originally allocated space, preventing direct overflow into internal memory while enabling continuous decompression operations. This intermediary structure resolves the conflict between maintaining memory safety and sustaining decompression throughput.
2Reliability
If larger cache region is allocated to avoid overflow, then data loss is prevented, but cache resource utilization deteriorates due to excessive memory allocation
Solution Approach 1:
The cache allocation is made dynamic rather than static. The host cache serves as a flexible buffer that accommodates variable decompression sizes without requiring pre-allocation of maximum possible space. The system dynamically adjusts to actual decompression needs by allowing controlled overflow into the host cache region, optimizing memory utilization while preventing data loss.
Solution Approach 2:
The system changes the parameter of cache region accessibility by introducing a read-only port restriction during overflow conditions. When overflow is detected, the host cache's write port remains open to accept additional decompressed data, but the read port is controlled to prevent premature reading that would cause data loss. This parameter change enables the cache to expand effectively without permanently allocating excessive memory resources.
3Productivity
If decompression continues without suspension to maintain efficiency, then productivity is improved, but data overflow into internal memory occurs causing data loss
Solution Approach 1:
The host cache is pre-configured as an overflow-safe region before decompression begins. By establishing this protected buffer space in advance and configuring its read/write port behaviors, the system prepares for potential overflow conditions without interrupting the decompression flow. This preliminary setup allows continuous high-speed decompression while guaranteeing data safety through the pre-arranged buffer structure.
Solution Approach 2:
The system creates a copy mechanism where decompressed data is first written to the host cache (a safe copy location) before being transferred to final storage. This copying approach with controlled access ports ensures that even if the decompression exceeds original expectations, the data exists in a protected intermediate location, preventing loss while maintaining continuous processing throughput.
Data Source
AI summary
A method for processing data overflow in a decompression process, includes: decompressing an original text, and detecting whether a data overflow event occurs in the decompression process; in response to detecting the data overflow event, storing first data obtained by decompression in a host cache into a target memory, and closing a data read-in port of a decoding engine; decompressing data which is being decompressed in the decoding engine to obtain second data, and storing the second data into a cache of the decoding engine; calculating a position of the decompressed data in the original text; obtaining, on the basis of the position, data which is not decompressed in the original text, re-decompressing the data which is not decompressed to obtain third data, and storing the second data into the target memory; and splicing the first data, the second data, and the third data to obtain complete decompressed data.