Memory Management for Large Data Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computing systems face inefficiencies in managing memory and storage space, particularly when dealing with large datasets, as they often rely on virtual memory and disk swapping, which can impact performance by requiring data to be spilled to disk, leading to slower processing times.
Innovation Solution
A computing system that dynamically manages working memory and overflow storage space by performing data operations in a way that allows for in-place updates and strategic storage of data in either memory or overflow storage, depending on key matches, thereby avoiding the need to spill data to disk and optimizing memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If virtual memory and disk swapping are used to manage memory for large datasets, then the apparent memory size is extended beyond physical memory limits, but processing performance deteriorates due to slower disk access speeds
Solution Approach 1:
The patent segments data processing into multiple passes, where each pass processes a portion of the dataset that fits in memory. Intermediate results are stored in sorted temporary files on disk, allowing the system to handle datasets larger than memory while maintaining efficient in-memory processing speeds for each segment.
Solution Approach 2:
The patent performs preliminary sorting of data before the main processing operation. By pre-sorting input data and organizing intermediate results in sorted temporary files, the system eliminates the need for slow disk swapping during the actual data operation, as data can be efficiently streamed and processed in order.
2Productivity
If all data is kept in working memory for fast access, then processing speed is improved, but memory usage exceeds available physical memory
Solution Approach 1:
The patent divides the dataset into segments that individually fit within available memory. Each segment is processed separately, with results written to sorted temporary files. This allows the system to process large datasets by maintaining only manageable portions in memory at any given time, avoiding memory exhaustion while preserving fast in-memory processing speeds.
Solution Approach 2:
The patent employs periodic processing where data is read from input, processed in memory, and written to output in periodic batches. This rhythmic flow of data through memory allows efficient utilization of available memory resources while continuously processing the entire dataset over multiple cycles.
3Quantity of substance
If data is spilled to disk during processing, then memory constraints are relieved, but processing time increases due to slower disk I/O operations
Solution Approach 1:
The patent performs preliminary sorting and organization of data before main processing. By pre-processing data into sorted temporary files with predictable access patterns, the system minimizes random disk I/O during the actual data operation, reducing the time penalty associated with disk access.
Solution Approach 2:
The patent maintains continuous processing by keeping data streams flowing through the system. Instead of stopping to perform expensive disk operations, the system continuously reads from sorted input files, processes data in memory, and writes to sorted output files, minimizing idle time and maintaining high throughput throughout the processing operation.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Processing a plurality of data units to generate result information, includes: performing a data operation for each data unit of a first subset of data units from the plurality of data units, and storing information associated with a result of the data operation in a first set of one or more data structures stored in working memory space of a memory device (108); after an overflow condition on the working memory space is satisfied, storing information in overflow storage space of a storage device (110); and repeating an overflow processing procedure multiple times during the processing of the plurality of data units, the overflow processing procedure including: updating a new set of one or more data structures stored in the working memory space using at least some information stored in the overflow storage space.