Shared Memory Data Aggregation for Multi-Process Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing data processing systems in computer applications face performance degradation due to resource-intensive data saving operations in shared storage, especially when multiple work processes compete for storage resources, and there is a need to reduce the volume of data stored directly in permanent storage since not all data is critical or required in its raw form.
Innovation Solution
Implementing a system where work processes write and read data units in shared memory, aggregate and process them before transferring to storage, using exclusive access mechanisms and versioning to manage parallel operations, thereby reducing the number of data saving operations and storage resource competition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple work processes save data directly to permanent storage, then data integrity is maintained, but resource consumption and storage competition increase significantly
Solution Approach 1:
The patent introduces shared memory as an intermediary buffer between work processes and permanent storage. Data is first written to shared memory areas by individual processes, then aggregated and saved to storage by a dedicated data saving process, reducing direct competition for storage resources while maintaining data integrity through structured access protocols.
Solution Approach 2:
The system performs preliminary data aggregation in shared memory before saving to permanent storage. The data saving process collects data units from multiple processes in advance, consolidates them into batches, and then performs unified storage operations, reducing the frequency and intensity of storage access operations.
2Reliability
If multiple work processes save data directly to permanent storage, then data is preserved immediately, but the performance of work processes decreases due to resource competition
Solution Approach 1:
Shared memory serves as a mediator that decouples the data preservation function from the work processes. Processes write to shared memory without blocking, and a dedicated data saving process handles the actual storage operations, allowing work processes to maintain high productivity while data is reliably preserved asynchronously.
Solution Approach 2:
Data is preliminarily stored in shared memory with minimal processing overhead, allowing work processes to continue execution without waiting for permanent storage operations. The data saving process performs the actual preservation in advance batches, maintaining both productivity and reliability.
3Loss of information
If all data is stored in raw form in permanent storage, then complete information is available, but storage space is wasted on non-critical data
Solution Approach 1:
The patent applies different quality levels to different data in shared memory. Critical data is marked for mandatory persistence, while non-critical data is marked for optional aggregation or compression. This local quality differentiation allows the system to optimize storage space by storing only essential data in permanent storage while maintaining complete information in the memory buffer.
Solution Approach 2:
The system performs partial data persistence by selectively saving only critical data units to permanent storage, while leaving non-critical data in the shared memory buffer. This partial action approach reduces storage space consumption while maintaining information completeness for essential operations.
4Quantity of substance
If data is aggregated and processed before storage, then storage space is optimized, but the complexity of data management increases
Solution Approach 1:
The data management system is segmented into distinct functional components: work processes that generate data, a shared memory structure that organizes data by process, and a data saving process that handles aggregation and persistence. This segmentation reduces management complexity by assigning specific responsibilities to each component while optimizing storage space through coordinated aggregation.
Data Source
AI summary
Various embodiments of systems and methods for processing data in shared memory are described herein. A number of work processes of an application server write data in corresponding areas of shared memory. At least one data unit for a first process is read from a first area of the shared memory by the first process. The first process also reads at least one unit of data for a second process from a second area of the shared memory. The first process writes information in a third area of the memory to indicate that the at least one unit of data for the first process and the at least one unit of data for the second process are read. The read data units are aggregated and saved in a storage by the first process.