Elastic Data Grid Memory Allocation via Resource Manager
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern computing systems face challenges in managing memory allocation efficiently in distributed data grids, particularly when dealing with high demand from millions of users, leading to exponential increases in transaction calls that exceed the limited capacity of databases and metadata stores.
Innovation Solution
Implementing a system that uses an elastic data structure with journal files, where a resource manager suspends processes when memory appears full and resumes them after reclaiming sufficient memory, allowing for asynchronous operations and push-back mechanisms to manage memory allocation effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the elastic data structure is used to store data with journal files, then the memory allocation flexibility is improved, but the system complexity increases
Solution Approach 1:
A resource manager is introduced as an intermediary component that manages memory allocation for the elastic data structure. The resource manager monitors memory usage, suspends processes when memory is full, and resumes them after reclaiming memory, thereby simplifying the overall system architecture while maintaining flexibility.
Solution Approach 2:
The system implements dynamic memory management where the elastic data structure can grow and shrink based on available memory. Processes are dynamically suspended and resumed based on memory conditions, allowing the system to adapt to changing memory availability without fixed allocation boundaries.
2Reliability
If the process is suspended when the elastic data structure appears full, then the memory overflow error is prevented, but the processing time increases
Solution Approach 1:
The resource manager performs preliminary actions by proactively suspending processes before memory overflow occurs. When the elastic data structure reaches its capacity, the resource manager suspends the process in advance, preventing overflow errors while managing the timing of memory reclamation and process resumption.
3Quantity of substance
If memory is reclaimed from the elastic data structure, then the available memory capacity is improved, but the system performance fluctuates
Solution Approach 1:
The resource manager implements periodic memory reclamation by suspending and resuming processes in a controlled manner. This periodic action allows memory to be reclaimed and made available while managing the impact on system performance through regulated process suspension and resumption cycles.
Data Source
AI summary
A system and method can support memory allocation control in a distributed data grid. The system can designate a process, such as a logical process, to handle a request that is received from a client for storing data in a data storage using an elastic data structure with one or more journal files. Then, a resource manager associated with the data storage can suspend the process when the elastic data structure appears to be logically full. Furthermore, the resource manager can resume the suspended process associated with the client, after the resource manager has reclaimed sufficient memory from the elastic data structure.