Dynamic Memory Allocation for Big Data Aggregator Functions

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current systems inefficiently allocate memory when processing massive amounts of data, particularly in big data applications with high-cardinality dimensions, leading to significant memory waste due to the unpredictable memory requirements of aggregator functions following a power-law distribution.

Innovation Solution

A method of dynamically allocating memory to aggregator functions by initializing a minimum amount and growing based on a predefined growth factor, retaining old memory spaces until the job is complete, and managing memory externally or internally to optimize usage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If fixed memory allocation is used for aggregator functions, then memory management is simple, but memory consumption increases significantly (up to 4095 times) compared to dynamic allocation

Engineering Contradiction:

Improvememory management complexityVSAvoidmemory consumption

Core Design Contradiction:

Device complexityVSQuantity of substance

Solution Approach 1:

The patent applies dynamic memory allocation where aggregator functions start with a minimum memory size and automatically grow their memory capacity based on actual data processing needs. The memory growth follows a geometric progression with a predefined growth factor, allowing the system to adapt memory allocation to the power-law distribution of data frequencies, thereby reducing overall memory consumption while maintaining manageable complexity through automated management

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes the memory allocation parameter from a fixed value to a dynamic value that evolves during data processing. Memory allocation transitions from a static parameter to one that adjusts based on the aggregator's actual requirements, utilizing geometric growth to optimize the balance between memory usage and processing capability

Inventive Principle:

Principle #35Parameter changes

2Quantity of substance

If dynamic memory allocation with geometric growth is used, then memory consumption is reduced (up to 4095 times), but memory management complexity increases

Engineering Contradiction:

Improvememory consumptionVSAvoidmemory management complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The aggregator functions perform self-service by automatically managing their own memory allocation and growth without external intervention. Each aggregator monitors its own memory needs and triggers geometric growth when capacity is insufficient, eliminating the need for centralized memory management complexity while achieving optimal memory consumption

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system implements dynamic memory management where allocation decisions are made autonomously by each aggregator based on its operational state, transforming memory management from a complex centralized control problem into simple local self-adjustment mechanisms

Inventive Principle:

Principle #15Dynamics

3Productivity

If memory is allocated for all possible aggregator functions upfront, then all functions are ready to process data, but memory waste increases due to unpredictable memory requirements following power-law distribution

Engineering Contradiction:

Improvedata processing readinessVSAvoidmemory waste

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The system performs preliminary action by pre-allocating only the minimum necessary memory for each aggregator function before data processing begins. This preliminary minimum allocation ensures all functions are ready to process data immediately, while the geometric growth mechanism activates only when actual data processing demands exceed the initial allocation, preventing memory waste on over-provisioning

Inventive Principle:

Principle #10Preliminary action

4Device complexity

If geometric growth with constant factor is used, then memory allocation is simple and automated, but the system cannot handle cases where memory exceeds upper size bound

Engineering Contradiction:

Improvememory allocation complexityVSAvoidmemory bound handling

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

The system incorporates feedback mechanisms where aggregators monitor their memory usage against predefined upper size bounds and report status to the memory management system. When an aggregator approaches its memory limit, the feedback loop triggers either expansion of the upper bound (if resources are available) or termination of the aggregator (if resources are exhausted), ensuring reliable handling of memory bound conditions while maintaining allocation simplicity

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12430174B2Systems and methods for memory management in big data applications

Publication Date: 2025.09.30 YAHOO ASSETS LLC

AI summary

The disclosed embodiments describe techniques for allocating memory to functions processing big data. In one embodiment, a method is disclosed comprising allocating a first memory space to a function, the memory space comprising an initial amount of memory for the function; declaring the first memory space as the current memory space; processing data using the function, the processing writing data to the current memory space; determining that the function requires additional memory space; allocating a new memory space based on the current memory space and a growth factor; copying all data in the current memory space to the new memory space; declaring the current memory space as the old memory space; declaring the new memory space as the current memory space; and not deallocating the old memory space.