Dual-Queue System for Dynamic Data Resource Scaling

The patent implements dynamic queue nodes that can be instantiated and terminated based on real-time data flow conditions. When data volume increases, additional queue nodes are automatically created to handle the load; when data volume decreases, nodes are terminated to free resources. This dynamic adaptation resolves the contradiction by making system resources flexible rather than static, allowing the system to scale up only when necessary.

The patent creates a unified queue management system where a single queue node can serve multiple functions: handling data ingestion, performing data operations, and managing data egress. The system can dynamically allocate these functions across multiple nodes based on workload requirements. This multi-functionality allows the system to achieve high processing capacity without proportionally increasing the number of dedicated components for each function.

The patent segments the data repository into multiple independent queue nodes, each handling a specific portion of the data load. Each queue node is a self-contained unit with its own data structures and processing logic. This segmentation allows the system to scale data capacity by adding more segments (nodes) rather than increasing the complexity of a single monolithic system, resolving the contradiction between data capacity and system complexity.

The patent introduces a queue manager as an intermediary component that coordinates between multiple queue nodes and the external system. The queue manager handles node instantiation, termination, and data routing, abstracting the complexity of managing multiple nodes from the rest of the system. This intermediary layer allows the system to handle large data volumes while maintaining manageable complexity through centralized coordination.

The patent implements ephemeral queue nodes that are created when needed and terminated when not needed, rather than maintaining permanent storage for all data. Data is retained only as long as it is actively being processed or needs to be accessed. Once data has been processed and is no longer needed, it is discarded rather than permanently stored. This approach maintains data availability during the active processing period while significantly reducing resource consumption compared to permanent storage.

The patent employs periodic monitoring of data queue conditions and dynamic adjustment of node lifecycle. The system periodically evaluates whether queue nodes should be instantiated or terminated based on current data flow patterns and storage requirements. This periodic action allows the system to maintain data availability when needed while minimizing resource consumption during low-activity periods, resolving the contradiction between reliability and resource usage.