Data Consolidation Agent for Cluster Storage Parallel Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Cluster-based storage systems face processing delays and challenges in parallel processing due to the primary storage appliance handling most external IO commands and conventional remote data collection and local data processing being heavily dependent on current processing loads.

Innovation Solution

Deploying an agent to communicate with a centralized database and multiple remote databases, polling them for data, consolidating it to the centralized database, and processing it in parallel, thereby reducing processing delays and improving efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data is distributed across multiple storage appliances in a cluster-based configuration, then data accessibility and storage capacity are improved, but processing delays occur because the primary storage appliance must process all external IO commands sequentially

Engineering Contradiction:

Improvestorage capacityVSAvoidprocessing delay

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

A data consolidation agent is introduced as an intermediary component that collects data from multiple remote storage appliances and consolidates it to a centralized database. This agent handles the data movement and processing tasks, freeing the primary storage appliance from sequential processing bottlenecks while maintaining distributed data architecture benefits

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system segments data processing functions by separating data collection, consolidation, and processing tasks into distinct components. Remote storage appliances focus on data storage while the data consolidation agent handles data movement and the centralized database manages processing, enabling parallel operation and eliminating the sequential processing bottleneck at the primary storage appliance

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If conventional remote data collection and local data processing is used, then data can be collected from remote sources, but performance is heavily dependent on current processing loads and parallel processing challenges arise

Engineering Contradiction:

Improvedata collection capabilityVSAvoidprocessing efficiency

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The data consolidation agent serves as a dedicated intermediary that handles remote data collection and consolidation tasks independently of the primary storage appliance's processing load. This separation allows data collection to proceed in parallel with local data processing, eliminating performance dependency on current processing loads

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system adds a new operational dimension by introducing a separate data consolidation pathway that runs independently from the traditional remote data collection and local processing pipeline. This parallel dimension allows both operations to occur simultaneously without interfering with each other, improving overall productivity

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS12235806B2System and method for distributed data consolidation

Publication Date: 2025.02.25 EMC IP HLDG CO LLC

AI summary

A method, computer program product, and computing system for deploying an agent configured to communicate with a centralized database and a plurality of remote databases. The plurality of remote databases may be polled, via the agent, for data for storage in the centralized database. The data may be consolidated from the plurality of remote databases to the centralized database.