Acceleration Layer for Distributed Cluster Data Transmission
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed-clustered applications face inefficiencies in data transmission and storage due to the use of direct-attached storage systems, leading to increased latency and resource consumption, especially with larger data amounts, making it difficult to implement consistent storage policies and manage redundancy.
Innovation Solution
The implementation of an acceleration layer and connectors that use a unique identifier to store and retrieve data from a storage repository, reducing the amount of data transmitted through the system, thereby decreasing latency and resource usage, while allowing for centralized data protection and governance policies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across multiple DAS servers for fault-tolerance, then reliability is improved, but storage space consumption and network bandwidth usage increase
Solution Approach 1:
The patent extracts the data storage function from the distributed-clustered application servers and places it in a separate storage repository. Only metadata (unique identifiers) remains in the distributed application, while actual data is stored externally. This separation reduces storage space consumption in the distributed system while maintaining fault-tolerance through the external repository's replication capabilities.
Solution Approach 2:
The patent introduces a storage repository as an intermediary between producers and consumers. This intermediary handles data storage and retrieval operations, allowing the distributed-clustered application to focus on processing while the storage repository manages data persistence and fault-tolerance, thereby reducing overall storage space consumption in the distributed system.
2Reliability
If data is replicated across multiple DAS servers for fault-tolerance, then reliability is improved, but network bandwidth consumption increases
Solution Approach 1:
The patent extracts large data payloads from the data transmission path between producers and consumers. Instead of transmitting full data through the distributed-clustered application, only small unique identifiers are transmitted. The actual data is stored in a separate repository, significantly reducing network bandwidth consumption while maintaining reliability through the repository's fault-tolerance mechanisms.
Solution Approach 2:
The patent uses unique identifiers as lightweight copies or references to the actual data. These identifiers serve as proxies that can be transmitted efficiently through the distributed system, while the full data resides in the storage repository. This copying approach reduces network bandwidth consumption while maintaining data integrity and reliability.
3Speed
If DAS is used for data storage in distributed-clustered applications, then data can be stored close to processing nodes, but implementing consistent storage policies for data retention, protection, and governance becomes difficult
Solution Approach 1:
The patent merges all storage operations into a single centralized storage repository, consolidating data retention, protection, and governance policies in one location. This centralization makes it easier to implement and enforce consistent storage policies across all data, while the distributed-clustered application maintains fast access through local metadata caching and efficient query mechanisms.
4Reliability
If full data payloads are transmitted through the distributed-clustered application, then data integrity is maintained, but latency between producers and consumers increases
Solution Approach 1:
The patent extracts large data payloads from the transmission path, leaving only small unique identifiers to be processed by the distributed-clustered application. This extraction dramatically reduces transmission time and latency between producers and consumers. Data integrity is maintained because the unique identifiers link to the full data in the storage repository, ensuring accurate retrieval of the original payloads.
Solution Approach 2:
The patent introduces a storage repository as an intermediary that handles bulk data storage and retrieval. The distributed-clustered application acts as a mediator that processes only lightweight metadata and coordinates with the storage repository for actual data access. This intermediary architecture reduces latency by minimizing data traversal through the distributed system while maintaining data integrity through coordinated access patterns.
Data Source
AI summary
The technology disclosed herein enables accelerated data transmission between producers and consumers. In a particular example, a method includes receiving a first request from a producer-connector component of a producer component to store a payload to a storage repository. In response to the first request, the method includes providing a unique identifier to the connector component. The connector component provides the unique identifier to the distributed-clustered application. The method further includes storing the payload in association with the unique identifier to the storage repository. The method also includes retrieving the payload from the storage repository using the unique identifier to identify the payload in the storage repository. The method includes receiving a second request from a consumer-connector component of the consumer component to retrieve the payload. In response to the second request, the method includes supplying the payload to the consumer component.