Immutable Batch Data Delivery System for SLA Compliance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional distributed systems face challenges in ensuring timely and complete data delivery to consumers, as threads waiting for each other can violate Service Level Agreements (SLAs), leading to incomplete data snapshots and lack of clean abstraction in data processing.
Innovation Solution
A method and system that retrieves data from multiple data centers, stores it in a directory, bundles it into batches, and delivers each batch to both batch and streaming consumers, ensuring immutability and parallel delivery to maintain SLA compliance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If threads pull data sequentially from remote data centers waiting for each other, then data delivery follows a predefined order, but Service Level Agreement (SLA) timing requirements are violated and data completeness is compromised
Solution Approach 1:
The patent segments the data delivery process into independent batch operations. Each batch is pulled from remote data centers as an independent unit rather than waiting for sequential thread completion. This segmentation allows parallel processing of multiple batches, ensuring SLA timing requirements are met while maintaining data completeness through proper batch ordering and delivery tracking.
2Stability of the object's composition
If threads wait for each other to complete data pulling, then data is delivered in a controlled sequence, but the consumer receives incomplete data snapshots and lacks clean abstraction
Solution Approach 1:
The patent introduces an intermediary batch delivery mechanism that decouples the complexity of thread coordination from the consumer. The system pulls multiple batches in parallel using multiple threads, but the intermediary batch delivery layer ensures proper ordering and completeness before presenting data to the consumer. This provides clean abstraction to the consumer while handling thread coordination complexity internally.
3Productivity
If parallel data pulling is implemented without batch bundling, then data delivery speed increases, but data consistency and immutability are compromised
Solution Approach 1:
The patent applies preliminary action by bundling data into immutable batches before delivery. Each batch is prepared, validated, and sealed as a complete unit before being delivered to consumers. This preliminary batching ensures data immutability and consistency while allowing parallel pulling of multiple batches to maintain high delivery speed. The batches are delivered in order, ensuring consistency despite parallel processing.
Data Source
AI summary
A method and system delivers data to a batch consumer and a streaming consumer. The method and system include retrieving data from a plurality of data centers, storing the data in a first directory, bundling the data into plurality of batches in the first directory, transferring each batch of the plurality of batches to the second directory, delivering the each batch of the plurality of batches in the second directory to the batch consumer and the streaming consumer and delivering the data in the first directory to the streaming consumer.