Distributed Processing Subset Management for Data Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed processing systems face challenges in managing output sets effectively after a server failure, leading to increased storage needs and inefficiencies in data consistency and duplicate output prevention.
Innovation Solution
A distributed processing method that divides data into subsets and assigns a set number to each subset, allowing for efficient deletion, prevention of duplicate outputs, and supplementation, while maintaining data consistency by using a division command in partial order delivery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is kept in the output set to prevent duplicate output among servers, then data consistency is maintained, but storage capacity requirement increases
Solution Approach 1:
The patent segments the output set into multiple subsets, where each subset contains data from a specific time period or processing batch. This segmentation allows individual subsets to be managed and deleted independently, reducing the total storage capacity required while maintaining data consistency through selective retention of necessary subsets.
Solution Approach 2:
The patent establishes preliminary rules for adding and deleting subsets in the output set before actual server operations occur. These pre-defined management rules enable automatic deletion of obsolete subsets while preserving necessary data for duplicate prevention, thereby reducing storage requirements while maintaining reliability.
2Reliability
If all servers keep complete output sets to prevent duplicate data, then data consistency is secured, but system complexity increases
Solution Approach 1:
By dividing the output set into manageable subsets, the patent simplifies the complexity of managing complete output sets across all servers. Each server only needs to track and manage subset identifiers rather than entire data sets, reducing the computational and memory overhead while maintaining data consistency through coordinated subset management.
Solution Approach 2:
The patent extracts the essential identification information (subset identifiers) from the complete output sets and keeps only these extracted identifiers in each server's memory. This extraction reduces system complexity by eliminating the need to store and compare complete data sets, while still enabling effective duplicate detection through identifier matching.
3Productivity
If servers switch processing after failure quickly, then productivity is improved, but data consistency may be compromised
Solution Approach 1:
The patent pre-establishes subset management rules and output set structures before server failures occur. When a server failure happens, the surviving servers can immediately apply these pre-defined rules to manage the output set, enabling quick switching while maintaining data consistency without requiring complex real-time negotiations or checks.
Solution Approach 2:
The output set management system operates autonomously using pre-defined rules, allowing servers to self-manage data consistency without extensive inter-server communication during failure scenarios. This self-service capability enables rapid switching while preserving data integrity, as each server independently applies the established subset management rules.
Data Source
AI summary
A distributed processing method for executing partial order delivery of data on a plurality of computers connected via a network, the distributed processing method includes receiving the data by the plurality of computers. An output order in which the data is output by the partial order delivery is determined by the plurality of computers. The data to be output by dividing the data into a plurality of subsets equivalent among the plurality of computers and then stored in the output set.