Dynamic Log Shard Persistence via Router-Writer Assignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log management systems in distributed systems face inefficiencies in scalability, load balancing, and resource consumption, particularly in high data throughput environments, due to complex load distribution among writers and lack of a central management system, which makes it difficult to adapt to dynamic system changes.
Innovation Solution
A method and system for dynamically persisting log data by transmitting shards to router-writer pairs based on a writer assignment map, which identifies writers based on criteria such as number of writers, load, shard size, or estimated capacity, and continuously monitors and reassigned shards as needed to optimize resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a prior log management system distributes load among writers without central coordination, then system autonomy and simplicity are maintained, but load balancing efficiency and adaptability to dynamic changes deteriorate
Solution Approach 1:
The patent introduces a central log persistence manager as an intermediary component that coordinates between multiple writers and manages shard assignments. This manager receives log data, determines optimal writer assignments based on current system state, and routes data accordingly, thereby achieving efficient load balancing while maintaining system autonomy through centralized intelligence rather than distributed coordination overhead
Solution Approach 2:
The system dynamically adjusts writer assignments and shard distributions based on real-time monitoring of writer status, load conditions, and system changes. The log persistence manager continuously evaluates the state of writers and reassigns shards to optimize load balancing, enabling the system to adapt quickly to dynamic changes in writer availability and performance
2Productivity
If the number of writers and log categories increases to handle higher data throughput, then system capacity and scalability are improved, but election complexity and resource consumption worsen
Solution Approach 1:
The patent segments log data into multiple shards categorized by different categories or groups, with each shard independently assigned to specific writers. This segmentation allows the system to handle high data throughput by parallelizing processing across multiple writers while simplifying the assignment process through category-based routing, avoiding complex elections for each individual log entry
Solution Approach 2:
The log persistence manager performs preliminary assignment of shards to writers based on predetermined criteria and current system state before data arrival. By pre-establishing writer assignments for different shard categories and maintaining an assignment map, the system eliminates the need for complex runtime elections, reducing resource consumption while maintaining high throughput capacity
3Reliability
If log data is persisted with strict durability guarantees to prevent data loss, then reliability is improved, but persistence latency and resource consumption worsen
Solution Approach 1:
The patent implements dynamic parameter adjustment for log persistence operations, allowing the system to modify persistence guarantees, batching intervals, and confirmation thresholds based on current system conditions, data priorities, and writer performance. This enables the system to maintain high reliability for critical logs while reducing latency for less critical data by adjusting persistence parameters in real-time
Data Source
AI summary
Disclosed is a method and system for dynamically persisting log data from member computers in a network. The log data is transmitted as a plurality of shards to a router of router-writer pairs which transmits the shards to writers of the router-writer pairs that persist the shards into a storage unit. A shard contains a distinct subset of the log data. The shards are transmitted to the writers based on a writer assignment map that identifies a writer assigned to persist each of the shards. The assignment of the shards to a writer is performed based on assignment criteria that include at least one of (i) number of writers, (ii) a load of the writers, (iii) number of shards, (iv) size of a particular shard, or (v) estimated load capacity of writers.