Centralized Logging Array With NMP for Multi-Writer WAL
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Write Ahead Logs (WAL) systems in databases suffer from limitations such as single writer architecture, high latency, inefficient storage, and complex resource management, which hinder scalability and performance, particularly in multi-client environments.
Innovation Solution
A centralized logging storage array with a near memory processing (NMP) module that supports multiple concurrent writers, eliminates the need for replication, and shares state across allocation units, enabling efficient storage and simplified error recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single writer architecture is used in WAL, then consistency and coordination are maintained, but write performance and scalability are limited
Solution Approach 1:
The patent divides the log storage system into multiple independent allocation units distributed across different servers, allowing multiple writers to operate concurrently on different segments without interfering with each other, thus improving write performance while maintaining consistency through centralized state management
Solution Approach 2:
The patent introduces a centralized state management system that acts as an intermediary to coordinate writes across multiple allocation units. This mediator maintains the global log state and ensures consistency without requiring complex peer-to-peer coordination protocols between writers
2Reliability
If replication is implemented for fault tolerance, then reliability is improved, but storage efficiency and performance degrade
Solution Approach 1:
The patent implements selective replication where only critical state information is replicated across servers, rather than full data replication. This partial replication approach provides fault tolerance for metadata while maintaining high storage efficiency for the actual log data
Solution Approach 2:
The patent creates lightweight copies of state information rather than full data replicas. Each server maintains a copy of the global state namespace, enabling fast recovery without the overhead of replicating entire allocation units across multiple servers
3Device complexity
If centralized metadata servers are used for state management, then coordination is simplified, but network overhead and latency increase
Solution Approach 1:
The patent segments the state management functionality by distributing state information across multiple servers while maintaining a unified namespace. This allows state access without centralized metadata servers, reducing network overhead while keeping coordination simple through standardized state access patterns
4Productivity
If large IO sizes are used in existing WAL, then network efficiency is improved, but latency and granularity are worsened
Solution Approach 1:
The patent implements dynamic allocation units with flexible size and structure that can be optimized for different workloads. Allocation units can be sized and configured based on specific performance requirements, allowing the system to adapt between small low-latency operations and large high-throughput operations
Data Source
AI summary
A storage array for centralized logging is provided. The storage array for centralized logging includes one or more servers and a near memory processing NMP module. The one or more servers are configured to host one or more allocation units allocated to a logical volume. The NMP module is configured for one or more allocation units, and the NMP module is configured to record and update a state of a respective allocation unit in a reserved area of the respective allocation unit. The state includes a virtual offset of the respective allocation unit within a virtual log and a range of valid data in the respective allocation unit, and the NMP module is configured to return the allocation unit state in response to a command from one of one or more client devices.