Distributed Log Compaction for State Machine Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current state machine systems that rely on relational databases or consensus-based datastores for tracking state transitions require sharding and operational upkeep, which can be inefficient and complex, especially when managing distributed systems.
Innovation Solution
Implementing an automatically compacting distributed log storage system that uses built-in partitioning to manage state machine states, eliminating the need for sharding and allowing for efficient transactional writes and parallel operations, with log compaction reducing storage needs and facilitating seamless worker replacement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If relational databases or consensus-based datastores are used to store state machine states, then state transitions can be tracked, but the system requires sharding and operational upkeep which increases complexity
Solution Approach 1:
The patent extracts the state machine state data from traditional relational databases or consensus-based datastores and places it into a dedicated log store. This separation allows the log store to be optimized specifically for state tracking without the operational complexity of sharding required by general-purpose databases, thereby maintaining reliability while reducing device complexity
Solution Approach 2:
The patent introduces a log store as an intermediary component between the state machine system and traditional database infrastructure. This log store acts as a mediator that simplifies data access and management for state transitions, eliminating the need for complex sharding operations while maintaining data integrity and reliability
2Adaptability or versatility
If distributed systems manage state transitions without built-in partitioning, then operational flexibility is maintained, but throughput and parallel operation efficiency are reduced
Solution Approach 1:
The patent implements built-in partitioning within the log store that automatically segments state data by state machine key. This segmentation enables parallel write operations to different partitions simultaneously, dramatically increasing throughput while maintaining operational flexibility because the partitioning is transparent to the application layer
3Loss of information
If full logs are retained for worker replacement, then complete state history is preserved, but storage requirements increase significantly
Solution Approach 1:
The patent performs log compaction as a preliminary action that consolidates state data before it is permanently stored. By compacting logs before storage, the system eliminates redundant intermediate states while preserving the essential state transition information needed for worker replacement, thereby reducing storage requirements without losing critical state history
Solution Approach 2:
The patent implements a selective discarding mechanism where intermediate and redundant log entries are discarded through compaction, while essential state information is recovered and retained. This allows the system to maintain only the necessary state history for worker replacement rather than preserving every intermediate log entry, significantly reducing storage requirements
Data Source
AI summary
Systems and methods are provided for receiving, at a server, a workflow definition and generating a unique key for the received workflow definition. A distributed log storage may store the internal workflow schema having the not-started states to a state topic of the distributed log storage using the generated unique key, where the state topic includes the states of the internal workflow schema. One or more workers at the server may perform at least one operation based on a received message. The state may be updated at the distributed log storage based on the performed at least one operation. The state topic of the internal workflow schema for the generated key may be compacted based on the updated state, where the compacting reduces the states of the internal workflow schema to the current states, without intermediary states.