Distributed Execution State Machine for Failure Localization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale distributed systems, such as public cloud platforms, face challenges in managing failures due to their complexity, with existing techniques like debugging, profiling, and log analysis providing limited insights into service provisioning and requiring new methods for effective failure handling and root cause analysis.
Innovation Solution
A tracing system utilizing distributed execution state machines (DESMs) to store and reconstruct service provisioning traces, linking logging information to states, and providing a higher abstraction level for failure localization and resolution, integrating developer and operator views through an independent tracing language and visualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If distributed systems grow in size and complexity to handle large-scale services, then service coverage and capability improve, but failure management difficulty and operational complexity worsen
Solution Approach 1:
The patent segments the complex distributed system into multiple independent services, each with its own state machine model. This allows failure management to be broken down into service-specific traces rather than requiring analysis of the entire system, reducing operational complexity while maintaining the ability to handle large-scale services.
Solution Approach 2:
The patent introduces distributed execution state machines (DESMs) as an intermediary layer between the complex distributed system and failure analysis tools. DESMs provide a standardized, simplified representation of service execution that mediates between the underlying system complexity and the need for manageable failure traces.
2Ease of manufacture
If traditional logging and debugging techniques are used to manage failures, then implementation simplicity is maintained, but failure analysis effectiveness and root cause identification capability deteriorate
Solution Approach 1:
The patent changes the fundamental parameter of how system state is recorded - instead of using traditional log messages with free-text descriptions, it uses structured state machine states and transitions. This structured representation dramatically improves failure analysis effectiveness while the automated trace reconstruction process keeps implementation complexity manageable.
Solution Approach 2:
The patent creates simplified copies of the actual system execution through DESM traces. Rather than analyzing the full complexity of distributed system logs, operators work with reconstructed state machine traces that copy only the essential execution path information needed for failure analysis, improving effectiveness without requiring complex implementation.
3Ease of operation
If developers use printf instructions and log facilities to record system status, then implementation ease is maintained, but operator understanding capability and failure localization capability worsen
Solution Approach 1:
The patent makes the logging mechanism universal by having all services adopt the same DESM-based state recording approach. This unified standard serves multiple functions simultaneously: it provides implementation ease through standardized logging calls, while also enabling operator understanding through consistent, structured trace representation that can be automatically reconstructed and analyzed.
Solution Approach 2:
The patent introduces DESMs as an intermediary that transforms developer-friendly log statements into operator-friendly structured traces. The state machine model acts as a mediator that preserves implementation simplicity for developers while enriching the information available to operators through automated trace reconstruction and standardized state representation.
4Reliability
If environment replication is attempted in complex distributed systems, then failure reproduction capability may improve, but time consumption and resource requirements worsen
Solution Approach 1:
The patent creates virtual copies of system execution states through DESM traces rather than replicating actual system environments. This allows failure reproduction by replaying recorded state transitions without needing to set up complex environment copies, dramatically reducing time consumption while maintaining the ability to reproduce and analyze failures.
Solution Approach 2:
The patent performs preliminary recording of execution states during normal system operation. By capturing DESM states and transitions as they occur, the system prepares failure analysis data in advance, eliminating the need for time-consuming environment setup and failure reproduction attempts later.
Data Source
AI summary
A tracing system for managing failures of a service provided by a distributed system, the tracing system comprising a state machine store configured to store a distributed execution state machine (DESM), wherein states of the DESM correspond to predetermined milestones of the service and transitions of the DESM correspond to one or more instructions of the service; a logging unit configured to record events of the distributed system in a log store, wherein an event indicates a state and/or a transition of the DESM; a process reconstruction subsystem configured to reconstruct a trace of the DESM from the events stored in the log store; and a linking unit configured to link logging information of the distributed system to the states.