EC-Based Crash Handler for Cross-OS Freeze Debugging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Information handling systems often experience hang and freeze issues due to a lack of effective logging mechanisms for cross-operating environment interactions, making it difficult to debug and perform root cause analysis.
Innovation Solution
Implementing a native firmware-level solution that tracks and handles interactions across OS/firmware boundaries, including the use of crash handlers and EC-based monitoring to detect and remediate crashes, and employing MBOX messages and enhanced Serial Peripheral Interface (eSPI) for communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Difficulty of detecting and measuring
If traditional logging mechanisms are used for OS-device interactions, then system simplicity is maintained, but the ability to debug and perform root cause analysis on hang and freeze issues deteriorates
Solution Approach 1:
An intermediary logging infrastructure is introduced between the OS and device firmware layers. This intermediary captures and logs cross-boundary interactions, calls, and returns without disrupting the existing system architecture. The logging mechanism acts as a mediator that records diagnostic information while maintaining system simplicity, enabling effective root cause analysis of hang and freeze issues.
2Loss of information
If comprehensive logging of cross-operating environment interactions is implemented, then root cause analysis capability is improved, but system complexity increases
Solution Approach 1:
The logging infrastructure is established in advance and remains dormant until needed. Pre-boot logging is enabled before the OS loads, and the system is prepared to capture interactions across OS/firmware boundaries. This preliminary setup ensures that diagnostic information is retained without adding operational complexity during normal system execution.
Solution Approach 2:
The logging infrastructure is designed to be universal, handling multiple types of interactions (OS-to-EC, OS-to-BIOS, OS-to-NVMe firmware, OS-to-monitor firmware) through a single unified mechanism. This multi-functional approach retains comprehensive information while avoiding the complexity of implementing separate logging systems for each interaction type.
3Reliability
If firmware-level crash handling is implemented, then reliability of crash detection is improved, but ease of operation deteriorates
Solution Approach 1:
The firmware-level crash handling system operates autonomously without requiring user intervention. The EC and logging infrastructure automatically detect crashes, log relevant information, and initiate remediation processes. This self-service approach maintains high reliability while preserving ease of operation, as users experience no additional complexity in system usage.
Data Source
AI summary
Disclosed systems and methods for handling failures in an information handling system enable one or more crash handlers to communicate crash handler notifications to an EC of the information handling system. The EC is configured to perform crash operations including detecting a crash occurrence associated with either a crash handler notification from any of the one or more crash handlers or an SMM crash event. The EC may extract and store crash context information associated with the crash occurrence. The crash handler notifications may be communicated to the EC as MBOX commands via a peripheral interconnect, e.g., an enhanced serial peripheral interconnect (eSPI). Detecting a crash occurrence associated with the SMM event may include initiating an EC timer responsive to receiving an SMM entry message from an SMM handler and detecting the EC timer reaching a threshold value before the EC receives an SMM exit message.