Distributed Log File Diagnostics via Segmented Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Diagnosing errors in information handling systems (IHS) with large log files is challenging due to their size, requiring extensive time and being feasible only in specific circumstances.
Innovation Solution
A distributed processing IHS system that receives and processes large log files over a network, parsing and structuring the data across multiple nodes to generate job results that identify the cause of errors, using Apache Hadoop and Cascading APIs for batch processing and data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If log files are stored locally on the IHS as rolling logs, then the log files can be generated and stored in a relatively short time span, but the size of the log files becomes very large (tens to hundreds of gigabytes) making diagnosis difficult and time-consuming
Solution Approach 1:
The patent divides the large log file into multiple smaller partitions or segments that can be distributed across multiple nodes in a cluster. Each node processes a specific partition independently, allowing parallel processing and reducing the complexity of handling monolithic large files. This segmentation enables the system to manage and analyze log data of any size by breaking it into manageable chunks.
2Measurement precision
If support technicians review log files line by line to determine error causes, then they can identify specific error events, but the process requires a large amount of time due to the size of the log files
Solution Approach 1:
The system performs preliminary actions by automatically parsing, cleaning, and structuring log data before analysis is needed. It pre-processes the log files to extract relevant information, create structured representations, and organize data by error types or components. This preliminary processing eliminates the need for manual line-by-line review while preserving the ability to identify specific error causes accurately.
Solution Approach 2:
The patent introduces an intermediary processing layer between the raw log files and the analysis process. This intermediary system automatically parses unstructured log data, extracts meaningful information, and presents it in a structured format that facilitates rapid error identification. The intermediary handles the time-consuming processing tasks, allowing technicians to focus on analysis rather than data preparation.
3Adaptability or versatility
If conventional methods are used to diagnose errors in large log files, then diagnosis is only plausible in a few specific circumstances, but the system cannot handle large-scale log file analysis effectively
Solution Approach 1:
The system is designed with universal capabilities to handle log files of any size and from any source. It can process logs from multiple IHS instances, aggregate them into a single analysis, and adapt to different log formats and structures. The multi-functional architecture allows the same system to handle various diagnostic scenarios regardless of the volume or complexity of the log data, making it applicable to diverse situations rather than being limited to specific circumstances.
Data Source
AI summary
An information handling system (IHS) diagnostic system includes a user IHS that is coupled to a network and operable to generate a log file that is associated with a user IHS error. A distributed processing IHS is coupled to the user IHS through the network and operable to receive the log file generated by the user IHS over the network. That log file is then stored on a plurality of node IHS's in the distributed processing IHS. When a job request is received from the user IHS over the network, a job associated with the job request is performed on the log file using the plurality of node IHS's to create a job result. The job result that includes at least one cause of the user IHS error is then provided to the user IHS.