Log File Extraction Module for Searchable Repository Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing test execution servers face challenges in storing log files of different formats into searchable repositories that provide full-text search engines, as these repositories are often limited to text file formats and cannot handle files in other formats effectively.
Innovation Solution
A method and system that include a listener, extraction module, and logger to receive notifications of job completion, retrieve log files, determine their format, and extract or store appropriate files into a searchable repository, enabling the storage of log files in various formats, including compressed and archive formats, by using APIs and libraries for decompression and extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a searchable repository is designed to store only text file formats, then the full-text search engine can operate efficiently, but the repository cannot handle files in other formats (such as compressed or archive formats) effectively
Solution Approach 1:
The patent introduces an intermediary component (extraction module or gateway) that sits between the log file source and the searchable repository. This intermediary detects the file format, performs appropriate extraction or conversion operations, and transforms diverse formats into a standardized text format that the search engine can handle. This resolves the contradiction by adding format handling capability without requiring the search engine itself to become more complex.
Solution Approach 2:
The system is segmented into distinct functional modules: a format detection component, an extraction/conversion component, and the searchable repository component. Each module has a specific responsibility, allowing the system to handle multiple formats while maintaining a simple search engine core. The segmentation enables the system to process different formats through specialized handlers without increasing the overall system complexity significantly.
2Adaptability or versatility
If the repository stores log files in various formats including compressed and archive formats, then the versatility of the repository increases, but the complexity of processing and extracting data from these files increases
Solution Approach 1:
The system employs dynamic format detection and adaptive processing. When a log file is received, the system dynamically determines its format and selects the appropriate extraction or conversion method. This dynamic approach allows the system to support multiple formats without requiring complex static processing logic for each format, as the processing path is determined at runtime based on the actual file characteristics.
Solution Approach 2:
The system changes the state parameter of the log files by transforming them from their original formats (compressed, archive, etc.) into a standardized text format suitable for the search engine. This parameter change (format transformation) is performed by the extraction module, which applies appropriate decoding or extraction algorithms based on the detected file type, thereby enabling format versatility while keeping the search engine implementation relatively simple.
3Loss of time
If log files are extracted and processed in real-time upon job completion, then the availability of test data for search is improved, but the processing time and system resource usage increase
Solution Approach 1:
The system performs preliminary actions by setting up event listeners that monitor for job completion notifications. When a job completes, the listener immediately triggers the extraction and indexing process, ensuring that test data becomes available for search as quickly as possible. This preliminary monitoring and immediate response mechanism minimizes the delay between job completion and data availability, while the on-demand nature of the processing keeps resource usage optimized.
Solution Approach 2:
The system uses periodic polling or event-driven periodic processing where the extraction and indexing operations are triggered at specific intervals or events (job completion). This periodic action pattern allows the system to balance between making data available quickly and avoiding continuous resource consumption, as processing occurs only when necessary rather than continuously.
Data Source
AI summary
Systems and methods are provided for storing test results. An example method includes receiving a notification that a job has completed execution in a test execution server. The notification includes an identifier and a name of the job, and is sent from the test execution server. The method also includes retrieving a log file from the test execution server and determining a format of the log file. The log file includes a plurality of files storing a set of test results based on the execution of the job. The method further includes extracting, based on the format of the log file, the plurality of files from the log file. The method also includes storing one or more files of the plurality of files into a searchable repository that provides a full-text search engine.