Mainframe Log Data Extraction for Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mainframe log records, such as those generated by IBM's IMS subsystem, are lengthy and cryptic, making it difficult to access and analyze specific data, which limits their use in analytics due to their system-friendly but not user-friendly format, and the overhead of existing methods for accessing these records can cause delays and resource inefficiencies.
Innovation Solution
A two-phase extraction method that minimizes disruption to the mainframe subsystem by extracting only relevant log record fields and types, using a user interface to select and filter data, and processing these fields outside the mainframe control region to reduce data transmission and processing costs, with optional use of an intercept method to avoid mode switching overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all log record fields are extracted and transmitted for analysis, then complete data availability for analytics is improved, but data transmission volume and processing costs increase significantly
Solution Approach 1:
The patent extracts only the specific fields of interest from log records rather than transmitting complete log data. The field extraction module identifies and extracts only those fields that are relevant to analytics requirements, filtering out unnecessary data to reduce transmission volume and processing costs while maintaining data availability for analytical purposes.
Solution Approach 2:
The patent segments the log data processing into distinct phases: field identification, field extraction, and data transmission. By dividing the processing workflow, the system can selectively handle only the necessary portions of log records, avoiding the overhead of processing and transmitting entire log entries while ensuring that analytical needs are met.
2Loss of information
If comprehensive field extraction is performed within the mainframe control region, then data access completeness is improved, but overhead to the mainframe subsystem increases
Solution Approach 1:
The patent removes the heavy extraction processing from the mainframe control region by implementing extraction in a separate address space. Only essential field identification occurs within the control region, while the actual extraction and transmission of selected fields happens outside, minimizing overhead to the mainframe subsystem while maintaining complete data access capability.
3Loss of information
If detailed filtering and field selection is implemented, then data relevance for analytics is improved, but system complexity increases
Solution Approach 1:
The patent performs preliminary field identification and extraction configuration before actual log processing begins. The system pre-configures which fields are of interest and establishes extraction rules in advance, allowing the main extraction process to simply follow predefined criteria rather than making complex decisions during real-time processing, thus reducing operational complexity while maintaining high data relevance.
Data Source
AI summary
Systems and methods are provided that extract information from IMS log records to reduce the amount of data transmitted and input to an analysis engine. An example method includes writing IMS log records matching log types identified in an extraction list to a file within an IMS control region of a mainframe computer and outside of the IMS control region, and reading records from the file. For each record read, the method may also include extracting fields of interest from the log record based on fields of interest associated with the log type of the log record in the extraction list and a data-to-field mapping for the log type, converting the fields of interest to a predetermined format based on the data-to-field mapping, and writing at least one field of interest to an output file. The method may also include transmitting the output file to an analytics engine for processing.