Streaming Copybook Data Conversion with Inline XSLT Transformation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data message processing in high-volume enterprise systems, such as financial institutions, requires multiple passes through data messages and significant memory usage, leading to inefficiencies and increased complexity, especially in bidirectional and unidirectional data flows.
Innovation Solution
The system transforms data request messages from a flat file format into a streaming intermediary format using XSLT processing, allowing direct transformation into a target format without fully holding the data in memory, thereby reducing memory usage and processing complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire data message is read into memory and processed using tree building and analysis, then the data can be fully analyzed and transformed, but the memory usage increases and processing complexity increases
Solution Approach 1:
The patent segments the data processing into a streaming approach where data is processed in chunks rather than loading the entire message into memory. The system uses a streaming parser that processes data incrementally, building the transformation tree portion-by-portion as data arrives, rather than requiring the complete data message to be in memory at once. This reduces memory requirements while maintaining transformation accuracy.
Solution Approach 2:
The patent introduces an intermediary streaming format as a mediator between the input data format and the target format. This intermediary layer allows the system to transform data incrementally without needing to hold the entire original data message in memory, reducing both memory usage and processing complexity while maintaining reliable transformation.
2Productivity
If the entire data message is read into memory for processing, then the data can be fully processed, but the memory consumption increases significantly
Solution Approach 1:
The system segments the data message into streaming chunks that are processed incrementally. Instead of loading the entire data message into memory, the streaming parser processes data in smaller portions as it arrives, significantly reducing peak memory consumption while maintaining processing efficiency through continuous transformation.
Solution Approach 2:
The patent implements continuous streaming transformation where data is processed continuously as it arrives rather than in batches. This continuous processing approach eliminates the need to hold entire messages in memory while maintaining high processing throughput, improving productivity without increasing memory consumption.
3Measurement precision
If conventional data processing methods are used with multiple passes through data, then complete data analysis is achieved, but processing time increases
Solution Approach 1:
The patent performs preliminary parsing and transformation actions during the single pass through the data stream. By preparing the data structure incrementally as it arrives and building the transformation tree in the same pass, the system eliminates the need for subsequent passes through the data, reducing processing time while maintaining analysis completeness.
Solution Approach 2:
The system performs all necessary data analysis and transformation actions during the single streaming pass rather than requiring multiple sequential passes. This continuous processing approach completes data analysis in one pass through the data, significantly reducing processing time while maintaining measurement precision.
Data Source
AI summary
The present invention provides copybook flat data conversion with inline transformation. Specifically, a streaming intermediary formatted data message is generated as a result of converting/transforming a flat file format (non-XML (Extensible Markup Language) format), such as raw fixed-length field COBOL (Common Business-Oriented Language) copybook format or the like. The streaming intermediary format is not fully held in memory, but rather is directly transformed/converted, using XSLT (Extensible Stylesheet Language Transformations) processing, into a target format, such as structured XML or the like. By directly streaming the intermediary format to the XSLT process without holding the entire data message in memory, the present invention utilizes less memory and, as such, less memory is needed to be reclaimed.