MapReduce EDI Document Processing via Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for processing large documents, such as EDI documents, are time-consuming and resource-intensive, requiring specialized high-end hardware and complex messaging middleware, and often involve sequential processing that is not efficiently adaptable to existing programming logic.
Innovation Solution
A method that maps large EDI documents into smaller intermediate documents, processes them in parallel using commodity hardware, and reduces the results in a two-step process across different machines to achieve faster processing and reduced resource consumption, allowing for flexible adaptation to existing programming logic with minimal changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If sequential processing is used to process large documents, then processing logic is simple, but processing time is very long and it requires special high-end hardware
Solution Approach 1:
The patent divides large documents into multiple smaller segments or chunks that can be processed in parallel. The FLOW service is executed multiple times on different segments simultaneously, transforming a single sequential processing task into multiple parallel processing tasks, thereby reducing overall processing time without requiring complex hardware modifications.
2Productivity
If a broker distributes large documents to Integration Server instances for parallel processing, then processing speed improves, but network bandwidth requirements and resource consumption increase significantly
Solution Approach 1:
The patent makes the FLOW service itself multi-functional by enabling it to handle both sequential and parallel processing modes. The same FLOW service logic is reused across multiple parallel executions on different document segments, eliminating the need for separate broker and messaging middleware infrastructure, thereby reducing network bandwidth consumption while maintaining improved processing speed.
3Productivity
If MapReduce programming model is employed on existing applications, then parallel processing capability is achieved, but programming logic requires in-depth adaptation
Solution Approach 1:
The patent copies the existing FLOW service logic and executes it multiple times in parallel on different segments of the document. Instead of adapting the programming logic to fit MapReduce, the approach copies the original service and runs instances in parallel, preserving the existing programming logic while achieving parallel processing capabilities.
4Power
If specialized high-end hardware is used for processing large documents, then processing power is sufficient, but hardware maintenance cost and complexity increase
Solution Approach 1:
The patent segments the processing task into multiple parallel executions of the same FLOW service on standard hardware. By dividing the large document into smaller segments and processing them concurrently, the system achieves the processing power of high-end hardware using multiple standard servers, thereby reducing hardware maintenance complexity and cost.
Data Source
AI summary
The present invention refers to a method for MapReducing the processing of an Electronic Data Interchange (EDI) document (1, the method comprising the following steps:a. mapping the EDI document (1) into a plurality of intermediate documents (10, 11);b. processing the intermediate documents (10, 11) to produce a plurality of intermediate results (20-23);c. reducing the plurality of intermediate results (20-23) to produce a plurality of reduced intermediate results (30, 31); andd. reducing the reduced intermediate results (30, 31) to produce a final result (2) representing the processed EDI document (1).