Parallel Tabular Data Extraction Using Dual Iterators

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for extracting tabular data from application layer hosted logical databases and transmitting it to remote computers over the Internet are inefficient, particularly when large tables are involved, due to the necessity of serial processing and formatting, which can take months or years, and are constrained by low-speed internet connections.

Innovation Solution

The method involves separating the table iteration from formatting and compression tasks into parallel processes, using dual iterators that traverse the table from both ends simultaneously, and employing dual compression to optimize disk write and processor performance, using a combination of fixed-length and delimiter-separated text formats for efficient data transfer.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If tabular data is extracted and formatted in a single serial process, then data accuracy is maintained, but extraction time becomes extremely long (months or years for large tables)

Engineering Contradiction:

Improvedata accuracyVSAvoidextraction time

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent divides the extraction process into separate work processes: iteration work processes that traverse the table and secondary work processes that handle formatting and compression. This segmentation allows parallel execution of multiple tasks simultaneously, dramatically reducing extraction time while maintaining data accuracy through coordinated processing.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements preliminary formatting and compression by secondary work processes that prepare data packets before they are needed for transmission. This preliminary action allows the main iteration process to continue without waiting for formatting operations, reducing overall extraction time while ensuring data integrity.

Inventive Principle:

Principle #10Preliminary action

2Device complexity

If a single work process handles both iteration and formatting, then system complexity is low, but productivity is extremely low (orders of magnitude slower)

Engineering Contradiction:

Improvesystem complexityVSAvoiddata transfer rate

Core Design Contradiction:

Device complexityVSProductivity

Solution Approach 1:

The patent segments the single work process into multiple specialized work processes: iteration work processes for traversing the table and secondary work processes for formatting and compression. This segmentation enables parallel processing that increases productivity by orders of magnitude while managing complexity through clear process separation.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces intermediary data packets that carry information between iteration work processes and secondary work processes. These intermediaries enable efficient communication and coordination between parallel processes, allowing high productivity while maintaining manageable system complexity through standardized interfaces.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If data is transmitted in binary format, then transmission efficiency is high, but compatibility with web services is poor (cannot use generic SOAP or REST APIs)

Engineering Contradiction:

Improvetransmission efficiencyVSAvoidweb service compatibility

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent transforms data from binary format to text-based formats (XML, JSON, CSV) that are compatible with web services. This parameter change in data representation maintains adaptability and versatility for generic web service protocols while the underlying binary-to-text conversion is optimized through compression to minimize efficiency loss.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent creates text-based copies of binary data structures that are suitable for web service transmission. These text copies (XML, JSON, CSV representations) maintain the essential information while being compatible with generic web service protocols, enabling versatility without sacrificing transmission efficiency through compression optimization.

Inventive Principle:

Principle #26Copying

4Quantity of substance

If compression is applied to reduce data size, then transmission bandwidth is optimized, but processing time increases

Engineering Contradiction:

Improvedata sizeVSAvoidcompression time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent applies compression as a preliminary action in secondary work processes before data is transmitted. This preliminary compression reduces the size of data packets that need to be transmitted over the network, optimizing bandwidth utilization. The compression is performed in parallel by multiple secondary work processes, which minimizes the impact on overall processing time.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20250284701A1Distributing Tabular Data From A Relational Database Over The Internet To A Remote Subscriber Computer

Publication Date: 2025.09.11 BUSINESS MOBILE AG

AI summary

Methods are described herein related to extracting tabular data from a transmission server to a remote subscriber computer in a setting wherein the tabular data can only be transferred in a textual representation over an Internet connection, such as a web service, for example. An architecture based on parallel processing may speed up the extraction process by separating iteration from data processing and by introducing forward and reverse iterators for iterating a table from two ends simultaneously.