Data Pipeline Execution with In-Memory Intermediate Results
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data pipeline systems are difficult to manage and develop, especially for users without specialist expertise, and they often result in unstable systems that fail to meet their intended purposes, hinder result display, and prioritize data integrity over performance.
Innovation Solution
A pipeline development system that allows users to specify data and transforms using an intuitive interface, provides specialized APIs, and offers transform templates, along with an enhanced implementation that stores intermediary results in temporary memory to reduce resource usage and speed up execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data pipeline systems use traditional ad-hoc collections of scripts and programs, then data processing capability is provided, but system complexity and difficulty of management increase
Solution Approach 1:
The patent segments the data pipeline system into distinct modular components including data source connectors, transformation modules, and data sink connectors. Each component can be independently developed, configured, and managed through a visual interface, reducing overall system complexity while maintaining processing capability.
Solution Approach 2:
The patent introduces a visual pipeline editor and configuration interface as an intermediary layer between users and the underlying complex script-based processing system. This mediator allows users to construct and manage pipelines through graphical elements without directly handling the complexity of individual scripts and programs.
2Reliability
If data pipeline systems prioritize data integrity through traditional methods, then data reliability is maintained, but execution performance and speed decrease
Solution Approach 1:
The patent implements dynamic execution modes that can adjust between strict data integrity verification and performance-optimized processing based on pipeline configuration and data characteristics. The system can dynamically select processing strategies for different pipeline stages, maintaining reliability where critical while improving overall execution performance.
Solution Approach 2:
The patent allows modification of processing parameters such as batch size, parallelism level, and verification depth to balance data integrity and performance. Users can configure pipeline parameters to optimize for either reliability or speed depending on specific requirements, enabling flexible trade-offs between these conflicting objectives.
3Productivity
If data pipeline systems process large amounts of data through multiple transformation steps, then comprehensive data processing is achieved, but resource usage increases
Solution Approach 1:
The patent implements preliminary actions such as data sampling, schema validation, and transformation pre-compilation during pipeline design and configuration phases. These preliminary actions prepare the processing logic in advance, reducing the computational resources required during actual data execution and enabling more efficient processing of large datasets.
Solution Approach 2:
The patent enables continuous data processing through stream-based transformations and in-memory processing techniques that avoid unnecessary data persistence to disk between transformation steps. By maintaining data in memory and processing it continuously through the pipeline, the system reduces I/O operations and resource consumption while maintaining comprehensive processing capability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method performed by one or more processors comprises displaying code, receiving user selection of a portion of code, determining one or more settable data items, generating a template, displaying the template, receiving a user input value for the settable data items by the template, and executing the code with each of the settable data items set to the received user input value. A data processing pipeline is configured to pass a data item to a first transformer to provide first transformed data, store the first transformed data in a temporary memory, write the first transformed data to the data storage system, and pass the transformed data from the temporary memory to a second transformer.