Sharing Processing Modules Between Data Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data transformation systems do not allow for on-demand reuse of activated data transformers in new data pipelines, leading to increased activation time and resource consumption.
Innovation Solution
A method to detect and share processing modules between existing and new pipelines, connecting non-shared modules to shared ones, while considering current load and shareability to prevent processing speed drops and optimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If processing modules are created anew for each pipeline, then pipeline independence is maintained, but activation time and resource consumption increase
Solution Approach 1:
The patent implements on-demand copying of processing modules from existing pipelines to new pipelines. When a new pipeline is activated, the system detects suitable processing modules from existing pipelines and creates copies for reuse, rather than instantiating entirely new modules. This copying mechanism reduces activation time while maintaining pipeline independence through controlled sharing and separation protocols.
Solution Approach 2:
The system performs preliminary actions by pre-activating processing modules in existing pipelines that can be shared. When a new pipeline needs these modules, they are already activated and ready for immediate reuse, eliminating the need for re-activation. The system also pre-establishes sharing permissions and module metadata to enable rapid on-demand sharing.
2Reliability
If processing modules are created anew for each pipeline, then module isolation is ensured, but system resource consumption increases
Solution Approach 1:
The patent enables processing modules to serve multiple pipelines simultaneously through controlled sharing. A single activated processing module can be shared across multiple pipelines that have compatible processing requirements, allowing the module to perform multiple functions for different pipelines. This universality reduces the total number of module instances needed, thereby decreasing overall system resource consumption while maintaining appropriate isolation through sharing permissions.
Solution Approach 2:
The system merges the functionality of multiple pipeline-specific modules into shared processing modules when possible. By combining common processing operations into single shared modules that can be accessed by multiple pipelines, the system reduces redundancy and optimizes resource utilization while maintaining the logical separation and isolation needed for different pipeline operations.
3Loss of time
If processing modules are shared between pipelines, then activation time is reduced, but processing speed may drop due to load
Solution Approach 1:
The patent implements feedback mechanisms that continuously monitor the load and processing speed of shared processing modules. When a module is shared between pipelines, the system tracks its performance metrics and adjusts sharing decisions based on observed processing speeds. If a shared module's processing speed drops below acceptable thresholds, the system can dynamically adjust the sharing arrangement, such as limiting concurrent access or triggering separation, thereby preventing performance degradation while maintaining activation time benefits when conditions are favorable.
4Use of energy by moving object
If processing modules are shared between pipelines, then resource consumption is reduced, but system complexity increases
Solution Approach 1:
The patent implements self-service mechanisms where processing modules automatically manage their own sharing state and separation. Modules track their own activation status, sharing permissions, and associated pipelines, enabling automatic on-demand sharing without complex external coordination. When separation is needed, modules can autonomously initiate the separation process, reducing the complexity of external management systems while enabling resource-efficient sharing.
Data Source
AI summary
Provided is a method, wherein the system includes an existing pipeline having at least one processing module that is activated and information relating to the existing pipeline. The system operating method includes detecting a processing module to be shared that is shared with the existing pipeline, among the processing modules included in a new pipeline that is newly activated or in an activated pipeline that is already activated; and connecting a non-shared processing module that is not the processing module to be shared, among the processing modules included in the new pipeline or in the activated pipeline, to a shared processing module corresponding to the processing module to be shared in the existing pipeline.