Data Extractor Segmentation for Pipeline Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data extractor systems in data pipeline systems are unreliable due to corruption or non-compliance with specifications, leading to critical malfunctioning and frequent software bugs, especially when handling new data item formats, which can cause cascading failures in the pipeline.
Innovation Solution
An extractor management program that allocates data items to processes containing extractors, monitors their performance, and preemptively kills or reattempts extraction with alternative extractors to handle malfunctions, while managing system resources and segregating prone-to-failure extractors to prevent system-wide failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If extractors are reused for similar extraction events, then productivity is improved, but reliability deteriorates due to corruption or non-compliance with specifications
Solution Approach 1:
The system segments extractors into different types (prone-to-failure extractors vs. stable extractors) and routes data items to appropriate extractor types based on format characteristics. This segmentation allows the system to maintain high productivity by reusing stable extractors while isolating prone-to-failure extractors to handle specific cases, thereby resolving the contradiction between productivity and reliability.
Solution Approach 2:
The system changes the parameter of extractor selection by introducing format-based routing rules that determine which extractor type to use for each data item. By changing from a uniform extractor reuse strategy to a differentiated strategy based on data format parameters, the system achieves both high productivity (through reuse of stable extractors) and high reliability (by avoiding corruption-prone extractors for problematic formats).
2Reliability
If prone-to-failure extractors are isolated to handle specific data formats, then reliability is improved, but device complexity increases
Solution Approach 1:
The format-based routing mechanism serves multiple functions: it identifies data formats, selects appropriate extractor types, and manages extractor allocation. This multi-functionality reduces the need for separate complex management systems for each extractor type, thereby improving reliability through systematic isolation while minimizing the increase in device complexity.
3Adaptability or versatility
If format-based routing rules are implemented, then adaptability is improved, but device complexity increases
Solution Approach 1:
The system performs preliminary classification of data items by format before extraction, using pre-defined routing rules. This preliminary action allows the system to adapt to new data formats by simply adding new routing rules without modifying the core extraction logic, thereby improving adaptability while keeping the increase in device complexity manageable through a systematic rule-based approach.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A data extracting system comprises an extractor manager; and plural extractors. Each extractor is configured to modify input data items to provide modified output data items and wherein each extractor is contained within an extractor process that is distinct from one or more processes in which the extractor manager is contained. The extractor manager is configured to perform: parsing an instruction to perform extraction, the instruction relating to one or more data items; allocating, based on the instruction and/or the one or more data items, the one or more data items to one or more of the plurality of extractors; receiving one or more modified data items from the one or more of the plurality of extractors; and outputting the one or more modified data items.