Data Confidence Fabric Pipeline Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional systems for managing data from multiple devices require manual configuration and updating, which is inefficient and lacks automation in adapting to changing device configurations and workloads.
Innovation Solution
A method and system that utilize a data confidence fabric (DCF) configuration file to automatically manage local data systems, allowing for the registration of DCF pipelines, data processing, and resource optimization across sub-DCFs, enabling selective data access and resource charging based on workload-specific requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If manual configuration and updating is used for data management systems, then system simplicity is maintained, but automation and efficiency are lost
Solution Approach 1:
The system enables self-service automation through DCF configuration files that automatically manage data pipelines, device configurations, and workload routing without requiring manual intervention. The local data manager autonomously processes these configurations to establish and maintain data management infrastructure.
Solution Approach 2:
DCF configuration files are prepared in advance with predefined data pipeline specifications, device configurations, and workload parameters. These pre-configured files enable the system to automatically provision and manage data infrastructure before actual data processing begins, eliminating the need for manual setup during operation.
2Adaptability or versatility
If a single unified data management approach is used, then system simplicity is maintained, but workload-specific optimization is lost
Solution Approach 1:
The system segments data management into multiple independent DCF pipelines, each tailored to specific workload requirements. Configuration files define distinct pipelines for different data types, processing needs, and destination targets, allowing each workload to be optimized independently while maintaining overall system organization.
Solution Approach 2:
Each DCF pipeline is configured with local quality specifications appropriate to its specific workload, including device selection criteria, data processing parameters, and storage configurations. This enables each pipeline to be optimized for its particular purpose while the overall system manages multiple specialized pipelines through a unified configuration framework.
3Quantity of substance
If data from multiple devices is collected, then data comprehensiveness is improved, but manual configuration requirements increase
Solution Approach 1:
The system automatically discovers and configures multiple data devices through the local data manager, which reads DCF configuration files to identify required devices, their capabilities, and appropriate data pipelines. This self-service mechanism eliminates manual configuration effort while enabling comprehensive data collection from numerous devices.
Solution Approach 2:
The DCF configuration framework provides universal functionality for managing data from multiple diverse devices through a single standardized configuration approach. The system can handle various device types, data formats, and processing requirements using the same configuration file structure and pipeline registration mechanism.
Data Source
AI summary
In general, in one aspect, the invention relates to a method for managing data. The method includes obtaining, by a local data manager, a first data confidence fabric (DCF) configuration file, where the first DCF configuration file is associated with a first DCF pipeline and a first workload. The method further includes registering the first DCF pipeline in a DCF pipeline registry, obtaining a data set, identifying the first DCF pipeline using the DCF pipeline registry, and processing the data set based on the first DCF pipeline to obtain first processed data.