Meter Data Validation Parallel Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Utility service providers face challenges in processing large volumes of raw data from utility meters and sensors, leading to potential inaccuracies, incompleteness, and latency in converting raw data into quality information for analysis and billing.
Innovation Solution
A meter data management system that performs parallel validation, estimation, and work item assignment of utility data, using a rule-based algorithm to direct data blocks to appropriate pipelines for processing, pre-loading rules for different channels, and employing a Universal Calculation Engine to apply rules, thereby ensuring rapid availability of accurate data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If batch processing is used to validate utility data, then data accuracy is improved, but processing time and latency increase
Solution Approach 1:
The patent segments the validation process into multiple independent pipelines (validation pipeline, estimation pipeline, work item pipeline) that can process different data blocks simultaneously. Each pipeline handles specific validation rules and data types, enabling parallel processing while maintaining comprehensive validation accuracy through rule-based algorithms in each segment.
Solution Approach 2:
The system pre-loads validation rules, estimation algorithms, and work item definitions into memory before data processing begins. This preliminary preparation eliminates runtime loading delays and enables immediate parallel validation across multiple pipelines, reducing overall processing time while maintaining thorough validation standards.
2Quantity of substance
If frequent data collection is performed, then data completeness is improved, but the potential for bad data and confusion increases
Solution Approach 1:
The patent implements feedback mechanisms where validation results from the validation pipeline feed into the estimation pipeline for data that fails validation. The system continuously monitors data quality metrics and adjusts processing routes dynamically, using feedback loops to maintain high data reliability even as data volume increases through frequent collection.
Solution Approach 2:
The system introduces an intermediary validation layer between data collection and final processing. The validation pipeline acts as a mediator that filters and quality-checks data before it proceeds to estimation or work item assignment, preventing bad data from propagating through the system while maintaining complete data capture.
3Productivity
If multiple processing pipelines are used in parallel, then processing speed is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal data block structure and common rule-based algorithm framework that serves all three pipelines (validation, estimation, work items). This universal foundation reduces complexity by eliminating the need for separate processing logic for each pipeline, while still enabling parallel execution through standardized interfaces and shared memory structures.
4Loss of time
If data is validated and processed quickly, then latency is reduced, but thorough validation may be compromised
Solution Approach 1:
The system pre-loads all validation rules, estimation algorithms, and reference data into memory before processing begins. This preliminary preparation ensures that comprehensive validation can be performed immediately upon data arrival without runtime loading delays, maintaining both low latency and thorough validation through pre-computed validation criteria.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A meter data management (MDM) system processes imported blocks of utility data collected from a plurality of utility meters, sensors, and/or control devices by using independent parallel pipelines associated with differing processing requirements of the data blocks. The MDM system determines processing requirements for each of the imported data blocks, selects one of the pipelines that matches the determined processing requirements for each of the imported data blocks, and directs the data blocks to the selected one of the pipelines for processing. The pipelines may include a validation pipeline for validation processing, an estimation pipeline for estimation processing and a work item pipeline for work item processing.