Anomaly Detection in Workflow Iteration Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of data generated and processed in workflows makes it challenging to identify errors and inefficiencies, which are often masked by the sheer amount of data, hindering accuracy and efficiency in data generation and processing.
Innovation Solution
The implementation of machine-learning techniques to analyze data workflows, specifically by accessing and processing iteration data to identify anomaly subsets, such as tasks with long processing times or statistically different data sources, and selectively validating sparse indicators to improve data processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine-learning techniques are used to analyze iteration data and identify anomaly subsets, then measurement precision and reliability are improved, but device complexity and computational resources increase
Solution Approach 1:
The patent introduces an intermediary anomaly detection system that processes iteration data through machine-learning techniques. This intermediary layer analyzes processing times and data characteristics to identify anomalies before they propagate through the full data pipeline, improving detection precision while containing complexity within the intermediary component rather than the entire system
Solution Approach 2:
The patent segments the data processing workflow into distinct iterations with identifiable start and end points. By dividing the comprehensive data set into discrete workflow iterations and analyzing them individually, the system achieves precise anomaly detection without requiring complex analysis of the entire data volume simultaneously, thus managing computational complexity
2Reliability
If comprehensive data sets from multiple workflow iterations are analyzed, then reliability of error identification is improved, but loss of time and computational resources increase
Solution Approach 1:
The patent performs preliminary actions by establishing workflow structures with defined iterations before comprehensive analysis. Iteration data including processing times and data characteristics are collected and pre-processed during normal operations, allowing subsequent anomaly detection to rely on this pre-prepared data rather than analyzing raw comprehensive datasets from scratch, thus reducing analysis time while maintaining reliability
Solution Approach 2:
The patent replaces manual or traditional mechanical analysis methods with machine-learning-based automated analysis. The machine-learning model processes iteration data to identify anomaly subsets, substituting computational algorithms for time-consuming manual review processes and achieving reliable error identification more efficiently
3Productivity
If processing times of individual tasks are tracked and analyzed, then productivity bottlenecks are identified, but device complexity and data processing overhead increase
Solution Approach 1:
The patent implements a universal timing mechanism that serves multiple functions: it tracks processing times for individual tasks, defines workflow iteration boundaries, and provides data for anomaly detection. This multi-functional approach identifies productivity bottlenecks without requiring separate complex tracking systems, as the same infrastructure supports multiple analytical purposes
Data Source
AI summary
Methods and systems disclosed herein relate generally to data processing by applying machine learning techniques to iteration data to identify anomaly subsets of iteration data. More specifically, iteration data for individual iterations of a workflow involving a set of tasks may contain a client data set, client-associated sparse indicators and their classifications, and a set of processing times for the set of tasks performed in that iteration of the workflow. These individual iterations of the workflow may also be associated with particular data sources. Using the iteration data, anomaly subsets within the iteration data can be identified, such as data items resulting from systematic error associated with particular data sources, sets of sparse indicators to be validated or double-checked, or tasks that are associated with long processing times. The anomaly subsets can be provided in a generated communication or report in order to optimize future iterations of the workflow.