Suspicious Activity Detection Pipeline Governance Through Run Traceability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems lack transparency in tracing back the results of suspicious activity detection to the input data, making it difficult to explain unusual patterns and inconsistencies, and there is a need for improved governance of the data analytics pipeline.
Innovation Solution
A system and method for capturing and storing data and metadata at each stage of the analytics pipeline, allowing for traceability and visualization of the data processing steps, including the ability to compare different pipeline runs and generate reports on differences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data analytics pipeline processes are run on all data to detect suspicious activity, then detection capability is improved, but transparency and traceability of results to source data deteriorates
Solution Approach 1:
The system performs preliminary actions by capturing and storing metadata about data processing steps before the actual analytics pipeline execution. This includes recording which data sources were used, what transformations were applied, and which models were invoked, enabling later traceability without reprocessing the data.
Solution Approach 2:
The system introduces an intermediary metadata layer that connects source data to detection results. This metadata acts as a mediator that tracks the provenance of data through various processing stages, allowing results to be traced back to original sources without requiring direct access to the processed data itself.
2Loss of information
If detailed data processing steps are captured for traceability, then transparency is improved, but system complexity increases
Solution Approach 1:
The governance system is segmented into distinct functional components: metadata capture modules that record processing steps, metadata storage that preserves provenance information, and metadata retrieval that enables traceability queries. This segmentation allows each component to handle specific aspects of traceability independently.
Solution Approach 2:
Instead of duplicating or reprocessing actual data, the system creates and manages copies of metadata that describe data processing operations. This metadata copying approach provides full traceability information without the complexity of managing actual data copies or re-executing processing pipelines.
3Loss of information
If metadata is captured at each pipeline stage for traceability, then traceability is improved, but data processing time increases
Solution Approach 1:
Metadata capture operations are performed as preliminary actions during the natural flow of data processing, recording essential provenance information at each stage without pausing or blocking the main analytics pipeline execution. This allows traceability information to be collected concurrently with data processing.
Solution Approach 2:
The system captures only the essential metadata needed for traceability rather than every possible detail of data processing. This partial action approach focuses on recording key provenance information (data sources, transformations, models) while omitting redundant details, achieving adequate traceability with minimal overhead.
Data Source
AI summary
A computer-implemented system, platform, computer program product, tool, and/or method for capturing data and metadata for providing traceability of data analytic pipeline runs that includes: triggering a first pipeline run; obtaining a pipeline configuration for the first pipeline run, including one or more tasks; capturing and storing the pipeline configuration and starting inputs for the first pipeline run; executing the first pipeline run; capturing, before each task in the first pipeline run is executed, input data and input metadata for each task; capturing, after each task in the first pipeline run is executed, output data and output metadata for each task; and capturing the alerts for the first pipeline run.