Decentralized Data Provenance Tracking in Edge Microservices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In edge systems, determining the provenance and lineage of data from a large number of sources processed through microservices becomes increasingly challenging, especially in compliance with data governance requirements such as GDPR.
Innovation Solution
A decentralized approach where a device executing a portion of a distributed application extracts label information from sensor data, performs data transformations, and combines label information to track data provenance and lineage across the network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is collected from multiple sources and processed through microservices in edge systems, then data processing capability and versatility are improved, but tracking data provenance and lineage becomes increasingly challenging
Solution Approach 1:
The patent segments the provenance tracking system into distributed components, where each microservice and data source maintains its own metadata and provenance information independently. This allows the system to handle multiple data sources and processing operations without creating a monolithic tracking complexity, as each segment manages its own tracking responsibilities.
Solution Approach 2:
The patent introduces metadata as an intermediary layer that mediates between diverse data sources and the provenance tracking requirements. This metadata structure serves as a standardized interface that captures provenance information from various sources without requiring complex integration logic, thereby simplifying the overall tracking mechanism while maintaining versatility.
2Reliability
If data governance requirements are implemented to ensure compliance, then data security and reliability are improved, but system complexity and operational overhead increase
Solution Approach 1:
The patent implements preliminary action by embedding provenance tracking and data governance metadata generation at the point of data creation and processing, rather than adding compliance checks as separate post-processing steps. This approach ensures data governance requirements are met from the outset, reducing the need for complex remediation systems later.
Solution Approach 2:
The system enables self-service by allowing each microservice and data source to automatically generate and manage its own provenance metadata and compliance information without requiring centralized control or manual intervention. This decentralized approach maintains reliability through consistent governance while reducing operational overhead and system complexity.
3Ease of operation
If centralized systems are used for data processing, then provenance tracking is relatively trivial and simple, but scalability and distributed processing capability are limited
Solution Approach 1:
The patent creates a universal metadata structure that serves multiple functions simultaneously: it tracks provenance information, enables data lineage tracking, supports data governance compliance, and facilitates distributed processing coordination. This multi-functional approach maintains the simplicity of centralized tracking while enabling the system to operate effectively across distributed microservices.
Data Source
AI summary
In one embodiment, a device executing a first portion of a distributed application extracts label information from sensor data sent to the device by a sensor that indicates the sensor as a source of the sensor data and one or more data governance policies applicable to the sensor data. The device performs, based on the label information, a first data transformation of the distributed application on the sensor data using stored data, to form transformed data. The device forms combined label information for the transformed data by appending the label information with additional label information associated with the stored data and adding an indication of the first data transformation. The device provides the transformed data and combined label information to a remote device executing another portion of the distributed application.