Activity Schema Discovery and Change Detection for Data Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Online services face challenges in managing diverse data formats of activity information generated by different producer applications, leading to inefficiencies and potential failures in data pipeline operations due to uncommunicated format changes, which can break routing rules and affect consumer applications' functionality.
Innovation Solution
Automatically discovering schemas in activity information and detecting changes, with notifications sent to responsible personnel when schema changes occur, ensuring that consumer applications are updated to expect the new formats and maintaining data pipeline integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If producer applications generate activity information in different data formats, then flexibility and ease of manufacture are improved, but data pipeline reliability and consumer application functionality deteriorate due to uncommunicated format changes
Solution Approach 1:
The system implements automatic schema validation where consumer applications register their expected data formats with the data pipeline. The data pipeline then automatically validates incoming activity information against these registered schemas and sends notifications when format changes are detected, creating a feedback loop that maintains reliability while allowing format flexibility
Solution Approach 2:
Consumer applications perform preliminary registration of their expected data formats with the data pipeline before receiving activity information. This advance preparation allows the system to establish validation rules upfront, ensuring that future format changes are detected and communicated before they cause pipeline failures
2Adaptability or versatility
If data formats of activity information change over time, then adaptability is improved, but routing rule functionality and consumer application operation deteriorate due to uncommunicated changes
Solution Approach 1:
The system uses automatic schema validation to create a feedback mechanism that detects format changes in activity information and notifies relevant personnel. This allows the system to adapt to format changes while maintaining routing rule functionality through timely awareness of changes
Solution Approach 2:
The data pipeline acts as an intermediary between producer applications and consumer applications, validating activity information formats against registered schemas and detecting changes before they reach consumer applications. This intermediary function protects routing rules from breaking due to uncommunicated format changes
3Device complexity
If manual communication of format changes between software developers is used, then device complexity is reduced, but information loss and operational inefficiency increase
Solution Approach 1:
The system implements self-service automatic schema validation where the data pipeline autonomously validates activity information formats, detects changes, and sends notifications without requiring manual intervention from software developers. This eliminates information loss while keeping the system relatively simple
Solution Approach 2:
The automatic notification system provides feedback to software developers about format changes in activity information, ensuring that information is not lost and developers are timely aware of changes that may affect consumer applications
Data Source
AI summary
Techniques for activity information schema discovery, schema change detection, and notification. In one embodiment, for example, a computer-implemented method for activity information schema discovery and schema change detection and notification comprises: analyzing a first set of related activity messages obtained during a first sample period; determining first schema counters for uniquely named properties identified in the first set of messages based on the analyzing of the first set of activity messages; after the first sample period, inferring a first schema from the first schema counters; analyzing a second set of related activity messages obtained during a second sample period; determining second schema counters for uniquely named properties identified in the second set of messages based on the analyzing of the second set of activity messages; after the second sample period, inferring a second schema from the second schema counters; comparing the first schema and the second schema for any differences.