Data Product Version Checks for Backward-Compatible Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data engineering methodologies fail to address the challenges of facilitating the development of data products that are reliable, robust and backwards-compatible, such that a data product can be iteratively refined and improved without impairing the functioning of data consumers relying on the data product.
Innovation Solution
A computer-implemented method for developing data products that are reliable, robust and backwards-compatible, such that a data product can be iteratively refined and improved without impairing the functioning of data consumers relying on the data product.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data product modifications are made to add new features and improve functionality, then the data product can be iteratively refined and improved, but data consumers relying on the existing data product may be impaired or caused to fail
Solution Approach 1:
The system performs preliminary compatibility checking before deploying modifications to data products. The compatibility determination module analyzes proposed changes against the data product manifest and service level objectives to predict potential impacts on data consumers, allowing developers to catch breaking changes before they reach production
Solution Approach 2:
The system implements feedback mechanisms through service level indicators and objectives that monitor data product performance and consumer impact. When modifications are proposed, the system provides feedback on compatibility status, alerting developers to potential breaking changes and allowing them to adjust their changes accordingly
2Reliability
If data consumers verify and sanitize data to ensure reliability, then data accuracy and integrity are improved, but computational resources such as processor time, memory pages and bandwidth are unnecessarily consumed
Solution Approach 1:
The system implements self-service through automated compatibility checking and service level monitoring. Instead of requiring data consumers to manually verify and sanitize data, the system automatically checks compatibility against the data product manifest and service level objectives, providing guarantees without requiring additional computational resources from consumers
3Reliability
If data consumers build in verification functionality to double-check data, then data trustworthiness is improved, but developer workload and system complexity increase
Solution Approach 1:
The system introduces an intermediary layer consisting of the data product manifest and service level objectives that mediates between data producers and consumers. This intermediary contains the verification logic and compatibility rules, eliminating the need for consumers to implement their own verification functionality while maintaining data trustworthiness
Data Source
Figure 1
Figure 2
Figure 3a~3b
AI summary
This disclosure relates to methods, devices, and computer-readable media for use in developing data products. One such method comprises receiving an existing build of the data product, identifying a data product version associated with the existing build, receiving a user-specified modification for the data product, in response to a user input, automatically determining a compatibility result for the modification with the identified data product version, based on the existing build of the data product, and in response to the determined compatibility result being a negative compatibility result, triggering a failure event in relation to the identified data product version.