Schema-Driven Data Processing with Compatibility Checks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increased collection, storage, and processing of data by organizations lead to resource overhead and complexity due to the lack of centralized tracking and management of datasets and data pipelines, resulting in duplication and inefficient resource consumption.
Innovation Solution
A schema-driven distributed data processing system that includes a controller for centralized tracking and management of data processors and schemas, performing compatibility checks and propagating compatible schema changes while preventing incompatible ones, thereby improving discoverability and reusability of data and data processors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data processors and schemas are managed in a decentralized manner without centralized tracking, then teams can independently develop and modify data pipelines, but this leads to duplication of datasets and pipelines across teams, increasing resource overhead and consumption
Solution Approach 1:
The system implements automated feedback mechanisms where schema changes are detected and compatibility checks are automatically performed across all data processors. When a schema change is made, the system notifies affected processors and coordinates updates, ensuring that independent development does not lead to duplication or conflicts.
Solution Approach 2:
A centralized schema registry acts as an intermediary between distributed data processors and schemas. This intermediary tracks all schemas and their relationships, enabling teams to independently develop while the registry prevents duplication by providing discoverability and coordination mechanisms.
2Productivity
If schema changes are made without compatibility checks, then data processors can be updated quickly and independently, but incompatible schema changes cause data processors to fail when processing data
Solution Approach 1:
The system performs compatibility checks as a preliminary action before deploying schema changes to production. The compatibility check framework evaluates proposed schema changes against existing data processor requirements, identifying potential incompatibilities before they cause failures.
Solution Approach 2:
For compatible schema changes, the system enables rapid propagation across all data processors without manual intervention. Once compatibility is confirmed, changes are automatically propagated and deployed, allowing quick updates while maintaining reliability through the preliminary compatibility validation.
3Reliability
If compatibility checks are performed for all schema changes, then data processing reliability is maintained, but the complexity of schema management increases
Solution Approach 1:
The compatibility check framework operates autonomously without requiring manual intervention. The system automatically detects schema changes, performs compatibility evaluations, identifies affected data processors, and coordinates updates. This self-service approach maintains reliability while minimizing the operational complexity for users.
4Stability of the object's composition
If schema changes are propagated automatically to all data processors, then consistency across the system is improved, but incompatible changes can cause widespread failures
Solution Approach 1:
The system applies preliminary anti-action by performing compatibility checks before automatic propagation. The compatibility check framework identifies potential harmful effects of schema changes on specific data processors and prevents propagation to incompatible processors, while still maintaining consistency across all compatible processors.
Data Source
AI summary
One embodiment of the present invention sets forth a technique for performing schema-driven data processing. The technique includes detecting a first change to a first producer schema for a first dataset produced by a first data processor. The technique also includes performing a compatibility check between the first change and a first consumer schema associated with processing of the first dataset by a second data processor, wherein the first consumer schema includes a set of fields required by the second data processor. The technique further includes modifying an operation of the second data processor based on a result of the compatibility check.