Event Stream Data Generation for Schema Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face challenges in handling data from different schemas, requiring frequent updates to data input jobs and custom data loading/translation tools, which can be slow and unsuitable for parallel computing architectures, and may result in data loss during correction, limiting analytics capabilities and data auditing.
Innovation Solution
The system generates event stream data using a subject-event data model, normalizing data from different schemas and allowing for schema-agnostic rule application, enabling the reuse of rules across multiple schemas and non-destructive correction of data errors, thus facilitating the integration and analysis of data from various sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If custom data loading/translation tools are used to handle schema changes, then data can be loaded and transformed, but execution speed is slow and parallel computing architectures cannot be utilized
Solution Approach 1:
The patent replaces custom mechanical data loading tools with a distributed computing system that uses standardized database connectors and in-memory data structures. This substitution enables parallel processing across multiple processors while maintaining schema adaptability through configurable data source interfaces and dynamic schema validation rules.
2Reliability
If data source is modified to correct errors, then data accuracy improves, but original data is lost and auditing capability is reduced
Solution Approach 1:
The patent segments the data correction process into distinct components: original data storage, correction rule application, and validated result generation. By separating correction logic from data storage and using version control mechanisms, the system maintains original data integrity while applying corrections through configurable rules, enabling both accuracy improvement and audit trail preservation.
3Adaptability or versatility
If data input jobs are reprogrammed for each schema change, then data can be processed according to new schemas, but development time and resource allocation increase
Solution Approach 1:
The patent implements a universal data input system using standardized database connectors that can interface with multiple data sources and schemas. The system employs configurable validation rules and dynamic schema mapping capabilities, allowing a single data input job to adapt to different schemas without reprogramming, thereby reducing development time while maintaining schema compatibility.
4Productivity
If rules and logic are written against a specific schema, then data processing is efficient for that schema, but the rules cannot be reused for other schemas
Solution Approach 1:
The patent transforms fixed schema-specific rules into parameterized validation logic that can adapt to different schemas. By using configurable parameters for data types, validation criteria, and processing logic, the system maintains processing efficiency while enabling rule reuse across multiple schemas through dynamic parameter configuration rather than hard-coded schema dependencies.
Data Source
AI summary
A method includes receiving, at a processor, first source data structured according to a first schema. The method further includes receiving, at the processor, second source data structured according to a second schema different from the first schema. The method further includes outputting, by the processor, event stream data for storage at a storage device, the event stream data representing a first event extracted from the first source data and a second event extracted from the second source. The event stream data includes a plurality of events corresponding to changes associated with a subject instance. The method also includes generating a profile of the subject instance based on the event stream data. The profile indicates a chronology of the changes associated with the subject instance.