Independent Executable Units for Non-Blocking Data Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data migration and integration processes are costly and risky, especially when dealing with large and complex database systems, often resulting in data corruption and requiring significant resources due to the ad hoc nature of existing data integration applications that are not reusable.
Innovation Solution
The development of data integration applications using executable units that operate independently, with a semantic model allowing for data transformation and integration without knowledge of physical data identities, and a hybrid versioning system for project management, enabling parallel and distributed execution across multiple hosts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data integration applications are developed ad hoc for specific tasks, then they can be quickly implemented, but they become useless after completion and cannot be reused, substantially increasing development costs
Solution Approach 1:
The patent segments data integration applications into independent, reusable components including executable units, semantic identifiers, and functional rules. Each component can be developed, tested, and reused separately across different data integration projects, transforming monolithic ad hoc applications into modular building blocks that maintain both quick implementation and reusability.
Solution Approach 2:
The patent creates universal semantic identifiers and functional rules that can be applied across multiple data integration scenarios. These reusable components serve multiple functions and can be deployed in various contexts, allowing a single component to address different data integration needs without requiring custom development for each specific task.
2Productivity
If data integration applications are written from scratch without comprehensive testing, then development is faster, but the likelihood of data corruption increases
Solution Approach 1:
The patent implements preliminary action through comprehensive testing of reusable components before they are deployed. Semantic identifiers, functional rules, and executable units undergo rigorous validation and testing during the development phase, ensuring data integrity is established beforehand. This preliminary testing prevents data corruption issues from arising during actual data integration operations.
3Reliability
If executable units operate independently with non-blocking status information channels, then system reliability improves, but communication overhead increases
Solution Approach 1:
The patent extracts status information communication from the main data flow channels, creating separate non-blocking status information channels for each executable unit. This separation allows executable units to report their status independently without interfering with the primary data integration operations, improving system reliability while managing communication overhead through dedicated-purpose channels.
Data Source
AI summary
Methods and systems for executing a data integration application using executable units that operate independently of each other. Methods include receiving a specification for a data integration application comprising programmatic rules and specifying data-flow, creating executable units corresponding to the programmatic rules, creating data-flow channels corresponding to the specified data-flow, and executing the executable units such that data is passed between them in a non-blocking manner.