Dynamic Shared Data Model Adaptation via Conflict Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data teams face challenges in managing the evolving shared data model in ELT layers due to constant changes in data sources and business needs, requiring a solution that balances data application development with data consistency across expanding and fragmented organizations.
Innovation Solution
A system that receives structured queries, applies heuristics to detect conflicts with the shared data model, generates new queries and modified data models, and initiates regression tests to ensure seamless adaptation and consistency, using a combination of software, firmware, and hardware to automate the change management process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the shared data model is frequently updated to adapt to changing data sources and business needs, then the adaptability of the data model is improved, but the consistency and reliability of data applications deteriorate
Solution Approach 1:
The system performs preliminary actions by automatically detecting conflicts between queries and the shared data model before they are executed. Heuristics are applied in advance to identify potential inconsistencies, and regression tests are run beforehand to predict impacts on other applications. This prevents conflicting changes from propagating through the system, thereby maintaining reliability while allowing frequent updates.
Solution Approach 2:
The system implements feedback mechanisms where each query is analyzed against the shared data model, and the results of regression tests are fed back into the change management process. This continuous feedback loop allows the system to learn from previous changes and adjust future updates to maintain consistency, resolving the contradiction between frequent updates and data application reliability.
2Productivity
If automated conflict detection and resolution is implemented, then the productivity of data teams is improved, but the device complexity increases
Solution Approach 1:
The system performs self-service by automatically detecting conflicts, generating regression tests, and identifying impacts without requiring manual intervention from data teams. The automated heuristic analysis and regression testing processes handle the complexity internally, freeing data teams to focus on business value while the system manages the technical complexity of change coordination.
Solution Approach 2:
The system manages complexity by changing parameters such as the level of automation, the strictness of conflict detection thresholds, and the scope of regression testing. These parameter adjustments allow the system to adapt to different organizational needs and complexity levels, making the solution scalable without forcing maximum complexity into all implementations.
3Reliability
If regression testing is performed on every data model change, then the reliability of data applications is improved, but the time required for changes increases
Solution Approach 1:
The system applies partial action by performing regression testing selectively rather than exhaustively on every possible data model change. The heuristic analysis identifies which queries and applications are most likely to be impacted, and regression tests are focused on those specific areas. This partial approach maintains reliability for critical paths while reducing the overall time required compared to full regression testing of all applications.
Data Source
AI summary
A system and method for dynamically adapting a query and shared data model, is presented. The method includes receiving a structured query directed to a shared data model; apply a plurality of heuristics to the received structured query and the shared data model; detecting a conflict between a first element of the structured query and a first element of the shared data model based on a first heuristic of the plurality of heuristics; generating, in response to detecting a first result of applying the first heuristic, a new structured query based on the received structured query and the first result; generating, in response to detecting a second result of applying a second heuristic of the plurality of heuristics, a modified shared data model, based on the shared data model and the second result.