Data Concurrency via Delta Chain Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data concurrency issues arise in data aggregation systems due to rigid data storage infrastructures, where updates in one source may conflict with locally modified data on client devices, requiring expensive infrastructure changes and conflict resolution mechanisms.
Innovation Solution
A data processing engine that aggregates data from multiple sources into flexible records, maintaining a delta chain for version tracking and using a data concurrency module to resolve conflicts by merging edits based on intervening modifications, ensuring data consistency across sources and devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored centrally in a rigid schema, then data consistency is maintained, but adaptability to different data sources and schema changes is reduced
Solution Approach 1:
The patent implements dynamic schema evolution by allowing the data store to adapt its structure over time. The system maintains a schema that can be modified to accommodate new data sources and field types without requiring complete schema redesign, enabling the infrastructure to evolve dynamically while preserving data consistency through version control and migration pathways.
Solution Approach 2:
The system changes the parameters of the data storage schema to accommodate different data sources. By allowing schema parameters (field types, constraints, structures) to be modified and extended, the system can integrate new data sources with different schemas while maintaining a unified data model, thus balancing consistency with adaptability.
2Adaptability or versatility
If schema updates are performed frequently, then adaptability to data sources is improved, but infrastructure complexity and data movement costs increase
Solution Approach 1:
The system performs preliminary schema validation and compatibility checking before data ingestion. By pre-defining schema templates and validation rules, the system can accommodate schema updates without requiring complex real-time infrastructure changes, reducing the complexity burden of frequent schema adaptations.
Solution Approach 2:
The patent introduces an intermediary schema layer that sits between the rigid storage infrastructure and the variable data sources. This intermediary schema acts as a mediator, translating diverse data source schemas into a unified internal representation, thereby reducing infrastructure complexity while maintaining schema flexibility.
3Reliability
If data is synchronized in real-time, then data concurrency is improved, but network bandwidth and system resources are consumed
Solution Approach 1:
Instead of continuous real-time synchronization, the system implements periodic data synchronization at defined intervals or triggers. This periodic action maintains data concurrency by ensuring data is updated regularly while significantly reducing network bandwidth consumption compared to continuous synchronization, as data is transmitted only at scheduled moments rather than continuously.
Solution Approach 2:
The system implements self-service conflict resolution mechanisms where the data store automatically detects and resolves concurrency conflicts using predefined strategies (e.g., last-write-wins, version comparison) without requiring constant network communication or manual intervention. This self-service approach maintains data concurrency while minimizing network resource usage by handling conflicts locally.
Data Source
AI summary
A data concurrency module maintains a delta chain for each record that stores the edits made to the flexible record over time. The delta chain stores the edits ordered by a version identifier. When the data concurrency module receives an edit to a record, the data concurrency module compares the version identifier associated with the edit with the most recent version identifier stored in the delta chain for that flexible record. If the version identifiers are different, then the data concurrency module merges the edit with all intervening edits in the delta chain, resolving any conflicts that result from the merging.