Data Propagation Mapping System for Incompatible Storage Formats
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data stores often face challenges in recognizing and processing data in incompatible storage formats, leading to difficulties in identifying changes and propagating these changes to other data stores that use different formatting systems.
Innovation Solution
A data propagation and mapping system that identifies changed data entries in one storage format, filters them, and transforms them into a compatible format for transmission to another data store, using a pipeline with blocks for receiving, transformation, and outputting data in the target format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is transmitted between data stores with incompatible storage formats, then data integration is achieved, but data recognition and processing become difficult
Solution Approach 1:
The patent introduces a data transformation service as an intermediary component that receives data in the first storage format, transforms it to the second storage format, and transmits it to the target data store. This mediator resolves the incompatibility between different storage formats (e.g., MongoDB's BSON format and SQL databases' tabular format) without requiring changes to the source or destination systems, thereby enabling data integration while maintaining recognition capability.
Solution Approach 2:
The transformation service dynamically changes the parameters of data representation by detecting the source storage format and target storage format, then applying appropriate transformation rules to convert data structures, data types, and formatting conventions. This parameter adaptation allows the system to handle multiple incompatible formats seamlessly.
2Productivity
If data transformation is performed to enable compatibility between different storage formats, then data propagation is enabled, but processing complexity increases
Solution Approach 1:
The transformation service is segmented into distinct functional blocks: a format detection block that identifies the source and target formats, a transformation rule selection block that chooses appropriate conversion rules, and a data conversion block that executes the transformation. This segmentation reduces overall complexity by making each component specialized and manageable.
Solution Approach 2:
The system pre-configures transformation rules and mappings between different storage formats in advance. When data transformation is needed, the service simply applies the pre-defined rules rather than generating transformations dynamically, which simplifies the processing logic and improves efficiency.
3Reliability
If all data entries are transmitted between data stores, then data consistency is maintained, but transmission efficiency decreases
Solution Approach 1:
The system implements change detection mechanisms that identify only the specific data entries that have been modified in the source data store. Instead of transmitting all data entries, the transformation service processes and transmits only the changed entries to the target data store. This partial action approach maintains data consistency while significantly improving transmission efficiency by reducing unnecessary data movement.
Data Source
AI summary
Systems and methods for data propagation and mapping are provided. In an aspect, one or more data entries storing changed information in a first database using a first storage format are identified. The identified data entries are received by the data propagation and mapping system. The received data entries may be filtered to generate a subset of filtered data entries. The filtered data entries are transmitted to a mapping pipeline configured to map a data entry stored in the first storage format to a data entry stored in a second storage format. The mapped data entries are transmitted to a recipient second database storing data entries using the second storage format.