Database Data Duplication Controller with Flattening Pipeline
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems face challenges in efficiently duplicating structured data, which is crucial for ensuring data integrity and accessibility across multiple repositories.
Innovation Solution
The proposed method involves a data duplication controller that writes data changes to a log table, extracts these changes using a data extractor, and then uses a data flattener to generate flattened data. This data is packaged into flat files, which are then made accessible through a listing API, enabling efficient duplication and access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is duplicated across multiple repositories in real-time, then data accessibility and consistency are improved, but system complexity and processing overhead increase
Solution Approach 1:
The patent introduces a data duplication controller as an intermediary component that sits between the data source and multiple repositories. This controller receives data changes, processes them through a standardized pipeline (extractor → flattener → packager), and distributes them to repositories. This mediator approach simplifies the overall system architecture by centralizing the duplication logic, avoiding the need for complex peer-to-peer synchronization mechanisms between repositories, and ensuring consistent data distribution across all repositories.
2Loss of time
If data changes are processed and distributed immediately, then data freshness is improved, but processing latency and resource consumption increase
Solution Approach 1:
The patent implements preliminary action by pre-defining extraction rules, flattening schemas, and packaging templates before data changes occur. The system prepares the processing pipeline in advance, so when data changes are detected, they can be quickly processed through the pre-configured steps. This reduces processing latency because the system doesn't need to decide on the fly how to process each data change, and it minimizes resource consumption by avoiding repeated setup and configuration operations.
3Productivity
If multiple extracts are packaged into a single data change file, then transmission efficiency is improved, but file management and version control become more difficult
Solution Approach 1:
The patent applies segmentation by organizing the data change file into distinct, clearly demarcated sections for each extract. Each extract's data is packaged as a separate unit within the overall file structure, with clear delimiters and metadata identifying the boundaries and characteristics of each segment. This segmented approach maintains transmission efficiency by consolidating multiple extracts into a single file while simplifying file management, as each extract can be independently identified, extracted, and tracked using its unique metadata markers.
Data Source
AI summary
Systems and methods for duplicating data in a data management system. As data is being updated in the data management system, a copy of the updated data is being written to a table for collecting the data changes. The data changes are flattened by a flattener to generate one or more flat files. A packaging controller may package the one or more flat files and generate a data change file. A listing API may list the data change files that are available. The data change file may be accessed via a data access API.