Materialized View Updates Using Binlog Record Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for replicating large-scale distributed computing platforms require excessive network data usage, storage, and compute costs due to the replication of complete database replicas or portions to edge servers, leading to high resource demands.
Innovation Solution
Implementing materialized views at edge databases and using lightweight database functions to selectively update only modified data, reducing network traffic and compute costs by processing database changes in parallel.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If complete database replicas are replicated to edge servers, then database availability and access speed are improved, but network data usage, storage costs, and compute costs increase excessively
Solution Approach 1:
The patent extracts only the modified portions of database records from the complete database replica and transmits them to edge servers, rather than replicating entire databases. This selective extraction of changed data reduces network bandwidth consumption, storage requirements, and synchronization overhead while maintaining the ability to provide updated data at edge locations.
Solution Approach 2:
The patent segments the database replication process into identifying individual modified records, extracting only those specific changes, and transmitting them separately to edge servers. This segmentation allows the system to send minimal data updates rather than complete database copies, resolving the contradiction between fast access and resource consumption.
2Device complexity
If lightweight database functions are used at edge servers, then resource demands are reduced, but database update capability is limited
Solution Approach 1:
The patent introduces a centralized database change identification system that acts as an intermediary between the core database and edge servers. This intermediary identifies modifications, extracts changed records, and prepares them for transmission to edge servers with lightweight functions. The intermediary compensates for the limited processing capability of edge servers by performing the complex analysis centrally.
Solution Approach 2:
The patent replaces the mechanical approach of executing full database update functions at edge servers with a centralized identification and extraction system. Instead of requiring edge servers to process and analyze entire databases locally, the system substitutes a centralized service that performs the heavy lifting of change detection and prepares minimal update packets for the lightweight edge servers to apply.
Data Source
AI summary
The server hosting the database scans a binlog for database change events. When a log consumer identifies a change event indicating that certain database records were modified, the consumer pulls only the identifiers of the modified records from the binlog. The log consumer then populates and pushes only the identifiers of the modified records into a non-transitory storage location that is accessible to a database streaming bus. The streaming bus publishes the identifiers for consumption by instances of materialization workers. The hosting server invokes parallel processor threads to execute the materialization workers. The materialization worker rewrites a query script for constructing a materialized view of certain database records, including the modified database records indicated by the identifiers. The materialization worker executes the query script to construct the materialized view, which contains only the modified database records used for generating the database updates to commit to the database.