List-Based Database Replication With Row-Level Entitlement Filtering

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Replicating entire databases across a networked environment is costly and induces high latency due to the size of the databases, and existing methods do not allow users to selectively replicate relevant portions.

Innovation Solution

Implementing sub-database replication through schema and table filtering, data sharing, and row-level filtering to replicate only necessary data to different regions, using a replication controller that applies user-defined inclusion and exclusion lists, entitlements, and materialized views.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If entire databases are replicated across networked regions, then data availability and reliability are improved, but replication cost and latency increase significantly

Engineering Contradiction:

Improvedata availabilityVSAvoidreplication latency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent segments databases into individual tables and further into rows, enabling selective replication of only the necessary portions. The replication controller processes tables one at a time and filters rows based on user-defined criteria, transforming the monolithic replication approach into a granular, selective process that reduces latency and cost while maintaining reliability for critical data.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different regions receive different subsets of data based on local requirements. The system applies user-defined inclusion and exclusion lists to determine which rows should be replicated to each specific region, ensuring that each region receives only the data it needs rather than the entire database, thereby reducing replication overhead and latency.

Inventive Principle:

Principle #3Local quality

2Reliability

If entire databases are replicated across networked regions, then data safety against failures is improved, but replication cost increases

Engineering Contradiction:

Improvedata safetyVSAvoiddata volume replicated

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent extracts and replicates only the necessary rows from each table based on user-defined criteria. The replication controller processes inclusion and exclusion lists to identify which rows should be replicated, extracting only the essential data portions rather than replicating the entire database, thereby reducing storage costs and network bandwidth requirements while maintaining data safety for critical information.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

Instead of replicating the complete database (excessive action), the system performs partial replication by processing only the necessary tables and rows. The replication controller applies filtering criteria to determine the minimum necessary data to replicate, avoiding unnecessary data duplication and reducing overall replication cost while ensuring adequate data safety.

Inventive Principle:

Principle #16Partial or excessive action

3Loss of time

If selective row-level filtering is implemented, then replication cost and latency are reduced, but system complexity increases

Engineering Contradiction:

Improvereplication latencyVSAvoidfiltering mechanism complexity

Core Design Contradiction:

Loss of timeVSDevice complexity

Solution Approach 1:

The system provides self-service through user-defined inclusion and exclusion lists. Users specify which rows they want replicated without requiring complex configuration of the filtering mechanism itself. The replication controller automatically processes these simple lists to identify and replicate the necessary rows, reducing the operational complexity burden on users while maintaining low latency replication.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

Users pre-define their replication requirements through inclusion and exclusion lists before data replication occurs. The replication controller then uses these pre-defined criteria to automatically filter and select rows during replication, eliminating the need for complex real-time filtering logic and reducing the perceived system complexity from the user's perspective.

Inventive Principle:

Principle #10Preliminary action

4Measurement precision

If user-defined filtering lists are applied, then data precision for replication is improved, but operational complexity increases

Engineering Contradiction:

Improvedata selection precisionVSAvoidconfiguration ease

Core Design Contradiction:

Measurement precisionVSEase of operation

Solution Approach 1:

Instead of requiring users to specify exactly which rows to replicate (positive definition), the system allows users to define which rows to exclude (negative definition through exclusion lists). This inversion simplifies the user interface and operational complexity, as exclusion lists are easier to manage than inclusion lists, while still achieving precise data selection through the combination of default inclusions and explicit exclusions.

Inventive Principle:

Principle #13The other way round (Inversion)

Data Source

PatentUS12475142B2System for list-based database replication

Publication Date: 2025.11.18 SNOWFLAKE INC

AI summary

A method of implementing sub-table replication starts with the processor detecting an update to an entitlements table. The processor performs filtering of a data table based on the update to the entitlements table. The data table including an entitlements column. The processor detects an update to the entitlements column and performs incremental replication of the data table by causing a version-based replication to be executed. Other embodiments are also described herein.