Cross-Region Data Processing with Automatic Region Switching

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Cloud-based data processing systems face limitations in time and region, leading to potential data loss due to execution context timeouts and region-specific failures, which can result in incomplete processing of batched input files.

Innovation Solution

Implementing a cross-region data processing system that formats data into portions, generates copies, and processes them across multiple regions, using raw data queues to handle incomplete processing and automatic region switching to ensure data integrity and availability.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of manufacture

If cloud-based data processing functions are used to eliminate infrastructure maintenance and enable pay-per-use, then operational cost and complexity are reduced, but execution time limitations cause data loss for partially processed files

Engineering Contradiction:

Improveinfrastructure maintenanceVSAvoiddata processing completeness

Core Design Contradiction:

Ease of manufactureVSReliability

Solution Approach 1:

The system performs preliminary actions by creating copies of input files before processing begins. These copies are stored in a data lake and made available in alternative regions before any processing occurs, ensuring that if the primary processing fails, the data is already preserved and can be reprocessed without loss.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements beforehand cushioning by establishing backup processing regions and storing file copies in advance. When processing fails in the primary region, the system can immediately switch to a backup region with the pre-stored copies, cushioning against the harmful effect of data loss from execution timeouts.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

2Adaptability or versatility

If execution contexts are limited to fixed time periods to ensure fair resource access, then resource allocation fairness is improved, but transient memory is erased and processing cannot continue for complex files

Engineering Contradiction:

Improveresource access fairnessVSAvoidexecution context lifetime

Core Design Contradiction:

Adaptability or versatilityVSDuration of action of moving object

Solution Approach 1:

The system segments the processing workflow into distinct phases: file ingestion, processing attempts in multiple regions, and result aggregation. Each phase operates within execution context limits, but the overall process can span multiple contexts by retrieving files from the data lake between phases, effectively extending processing duration beyond single context limits.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The data lake serves as an intermediary between the execution contexts and the processing logic. Instead of maintaining state within the transient execution context, the system stores and retrieves data from the persistent data lake, allowing processing to continue across multiple execution contexts without losing data.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Device complexity

If region-specific processing is used to simplify system architecture, then system complexity is reduced, but regional failures result in permanent data loss

Engineering Contradiction:

Improvesystem architectureVSAvoidprocessing availability

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

The system merges multiple processing regions with a central data lake into a unified architecture. The data lake acts as a shared resource that all regions can access, combining the simplicity of region-specific processing with the reliability of multi-region redundancy. When one region fails, others can access the same data lake to continue processing.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The data lake serves multiple functions: storing input files before processing, providing backup copies during processing, and enabling failover between regions. This universal resource supports both the simplicity of region-specific architecture and the reliability of distributed systems.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11687561B2Systems and methods for cross-region data processing

Publication Date: 2023.06.27 CAPITAL ONE SERVICES LLC

AI summary

A system may include processor(s) and a memory in communication with the processor(s) and storing instructions, that when executed by the processor(s), are configured to cause the system to perform a method for cross-region data processing. The system may receive data in a first processing region and may format the data into a plurality of data portions. The system may generate a copy of each data portion and may transmit each data portion copy to a cross-region processor. Responsive to determining the first processing region is functioning, the system may process data portion(s). The system may transmit, to a centralized location, the processed data portion(s). Responsive to determining the first processing region is not functioning, the system may transmit, from the cross-region processor, each data portion copy to a second processing region.