Adaptive Data Retrieval Set Sizing for Network Strain Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale data extraction and processing pose challenges due to strain on network and processing capabilities, particularly when transferring user accounts and associated statements across entities, as existing solutions struggle to efficiently target specific data without overburdening networks and processing systems.
Innovation Solution
A system and method for large-scale data extraction and processing that divides accounts into retrieval sets based on network capacity, time periods, and shared characteristics of user identifiers, allowing for efficient retrieval during low-usage times and consolidation into files for transfer, with adaptive adjustments based on actual network capacity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If large-scale data extraction is performed on all accounts simultaneously, then complete data retrieval is achieved, but network capacity is overwhelmed and system performance degrades
Solution Approach 1:
The patent segments the large-scale data extraction task by dividing accounts into multiple batches and creating multiple retrieval sets. Each retrieval set contains a subset of accounts that can be processed separately. This segmentation allows the system to retrieve data from all accounts completely while processing them in manageable portions, thereby avoiding network overload and system performance degradation.
2Loss of time
If data extraction is performed during high-usage time periods, then timely data retrieval is achieved, but network capacity is exceeded and retrieval fails
Solution Approach 1:
The patent implements dynamic retrieval set creation that adapts to changing network conditions. The system monitors network usage patterns and dynamically adjusts the timing and composition of retrieval sets. By scheduling retrievals during low-usage time periods when network capacity is available, the system achieves timely data retrieval without exceeding network capacity constraints.
Solution Approach 2:
The patent employs periodic action by scheduling data retrievals in multiple time periods rather than attempting single-pass retrieval. The system divides the extraction task across different time periods, performing retrievals during low-usage periods and pausing during high-usage periods. This periodic approach ensures complete data retrieval while respecting network capacity limitations at any given moment.
3Device complexity
If retrieval sets are created without considering network capacity, then simple processing logic is maintained, but actual network capacity is exceeded and retrieval pauses frequently
Solution Approach 1:
The patent incorporates feedback mechanisms in the retrieval set creation process. The system monitors actual network capacity and usage patterns, then uses this feedback to dynamically adjust the size and composition of retrieval sets. By considering expected network capacity when creating retrieval sets, the system avoids exceeding actual capacity, reducing the frequency of retrieval pauses, and maintaining high data extraction efficiency without overly complex processing logic.
Data Source
AI summary
Systems, computer program products, and methods are described herein for large scale data extraction and processing. The method includes receiving a transmission request. The transmission request indicates two or more accounts for which one or more account statements are to be retrieved. The method also includes selecting a first retrieval set of the two or more accounts. A first retrieval set size of the first retrieval set is based on a first expected network capacity. The method further includes causing a retrieval of one or more account statements associated with first retrieval set. The method still further includes selecting a second retrieval set of the two or more accounts. A second retrieval set size of the second retrieval set is based on a second expected network capacity. The method includes causing a retrieval of one or more account statements associated with the second retrieval set.