Entropy-Based Transaction Data Segmentation for Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large data sets, such as transaction data, often exhibit skewed distribution, leading to uneven processing times across computing resources due to varying numbers of transactions associated with different account identifiers, resulting in inefficient use of processing units and memory.
Innovation Solution
A method for load balancing that identifies segments in a transaction data set based on entropy, sorts transaction values as floating-point values, and distributes processing tasks across multiple processors to ensure balanced processing loads by adjusting segment scopes to match adjacent segment entropies within a predefined tolerance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If transactions are grouped by PAN or total spend per PAN, then data processing can be organized by account identifiers, but processing times become uneven across processing units resulting in inefficient resource utilization
Solution Approach 1:
The patent segments the transaction data set into multiple segments based on entropy calculations, where each segment contains a balanced distribution of transaction values. This segmentation approach divides the skewed data into manageable chunks that can be processed in parallel, resolving the contradiction by organizing data efficiently while maintaining balanced processing loads across units.
Solution Approach 2:
The patent changes the parameter used for data grouping from account identifiers (PAN) to entropy-based segmentation. By calculating entropy for different data groupings and selecting segments with balanced entropy values, the system transforms the organization criterion to achieve both efficient data structure and balanced processing times.
2Ease of operation
If processing units are assigned tasks based on account identifiers with numerous transactions, then data can be processed by grouping, but processing time increases significantly for high-volume accounts compared to low-volume accounts
Solution Approach 1:
The patent applies segmentation by dividing the transaction data into multiple segments based on entropy calculations. Each segment is designed to have balanced entropy, ensuring that processing time variance across segments is minimized. This allows data grouping to be maintained while eliminating the extreme processing time differences between high-volume and low-volume accounts.
Solution Approach 2:
The patent introduces asymmetry in the segmentation process by using entropy-based criteria to create segments with balanced characteristics. Rather than symmetric grouping by account identifier ranges, the asymmetric entropy-based approach ensures each segment has comparable processing complexity, reducing time variance.
3Reliability
If data is skewed with unequal distribution of transactions across account identifiers, then realistic data patterns are maintained, but computing resources are used inefficiently with some units overloaded and others underutilized
Solution Approach 1:
The patent segments the skewed data while preserving the realistic distribution patterns within each segment. By using entropy-based segmentation, the overall data realism is maintained while the segmentation ensures balanced resource utilization across processing units, resolving the contradiction between data authenticity and resource efficiency.
Solution Approach 2:
The patent applies local quality by ensuring each segment has balanced entropy characteristics while the overall data set maintains its realistic skewed distribution. Each local segment is optimized for balanced processing, while the global data structure preserves realistic patterns.
Data Source
AI summary
Systems, methods, and computer program products are provided for load balancing for processing large data sets. The method includes identifying a number of segments and a transaction data set comprising transaction data for a plurality of transactions, the transaction data for each transaction of the plurality of transactions comprising a transaction value, determining an entropy of the transaction data set based on the transaction value of each transaction of the plurality of transactions, segmenting the transaction data set into the number of segments based on the entropy of the transaction data set and balancing respective entropies of each segment of the number of segments, and distributing processing tasks associated with each segment of the number of segments to at least one processor of a plurality of processors to process each transaction in each respective segment.