High-Scale General Ledger Processing with Bit-Array Delta Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing ledger data processing systems face inefficiencies in handling large volumes of data, particularly in identifying and optimizing computationally costly database operations such as insert, update, and delete operations, especially when dealing with large-scale financial data from ERP systems.
Innovation Solution
A novel ledger data processing system utilizing a computing system with a processor, memory, and thread scheduler, employs a novel data structure and method to optimize data ingestion by generating bit arrays from comparing received and stored ledger data, enabling intelligent scheduling of database operations based on bit count sums to identify and batch costly operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional ledger data processing methods are used, then data can be processed, but processing efficiency is low and computational overhead is high when handling large volumes of data
Solution Approach 1:
The patent segments the comparison process by generating separate bit arrays for different data dimensions (e.g., account bit array, segment bit array, balance bit array). This allows the system to process and analyze specific segments of data independently, improving overall processing efficiency when handling large volumes of ledger data.
Solution Approach 2:
The patent replaces traditional row-by-row data comparison mechanisms with a bit array-based comparison system. By encoding data values into bit arrays and using bitwise operations for comparison, the system achieves significantly faster processing speeds and reduced computational overhead compared to conventional mechanical comparison methods.
2Productivity
If all database operations are processed uniformly, then processing is straightforward, but computationally costly operations (insert, update, delete) cannot be identified and optimized
Solution Approach 1:
The patent uses bit array states as indicators to identify different types of database operations. By setting specific bits in bit arrays based on data changes (e.g., inserting a bit when a new record is detected, updating bits when values change), the system can quickly identify and categorize costly operations like inserts, updates, and deletes without complex analysis logic.
Solution Approach 2:
The patent performs preliminary analysis by generating bit arrays and identifying costly operations before executing the actual database operations. This allows the system to batch and optimize costly operations in advance, improving processing speed while managing complexity through pre-computation.
3Measurement precision
If detailed comparison of all ledger data values is performed, then accurate change detection is achieved, but computational overhead increases significantly
Solution Approach 1:
The patent replaces detailed value-by-value comparison with bit array-based comparison mechanisms. By encoding data into bit arrays and using efficient bitwise operations (XOR, AND, OR) for comparison, the system maintains accurate change detection while dramatically reducing computational overhead compared to traditional comparison methods.
Solution Approach 2:
The patent transforms data values into bit array representations, changing the parameter form from numerical values to binary bit patterns. This transformation enables efficient bitwise operations for comparison, maintaining detection accuracy while reducing computational resource consumption.
4Quantity of substance
If large quantities of ledger data are ingested, then comprehensive data coverage is achieved, but processing speed and memory usage are adversely affected
Solution Approach 1:
The patent segments large volumes of ledger data into manageable bit array structures organized by different dimensions (accounts, segments, balances). This segmentation allows the system to process large quantities of data in parallel and efficiently manage memory usage, maintaining high ingestion speeds even when handling comprehensive data sets.
Solution Approach 2:
The patent transforms one-dimensional ledger data into multi-dimensional bit array structures, adding dimensions for accounts, segments, and balances. This dimensional transformation enables more efficient storage and processing of large data volumes by organizing information in a structure that optimizes both memory usage and processing speed.
Data Source
AI summary
A system comprises a computing system, an Enterprise Resource Planning (ERP) system, and a relational database management system (RDBMS). Ledger data is received onto the computing system and stored in a first data structure. Aggregated values over one or more time periods are determined and stored in the first data structure. Next, stored ledger data stored in a database is retrieved and stored in a second data structure. The first and second data structures are stored in continuous blocks of memory and are highly amenable to caching, thereby improving system performance. One or more bit arrays are generated by comparing values stored in the first and second data structures. These bit arrays indicate data change state between stored and received ledger data. Bit counting yields bit count sums that scalably identify costly database operations for received data across many accounts. Database operations are scheduled intelligently using bit count sums.