Distributed Event Log Sorting via Composite Key Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing number of event log files in database network systems poses a challenge due to high resource requirements for sorting, often resulting in incomplete or aborted processes due to insufficient resources.
Innovation Solution
A processing system distributes log lines based on composite keys, extracting natural key values to sort and group log lines across multiple components, generating tenant-specific files, thereby managing resource usage effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If event log files are sorted based on user/tenant and chronology, then the sorting completeness and accuracy are improved, but the resource requirements (memory, processing power) increase significantly
Solution Approach 1:
The patent divides the sorting process into multiple passes or stages. Instead of loading all log files into memory at once for a single comprehensive sort, the system performs incremental sorting operations that process subsets of data, reducing peak memory requirements while achieving complete sorting results.
Solution Approach 2:
The patent introduces a temporal dimension to the sorting process by using checkpointing and resumption capabilities. The sorting operation can be paused, saved to disk, and resumed later, transforming a single large in-memory operation into multiple smaller operations across time, thereby reducing memory pressure.
2Quantity of substance
If the number of event log files increases, then the data coverage and analytical value are improved, but the sorting process may become incomplete or aborted due to insufficient resources
Solution Approach 1:
The patent implements preliminary actions such as data validation, resource pre-allocation, and sorting strategy determination before the actual sorting process begins. This includes assessing available resources and adjusting the sorting plan accordingly, which helps ensure that the sorting process can complete successfully even with large numbers of log files.
Solution Approach 2:
The system incorporates feedback mechanisms that monitor resource usage during the sorting process and dynamically adjust the sorting strategy. If resources are becoming scarce, the system can pause, checkpoint progress to disk, and resume later when resources are available, ensuring sorting completion rather than abortion.
3Productivity
If all event log files are loaded into memory for sorting, then the sorting speed is improved, but the memory consumption increases to unsustainable levels
Solution Approach 1:
The patent segments the log file data into smaller chunks or blocks that can be processed individually in memory. Each chunk is sorted independently and then merged with others, allowing the system to maintain high sorting speeds by keeping active working sets in memory while reducing overall memory consumption through incremental processing.
Solution Approach 2:
The sorting process is structured as a periodic operation where data is loaded into memory in batches, sorted, written to intermediate storage, and then the next batch is processed. This periodic pattern allows memory to be cleared and reused, maintaining sorting throughput while preventing memory exhaustion.
Data Source
AI summary
In an example, composite keys for an event log may be provided. A partitioner may be configured to extract a natural key from the composite keys and distribute log lines of event log files to a plurality of reducer nodes based on a value of the natural key. A comparator may use a log time of the composite key to sort a received portion of the distributed log lines.