Selective Key Shearing for Record Processing Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data sorting processes consume significant computing resources due to extensive data movement across different storage locations, particularly when using higher performance storage, which is often limited by lower storage capacity and higher costs.
Innovation Solution
The method involves shearing a key from payload data, normalizing it, and storing the key in a first memory area while keeping the payload data in a second memory area if its size is outside a predetermined range, reducing data movement and processing overhead by decoupling the key from the payload data during record processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data sorting is performed using conventional methods with extensive data movement, then sorting functionality is achieved, but computing resource consumption increases significantly
Solution Approach 1:
The patent segments the record into two separate parts: the key portion and the payload data. The key is extracted and placed in a first buffer area while the payload remains in a second buffer area. This segmentation allows sorting operations to process only the smaller key portions without moving the entire large payload data, significantly reducing data movement and computing resource consumption while maintaining sorting functionality.
2Speed
If higher performance storage is used for data sorting, then sorting speed improves, but storage capacity decreases and cost increases
Solution Approach 1:
By segmenting the record into key and payload components stored in separate buffer areas, the patent enables sorting operations to work with only the small key portions in high-performance storage (first buffer area). The large payload data remains in lower-performance, higher-capacity storage (second buffer area). This resolves the contradiction by allowing fast sorting on keys while maintaining adequate storage capacity for the actual data.
3Productivity
If payload data is moved during sorting operations, then sorting can be performed, but data movement overhead increases processing latency
Solution Approach 1:
The patent extracts the key portion from the payload data and places it in a separate first buffer area. This extraction allows sorting operations to proceed using only the small key data without moving the large payload, significantly reducing data movement overhead and processing latency while maintaining record processing efficiency.
Data Source
AI summary
A computer-implemented method, according to one embodiment, includes: storing records in an input data buffer, where each of the records include a key which is appended to payload data in the respective record. Moreover, for each of the records: shearing the key associated with the record from the payload data, normalizing the sheared key, and storing the normalized sheared key in a first target area of memory. A determination is made as to whether a size of the payload data in the record is outside a predetermine range, and in response to determining that the size of the payload data in the record is outside the predetermine range, the payload data is stored in a second target area of memory. A data locator is also appended to the normalized sheared key in the first target area of memory to form a sheared record.