Analytic Function Data Block Buffering via Independent Reader Writer Movement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing frameworks for user-defined analytic functions face difficulties in efficiently managing large data blocks, particularly when the data does not fit in memory, leading to reduced performance and cumbersome programming, as they often require writing data to a hard-drive disk, which is inefficient.
Innovation Solution
The implementation of a method that allows independent movement of Readers and Writers within the data block, utilizing PreReader and PreWriter accessors for automatic buffering, enabling data to be stored in memory without user intervention, thus optimizing buffering efficiency and simplifying programming.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored on a hard-drive disk instead of memory when the distance between Reader and Writer exceeds buffer size, then memory usage is reduced, but access speed and performance deteriorate
Solution Approach 1:
The patent divides the data block into multiple segments or batches that can be processed independently. The Reader and Writer operate on different segments simultaneously, allowing the system to maintain efficient memory usage while avoiding the need to write intermediate results to slow disk storage. This segmentation enables the buffering mechanism to work effectively within memory bounds.
Solution Approach 2:
The patent implements preliminary buffering actions where data is pre-loaded into memory buffers before processing begins. By anticipating the data access patterns and pre-positioning data in memory, the system avoids the need for slow disk access during the actual processing phase, thus maintaining both low memory usage and high access speed.
2Quantity of substance
If the amount of data to be buffered is relatively large, then the analytic function can process more data, but the programming becomes relatively difficult and time consuming
Solution Approach 1:
The patent implements a self-managing buffering system where the framework automatically handles buffer allocation, data movement, and memory management without requiring explicit programmer intervention. The system monitors the distance between Reader and Writer automatically and adjusts buffering strategies accordingly, eliminating the need for programmers to manually program complex buffering logic for large data sets.
Solution Approach 2:
The patent creates a universal buffering framework that handles various analytic functions and data sizes through a single, standardized interface. This multi-functional design allows the same buffering mechanism to serve different purposes (different analytic functions, different data volumes) without requiring separate programming logic, thus reducing programming complexity while maintaining flexibility for large data buffering.
3Productivity
If Reader and Writer advance independently in the data block, then processing efficiency is improved, but coordinating their positions and managing buffers becomes more complex
Solution Approach 1:
The patent implements a feedback mechanism where the system continuously monitors the positions of Reader and Writer and automatically adjusts buffer allocation and data movement based on their relative distances. This real-time feedback allows independent advancement of Reader and Writer while automatically coordinating their operations, maintaining processing efficiency without requiring complex manual coordination logic.
Data Source
AI summary
In an example, an analytic function to be performed on data stored in an input block is managed through an interface to a framework through which a user is to define the analytic function. The framework is to buffer batches of the data into a memory through implementation of a Reader, a Writer, a PreReader, and a PreWriter on the data stored in the input block when the user-defined analytic function is performed, and wherein the Reader, the Writer, the PreReader, and the PreWriter are individually movable with respect to each other in the input block. In addition, the user-defined analytic function is received through the interface.