Intermediate Device Dynamic Buffer Allocation for Unknown Data Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In database search processes, the host lacks recognition of the data transfer amount for search queries, leading to inefficient buffer area allocation and utilization, especially when the number of search hit rows is small, resulting in low efficiency and unnecessary capacity usage.
Innovation Solution
An intermediate device is introduced between the host and the search target, which issues search requests in multiple processes and recognizes hit data volumes, allowing dynamic allocation and updating of write destination addresses based on recognized hit data volumes, reducing the number of buffer areas and improving search speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a buffer area is preliminarily allocated to the main memory area in the host to store all possible search hit rows, then the host can handle any search query without recognizing data transfer amount, but the buffer area capacity must be large enough to store all rows resulting in low efficiency when the number of actually search hit rows is small
Solution Approach 1:
An intermediate device is introduced between the host and storage to act as a mediator. The intermediate device receives search requests from the host, performs the search operations, and dynamically manages buffer area allocation based on actual search hit volumes. This intermediary resolves the contradiction by eliminating the need for the host to preliminarily allocate large buffer areas while maintaining the ability to handle any search query.
Solution Approach 2:
The buffer area allocation is made dynamic rather than static. The intermediate device adjusts the buffer area capacity based on the actual number of search hit rows returned by the storage. This dynamic adjustment allows the system to maintain reliability for any search query while improving productivity by allocating only the necessary buffer capacity needed for each specific search operation.
2Speed
If multiple buffer areas are allocated for parallel processing with node clustering and multi-threading, then the database search processes can operate in parallel, but the total buffer area capacity increases leading to low efficiency when the number of hit rows is small
Solution Approach 1:
Multiple buffer areas that would traditionally be allocated for parallel processing are merged into a single shared buffer area managed by the intermediate device. The intermediate device dynamically allocates portions of this shared buffer to different parallel search processes based on their actual needs. This merging maintains the parallel processing speed while improving overall buffer area usage efficiency by eliminating redundant allocations.
Solution Approach 2:
The shared buffer area managed by the intermediate device serves multiple functions and multiple parallel search processes simultaneously. Instead of dedicating separate buffer areas to each parallel process, a single universal buffer area is created that can be dynamically allocated to any number of parallel search operations, improving both speed and efficiency.
3Ease of operation
If the host issues search requests without recognizing data transfer amount, then the host can simplify the search query process, but the host cannot optimize resource allocation leading to wastage of buffer area capacity
Solution Approach 1:
The intermediate device serves as an intermediary that handles the complex task of data transfer amount recognition and buffer area optimization. The host can issue simple search requests without needing to recognize or calculate data transfer amounts, while the intermediate device automatically optimizes resource allocation based on actual search results, maintaining ease of operation while improving productivity.
Solution Approach 2:
The intermediate device autonomously performs resource allocation optimization without requiring complex host involvement. The system enables the host to focus on simple search operations while the intermediate device self-manages the buffer area allocation and resource optimization, achieving both ease of operation and improved productivity.
Data Source
AI summary
An intermediate device is disposed between a host and a search target. Search requests of m-multiplicity (requests involving a data transfer amount that is unknown to the host) for at least one among n-processes (n is an integer equal to or more than 1) to be executed by the host are issued as requests from the host to the intermediate device. A temporary area associated with the search requests is allocated in the host. For each of the search requests, the intermediate device recognizes a hit data volume in accordance with the search request within a search scope. For each of the search, the hit data is written in an area of the temporary area that corresponds to a write destination address. The write destination address is updated for each of the search requests on the basis of the recognized hit data volume.