Hash Table Data Processing with Pipeline and History Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing load on databases due to the growing number of users leads to performance issues, such as read delays and data inconsistency, particularly in systems using hash tables for key-value storage, where collisions occur and contention between queries results in inconsistent data.
Innovation Solution
A data processing apparatus with units for managing update and read requests, calculating update values, and generating histories to estimate update information, allowing for pipeline processing that minimizes the impact of read delays by queuing requests and updating data consistently, even during read operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a hash table is used for key-value storage to increase throughput, then the number of servers can be decreased and power consumption saved, but collisions occur causing read delays and data inconsistency
Solution Approach 1:
The system pre-calculates hash values for incoming keys before actual data access occurs. This preliminary hashing action allows the system to prepare access patterns and mitigate collision impacts in advance, reducing read delays while maintaining throughput efficiency
Solution Approach 2:
An intermediary processing layer is introduced between the hash table and the data access operations. This intermediary manages collision resolution and coordinates read/write operations to prevent data inconsistency, allowing the hash table to maintain high throughput while the intermediary ensures data consistency
2Speed
If DRAM is used as cache storage to respond quickly to individual accesses, then response time is improved, but read delays occur causing waiting queries to be delayed
Solution Approach 1:
The system implements pipeline processing that allows multiple queries to be processed continuously. While one query waits for DRAM read completion, the system continues processing subsequent queries, eliminating idle waiting time and maintaining continuous useful action throughout the system
Solution Approach 2:
The query processing system dynamically adjusts its behavior based on DRAM read status. When reads are in progress, the system queues additional queries and processes them as soon as resources become available, creating a dynamic response that maximizes throughput while minimizing waiting time
3Productivity
If pipeline processing is implemented to reduce query waiting time, then throughput is improved, but data inconsistency occurs due to read/write contention
Solution Approach 1:
The system implements feedback mechanisms that track the status of read and write operations in the pipeline. When write operations are detected, the system adjusts subsequent read operations to avoid contention, using feedback from operation status to maintain data consistency while preserving pipeline throughput
Solution Approach 2:
Write operations are scheduled and coordinated in advance within the pipeline processing system. By preliminarily determining write timing and location, the system can arrange read operations to avoid contention, ensuring data consistency is maintained while pipeline throughput is optimized
Data Source
AI summary
According to an embodiment, an update request reception unit receives a data update request to update data stored in a storage. A read request unit makes a read request to read data from the storage. A data reception unit receives the data from the storage. An update value calculator calculates an update value of the received data. A write request unit makes a write request to write the calculated update value into the storage. A data processing execution unit executes reading and writing on the storage. A history processing unit generates a history of the reading and deletes a history of reading corresponding to the writing. An update information estimator estimates, from the generated history, update information indicating how the data is to be updated. A data update unit updates the data read according to the estimated update information and output new updated data to the data reception unit.