Configurable Hash Engine for Real-Time Database Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hash engine technologies face challenges in achieving real-time hashing throughput across various applications, including network and database processing, due to varying throughput requirements and inefficiencies in hardware acceleration for lower bandwidth applications.
Innovation Solution
The implementation of a hash engine with multiple parallel hash lanes that utilize pipelining and horizontal scaling to sustain real-time hashing throughput, combined with software configurability through descriptors and progress pointers to manage data loading and hashing efficiently, allowing for preemptive and resumeable hashing operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hardware acceleration is used for hashing, then real-time throughput is improved, but device complexity increases
Solution Approach 1:
The hash engine is divided into multiple independent hash lanes (e.g., 16 lanes) that can operate in parallel. Each lane processes a portion of the data independently, allowing the system to achieve high throughput by distributing the hashing workload across multiple simpler processing units rather than using a single complex hardware accelerator.
Solution Approach 2:
The hash engine is designed to support multiple hashing algorithms (e.g., CRC32, CityHash, MurmurHash) through a unified hardware architecture. The same physical hash lanes can be configured to execute different hash functions by loading appropriate algorithm binaries into the lane buffers, eliminating the need for separate hardware circuits for each algorithm.
2Device complexity
If software hashing is used on processor cores, then device complexity is reduced, but productivity decreases
Solution Approach 1:
A hash engine interface acts as an intermediary between the software application and the parallel hash lanes. The interface receives hashing requests, manages the loading of data into lane buffers, coordinates the execution across multiple lanes, and aggregates the results. This intermediary layer allows software to access hardware acceleration capabilities without requiring complex direct hardware control.
3Productivity
If multiple parallel hash lanes are used, then real-time throughput is improved, but device complexity increases
Solution Approach 1:
Multiple hash lanes share common resources including a single lane buffer memory, control logic, and result aggregation circuitry. The lanes are synchronized to process data in lockstep, and their individual hash results are combined through a simple reduction operation to produce the final hash value. This merging of resources reduces overall complexity compared to having fully independent processing units.
Solution Approach 2:
The hash engine utilizes the parallel dimension by implementing multiple hash lanes that operate simultaneously on different portions of the input data. Instead of processing data sequentially through a single lane, the system distributes data across multiple lanes in parallel, effectively adding a temporal dimension to the processing architecture to achieve real-time throughput.
4Productivity
If data is loaded into hash lanes in parallel, then hashing performance is improved, but coordination complexity increases
Solution Approach 1:
Data is pre-loaded into the lane buffers before the actual hashing computation begins. The interface stage prepares the input data by distributing it to appropriate lanes and initializing the lane buffers with the necessary data segments. This preliminary action ensures that when hashing starts, all lanes have their data ready and can operate in synchronized parallel fashion without requiring complex real-time coordination during the computation phase.
Data Source
AI summary
Techniques are provided for configuring and operating hardware to sustain real-time hashing throughput. In an embodiment, during a first set of clock cycles, a particular amount of data items of a first data column are transferred into multiple hash lanes. During a second set of clock cycles, the same particular amount of data items of a second data column are transferred into the hash lanes. The transferred data items of the first and second data columns are then processed to calculate a set of hash values. When combined with techniques such as pipelining and horizontal scaling, the loading, hashing, and other processing occur in real time at the full speed of the underlying data path. For example, hashing throughput may sustainably equal or exceed the throughput of main memory.