Storage Accelerator Controller for Preprocessed Data Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing, each node must preprocess raw data, leading to wastage of computational resources and energy, as preprocessed data is not shared efficiently across the system, with nodes often performing redundant processing.
Innovation Solution
A storage accelerator system that includes a controller, cache memory for preprocessed data, and a larger accelerator memory for unprocessed data, which converts data on demand, using an address translation table to optimize data access and reduce redundant processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If each node performs preprocessing of raw data independently, then each node can access data in its local memory, but computational resources and energy are wasted due to redundant processing
Solution Approach 1:
The system performs preprocessing operations in advance at a central location and stores the preprocessed data in a shared memory accessible by all nodes. This eliminates the need for each node to perform redundant preprocessing, thereby reducing energy consumption while maintaining fast data access.
Solution Approach 2:
A shared memory structure acts as an intermediary between the raw data storage and the compute nodes. This intermediary contains preprocessed data that nodes can access directly, avoiding the need for each node to independently process raw data and reducing overall energy consumption.
2Loss of energy
If preprocessed data is stored in a shared location accessible by all nodes, then redundant processing is eliminated, but access time may increase due to shared resource contention
Solution Approach 1:
The shared memory is segmented into multiple caches at different levels (L1, L2, L3) with different access speeds. Frequently accessed preprocessed data is stored in faster caches closer to the compute nodes, while less frequently accessed data is stored in slower but larger shared memory, optimizing the balance between access time and energy efficiency.
Solution Approach 2:
The memory hierarchy introduces an additional dimension of spatial organization, arranging memory structures in a hierarchy from fast local caches to slower shared memory. This dimensional arrangement allows the system to provide fast access for frequently used data while maintaining energy efficiency for less frequently used data.
3Adaptability or versatility
If a large shared memory is allocated for preprocessed data, then all nodes can benefit from shared resources, but the memory size and complexity increase
Solution Approach 1:
Different portions of the shared memory system have different characteristics - local caches provide fast access for frequently used data, while remote shared memory provides large capacity for less frequently used data. Each node has customized cache configurations based on its specific access patterns, optimizing the balance between adaptability and complexity.
Data Source
AI summary
A data storage system has a CPU data bus for reading and writing data to data accelerators. Each data accelerator has a controller which receives the read and write requests and determines whether to read or write a local cache memory in preprocessed form or an attached accelerator memory which has greater size capacity based on entries in an address translation table (ATT) and saves data in a raw unprocessed form. The controller may also include an address translation table for mapping input addresses to memory addresses and indicating the presence of data in preprocessed form.