Sharing Data Fabric for Distributed Flash Coherent Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current disk-based storage systems face bottlenecks in performance and power consumption, particularly in web server farms, where the increased demand for data storage leads to high energy costs and inefficiencies due to the slow access times and rotational speeds of hard disks.
Innovation Solution
A global, shared flash memory system is implemented using a Sharing Data Fabric (SDF) middleware layer that distributes flash memory across multiple nodes, enabling coherent caching and reducing power consumption by leveraging DRAM caches for transient data storage while ensuring data consistency through SDF threads and APIs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If disk-based storage systems are used to meet increasing data storage demand, then storage capacity is improved, but power consumption and access performance deteriorate
Solution Approach 1:
The storage system is segmented into multiple nodes, each with its own flash memory and DRAM cache. This distributed architecture allows the system to scale storage capacity by adding nodes while each node operates independently, reducing the power consumption overhead of a single large disk-based system.
Solution Approach 2:
The patent replaces mechanical rotating hard disks with solid-state flash memory devices. This substitution eliminates the mechanical rotation and associated power consumption, while providing faster access times. The flash memory is complemented by DRAM cache to further enhance performance.
2Productivity
If additional servers are added to handle increased user workload, then processing capacity is improved, but database access bottlenecks worsen
Solution Approach 1:
The database is distributed across multiple nodes in the storage system. Each node can service requests independently, eliminating the single-point bottleneck of centralized database access. Multiple servers can access different portions of the database simultaneously, maintaining high processing capacity without access bottlenecks.
Solution Approach 2:
The SDF middleware layer acts as an intermediary between servers and the distributed flash memory storage. It manages data distribution, caching policies, and coherence protocols, allowing servers to access data efficiently without direct coordination overhead, thus maintaining high processing capacity and access speed.
3Speed
If database replication is implemented to reduce access bottlenecks, then read performance is improved, but data coherency and system complexity worsen
Solution Approach 1:
The SDF middleware serves as an intermediary that automatically manages data replication and coherence across nodes. It implements caching protocols and coherence mechanisms transparently, improving read performance through local caches while hiding the complexity of distributed coherency management from applications and servers.
Solution Approach 2:
The distributed flash memory system with SDF enables nodes to serve read requests from their local DRAM caches without requiring complex inter-node coordination for every access. The system self-manages cache coherence through the middleware, reducing the apparent complexity for individual nodes while maintaining overall data consistency.
4Use of energy by moving object
If flash memory is used to reduce power consumption, then energy efficiency is improved, but write speed and coherency management worsen
Solution Approach 1:
Data is preliminarily cached in fast DRAM memory before being written to flash memory. This allows write operations to be acknowledged quickly based on the fast DRAM cache, while the actual flash write occurs in the background. This preliminary caching action masks the slower flash write speed while maintaining energy efficiency.
Solution Approach 2:
The SDF middleware acts as an intermediary that manages the write buffer and coherence protocols between multiple nodes accessing flash memory. It handles the complexity of ensuring data coherency across nodes despite the asynchronous nature of flash writes, effectively masking the write speed limitation from applications.
Data Source
AI summary
A Sharing Data Fabric (SDF) causes flash memory attached to multiple compute nodes to appear to be a single large memory space that is global yet shared by many applications running on the many compute nodes. Flash objects stored in flash memory of a home node are copied to an object cache in DRAM at an action node by SDF threads executing on the nodes. The home node has a flash object map locating flash objects in the home node's flash memory, and a global cache directory that locates copies of the object in other sharing nodes. Application programs use an applications-programming interface (API) into the SDF to transparently get and put objects without regard to the object's location on any of the many compute nodes. SDF threads and tables control coherency of objects in flash and DRAM.