Data Compression in Distributed Datastores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed datastores like vSAN face significant CPU and bandwidth costs due to the need for individual compression of data across multiple layers, especially when mirroring is involved, leading to inefficient use of resources.
Innovation Solution
Implementing an efficient compression process that compresses data in the performance tier, generates metadata, and writes it to the performance tier, then flushes it to the capacity tier as full stripes, using RAID 1 for the performance tier and RAID 6 for the capacity tier, reducing CPU and bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is compressed individually for each layer and mirror copy in distributed datastores, then data storage efficiency is improved, but CPU cost and bandwidth consumption increase significantly
Solution Approach 1:
The patent merges multiple individual compression operations into a single compression process. Instead of compressing data separately for each layer and mirror copy, the system compresses the data once and then distributes the compressed data to multiple destinations (performance tier, capacity tier, and mirror copies), eliminating redundant CPU-intensive compression operations.
Solution Approach 2:
The patent performs data compression in advance during the write operation before data needs to be transferred or stored in multiple locations. By pre-compressing the data when it first enters the system, the compression result can be reused across multiple tiers and copies, avoiding repeated compression of the same data.
2Quantity of substance
If data is compressed individually for each layer and mirror copy in distributed datastores, then data storage efficiency is improved, but network and disk bandwidth consumption increase
Solution Approach 1:
The patent combines multiple data transfer operations into a single transfer of compressed data. Instead of transferring uncompressed data separately to each layer and mirror copy, the system transfers the compressed data once to the capacity tier and uses this same compressed data for all mirror copies, significantly reducing network and disk bandwidth consumption.
3Use of energy by moving object
If uncompressed data is transferred and written directly to storage, then CPU cost is reduced, but network and disk bandwidth requirements increase
Solution Approach 1:
The patent performs data compression in advance during the write operation before data needs to be transferred or stored in multiple locations. By pre-compressing the data when it first enters the system, the compression result can be reused across multiple tiers and copies, avoiding repeated compression of the same data.
Data Source
AI summary
A method for performing write operations on a set of one or more physical disks of a set of one or more host machines is provided. The method receives a data block to write on at least one physical disk in the set of physical disks and generates a first set of one or more compressed sectors based on the received data block. The method writes (i) a first entry having a first header and the first set of compressed sectors to a data log that is maintained in a cache, and (ii) the first set of compressed sectors to a bank in memory. The method further determines if a size of data including compressed sectors in the bank satisfies a threshold, and when the size of data in the bank satisfies the threshold, writes the data to the at least one physical disk in the set of physical disks.