Cassandra SSTable Deduplication via Fixed-Size Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cassandra data files pose challenges for deduplication due to constant changes, reorganization, and reordering, which hinder the identification of redundant data, resulting in low compression factors and inefficient data storage and transfer.
Innovation Solution
Transforming raw Cassandra data files into a format with fixed-sized segments allows for effective deduplication by segmenting at the row level, ensuring consistent fingerprinting and identification of duplicates across nodes, even with data reordering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If deduplication is performed on raw Cassandra data files, then storage space is reduced, but deduplication effectiveness is poor due to constant data reorganization and reordering
Solution Approach 1:
The patent segments Cassandra data files into fixed-size chunks with consistent boundaries, independent of internal data reorganization. By dividing files into uniform segments rather than relying on variable data boundaries, the system ensures that identical data segments produce matching fingerprints even when data is reorganized within the file structure.
Solution Approach 2:
The patent changes the parameter of data segmentation from variable (based on data content boundaries) to fixed (uniform chunk sizes). This parameter change ensures consistent fingerprinting across backup operations despite Cassandra's internal data reorganization, compaction, and reordering activities.
2Adaptability or versatility
If data is constantly reorganized and reordered in Cassandra files, then data flexibility and scalability are improved, but identification of redundant data becomes difficult
Solution Approach 1:
The patent introduces fixed-size segment boundaries as an intermediary layer between the variable Cassandra data structure and the deduplication process. These consistent segment boundaries act as a mediator that allows redundant data identification without being affected by internal data reorganization, compaction, or reordering operations.
Solution Approach 2:
By segmenting data into fixed-size chunks with consistent boundaries, the patent creates stable identification units that remain recognizable across different backup operations. This segmentation approach allows the system to identify redundant data even when Cassandra reorganizes or reorders data within the files.
3Reliability
If SSTables are copied from all nodes for backup, then data completeness is ensured, but network traffic and storage requirements increase
Solution Approach 1:
The patent changes the backup parameter from copying entire SSTables to copying only unique fixed-size segments. By using consistent segment boundaries and fingerprint-based deduplication, the system identifies and transfers only unique segments across the network, dramatically reducing network traffic while ensuring data completeness through comprehensive segment coverage.
Solution Approach 2:
Instead of copying complete SSTables (excessive action), the patent performs partial copying of only the unique segments required for data reconstruction. This partial action approach reduces network traffic and storage requirements while maintaining data completeness, as all unique segments are captured and duplicates are eliminated through deduplication.
Data Source
AI summary
Cassandra SSTable data is transformed to provide data rows that are a consistent size such that data in each row has a length that is contained within a selected fixed sized kilobyte segment for deduplication. Tables of a Cassandra cluster node are translated in parallel to JSON format using Cassandra SSTableDump and the table rows are parsed to provide data rows corresponding to the data in each table row. Each row of data is padded with a predictable pattern of bits such that the data row has a length corresponding to the selected fixed segment size and has boundary locations that correspond to multiple of the selected segment size. Since each row of data starts on a segment boundary, duplicate rows of data will be identified wherever they move within a table.