Compute Acceleration Device Offloading Data Health-Check
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for managing data in computing devices are inefficient in terms of performance due to the utilization of computing resources for data generation, storage, and backup, which can impact overall performance and reliability.
Innovation Solution
A data cluster system that includes a non-accelerator pool and data nodes equipped with compute acceleration devices (CADs) for continuous health-checking and reconstruction of data chunks, using storage metadata to identify and regenerate corrupted or inaccessible data, and updating metadata across nodes to ensure data integrity and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If computing resources are utilized for data generation, storage, and backup, then data reliability is improved, but overall system performance deteriorates
Solution Approach 1:
The system segments data into chunks and distributes them across multiple data nodes. Each node independently stores and manages specific chunks, allowing parallel processing and reducing the performance impact on any single node while maintaining overall system reliability through distributed storage
Solution Approach 2:
The patent introduces a dedicated backup and health-check mechanism that acts as an intermediary between data nodes and storage systems. This intermediary layer handles backup operations and health monitoring separately from primary data processing, preventing backup tasks from interfering with main computational workloads
2Reliability
If continuous health-check and reconstruction of data is performed, then data integrity is improved, but computing resource utilization increases
Solution Approach 1:
The system performs preliminary health checks on data chunks before they are actually needed. By proactively identifying and reconstructing corrupted or missing chunks in advance, the system prevents data integrity issues from affecting performance when data is accessed, rather than reacting to problems when they occur
Solution Approach 2:
Each data node autonomously performs health checks on its own stored chunks and can independently reconstruct missing data using erasure coding algorithms. This self-service approach eliminates the need for centralized health monitoring and reconstruction operations, distributing the computational load across all nodes and reducing overall system resource requirements
Data Source
AI summary
A method for managing data includes identifying, by a compute acceleration device (CAD), a first chunk to be health-checked using storage metadata, generating a first chunk identifier using the first chunk, making a determination that the first chunk identifier does not match a second chunk identifier stored in the storage metadata, and in response to the determination: obtaining a plurality of chunks associated with the first chunk, regenerating the first chunk using the plurality of chunks to generate a new first chunk, storing the new first chunk in a data node, wherein the CAD is executing in the data node, updating the storage metadata based on storage of the new first chunk to obtain updated storage metadata, and sending a copy of the updated storage metadata to at least one other CAD in a second data node.