Data Locality Erasure Coding for Real-Time Multidimensional Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current edge systems face challenges in analyzing large-sized multidimensional data such as still image/moving image data in real-time due to poor capacity efficiency in mirror configurations and low reading performance with Erasure Coding, which requires inter-node communication and increases latency.
Innovation Solution
Implementing data locality Erasure Coding (EC) processing within a Hyper-Converged Infrastructure (HCI) that writes data chunks to a single node and generates parities on other nodes, allowing for real-time analysis without inter-node communication and enhancing capacity efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a mirror configuration is used to improve performance, then reading speed is improved, but storage capacity efficiency deteriorates
Solution Approach 1:
The patent segments data into multiple chunks and distributes them across different nodes using Erasure Coding. Instead of duplicating entire datasets in mirror configurations, the system divides data into N chunks and stores them across N different nodes, allowing reconstruction of original data from any M chunks (where M < N). This segmentation approach reduces redundant storage while maintaining reading performance.
2Quantity of substance
If EC configuration is used to improve capacity efficiency, then storage capacity efficiency is improved, but reading performance deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing parity information during the data writing phase. When data is written to the system, erasure coding is performed in advance to generate parity chunks, which are stored alongside data chunks across multiple nodes. This preliminary encoding eliminates the need for complex real-time calculations during read operations, allowing fast data reconstruction without inter-node communication overhead.
3Quantity of substance
If data is distributed across multiple nodes to improve capacity efficiency, then storage capacity efficiency is improved, but inter-node communication is required which increases latency
Solution Approach 1:
The patent implements local quality by ensuring that each node stores both data chunks and parity information locally. This allows any single node to independently reconstruct original data using only locally stored information, eliminating the need for inter-node communication during read operations. The system achieves this by distributing different portions of the erasure-coded data (data chunks and parity) across nodes in a manner that makes each node self-sufficient for data reconstruction.
Data Source
AI summary
Every time a node computer receives multidimensional data from a data source, the node computer which has received the multidimensional data in a computer system: writes the multidimensional data; reads the multidimensional data; analyzes the read multidimensional data; and outputs a result of the analysis. Such writing of the multidimensional data is data locality EC processing (Erasure Coding with data locality). The data locality EC processing is to: write all of one or more data chunks constituting the multidimensional data to the node computer; and write a parity of the data chunk, with respect to each of the one or more data chunks, to a node computer(s) other than the node computer.