Collocated Stats-Block for Distributed File System Statistics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale distributed file systems face inefficiencies in statistical data collection and aggregation due to the separation of statistical data from data records, leading to complex network communication and resource-intensive operations.
Innovation Solution
Implementing a novel statistical data block (Stats-block) collocated with data blocks in each data node, allowing for configurable frequency of data collection and leveraging existing mechanisms like Hadoop Archive or Major Compaction, to store statistical data in memory close to the data, facilitating fast retrieval and aggregation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If statistical data are stored separately from data records in a distributed file system, then data access transparency and scalability are improved, but system complexity and network communication overhead increase
Solution Approach 1:
The patent merges statistical data and data records into a unified distributed data structure where statistical data are stored alongside their corresponding data records in the same distributed file system blocks. This eliminates the need for separate statistical data tables and reduces system complexity while maintaining data access transparency.
2Measurement precision
If statistical data are collected via full table scans, then measurement accuracy is improved, but computing cost and network bandwidth consumption increase
Solution Approach 1:
The patent implements preliminary action by collecting and maintaining statistical data incrementally as data records are inserted, updated, or deleted in the distributed file system. Statistical data are updated in real-time alongside data records, eliminating the need for computationally expensive full table scans while maintaining measurement accuracy.
3Reliability
If statistical data are stored in a separate external database, then data security and isolation are improved, but network communication complexity and retrieval time increase
Solution Approach 1:
The patent merges statistical data storage with the distributed file system itself, storing statistical data in the same data blocks as their corresponding data records. This eliminates network communication overhead between separate databases while maintaining data security through the existing distributed file system security mechanisms.
4Adaptability or versatility
If OLAP functions are implemented with separate statistical data tables, then analytical processing capability is improved, but system maintenance complexity increases
Solution Approach 1:
The patent merges OLAP statistical data processing capabilities directly into the distributed file system architecture. By storing statistical data alongside data records and providing programmatic access to this collocated data, the system enables OLAP functions while simplifying maintenance through a unified data management approach.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
System and method for storing statistical data of records stored in a distributed file system. In one aspect a statistical data block is allocated in a memory of a data node for storing statistical data of records stored in a storage disk of the data node. Each data block of the plurality of data blocks in the data node has a respective entry in the statistical data block, which is collocated with data blocks on the data node. Statistical data of records stored in the distributed file system are collected, and written to statistical data block in the memory of the data node.