The invention discloses an HDFS (Hadoop
Distributed File System) intelligent hierarchical storage method and
system based on multi-dimensional popularity
perception and
tape library integration, and belongs to the technical field of distributed file systems, and the method comprises the following steps: expanding HDFS
metadata to construct a unified
namespace; acquiring data block access features and dynamically dividing hot, warm and cold data by using a K-means clustering
algorithm; executing a self-adaptive grading strategy: adding a copy to hot data and storing in an SSD (
Solid State Disk), maintaining a default copy to warm data and storing in a disk, and migrating cold data to a
tape library and storing by adopting an
erasure code; scattered storage and
fault tolerance of relevance
perception are realized through a
message queue;
metadata persistence and periodical re-evaluation are carried out, and automatic migration and dynamic adjustment of data among the SSD layer, the disk layer and the
tape library layer are achieved. By adopting the method and the
system, the storage strategy and the
data access value are accurately matched, the storage cost is finally reduced, and the
system I / O performance and the
resource utilization rate are improved.