86 results about "Inode" patented technology

Techniques are described for a hierarchical caching mechanism enabling efficient cross-region replications. In some embodiments, replication-related information (e.g., key-value pairs) is stored in a particular layout in a binary tree (B-tree) of a file system for replication processing. A hierarchy of caches storing a first type of information (e.g., crypto keys associated with iNodes) may be arranged to match the particular layout in the B-tree to enable efficient parallel processing of a second type of information (e.g., files, file data, or symbolic links), where the replication-related information in the B-tree is partitioned into multiple key ranges for parallel processing. In some embodiments, the caches in different hierarchies may be shared by different parallel-processing key ranges and replication jobs in a file system.

The invention discloses a file hiding method and system based on an ext4 file system, and the method comprises the steps: obtaining a target file and metadata information of a directory where the target file is located according to a target file path needing to be hidden, and distributing a new mapping item in an internal mapping table of the ext4 file system for recording; creating a new hard link in a directory where the target file is located, and associating the new hard link with an inode node for storing metadata information of the target file; reopening the ext4 file system, and deleting the directory entry of the target file, so that the target file cannot be found in the ext4 file system; and re-mounting the ext4 file system, refreshing the Linux kernel cache, and recording the new hard link, the original name of the target file, the inode node number of the target file and the file type into a new mapping item. According to the invention, data security can be ensured, and efficient hidden storage of files is realized.

The invention relates to a distributed file system recycle bin performance optimization method, which comprises the following steps: in the execution process of an unlink interface of a Linux file system, sequentially completing recycle bin function starting judgment, file deletability verification and recycle bin storage suitability judgment, and obtaining metadata of a target deleted file parent directory from a server; when a file system is created, a global unified root recycle bin directory is synchronously generated, and when a file is deleted, a corresponding recycle bin subdirectory is created in a target MDS based on a parent directory inode number; an atomic operation is executed through a Linux standard rename interface, the file is moved to a corresponding recycle bin subdirectory, and the operation is limited to the interior of a single MDS; when the file is recovered, extracting an inode number of a parent directory, and iteratively querying metadata to restore an original complete path; and during batch recovery, if the original catalogue does not exist or needs to be recovered to a new catalogue, directly and integrally moving the catalogue corresponding to the recycle bin, otherwise, executing recovery operation one by one. The file deletion and recovery performance is remarkably improved, the resource overhead is reduced, and the data consistency is guaranteed.

The embodiment of the invention provides a metadata management method and related equipment, which are used for reducing read-write operation of metadata on a PM (Program Management) so as to improve the reliability of single writing of the metadata. The method provided by the embodiment of the invention comprises the following steps: generating a file creation package and a data updating package in the metadata single writing file system; after the file creating package and the data updating package are subjected to crash consistency processing, the file creating package and the data updating package are written into a persistent memory managed by the metadata single-writing file system, a data storage address of the user data is determined, and expanded index metadata for indexing the user data in the data updating package is determined according to the data storage address; analyzing the file creating package and the data updating package to establish a file index node through the file index node number, and establishing a mapping relation between the data storage address and the file index node number by expanding index metadata; and inserting the file index node corresponding to the user file into the index node table in the memory to complete access to the user file.

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.

User process to directly access a file in a file system. A user process first opens a file in the file system for access. In the process of opening the file, a file handle for the file is returned to the user process by an operating system kernel. The user process then makes a read request to a special function unit for one or more blocks of the file in the file system using the file handle. In response, the special function unit, which is coupled to the processor, bypasses the operating system kernel and returns the requested data directly to the user process. A write by the user process may be refused by the computer system or allowed on a selective basis based on a flag in a file system inode corresponding to the block.

This invention discloses a method and apparatus for hierarchical file data storage, comprising recursively traversing the size and type attributes of all files, and prioritizing the migration of files to the target layer of the file system based on the file attribute type. This hierarchical file data storage method calculates the migration score for each file by comprehensively considering file access frequency, the previous migration score, and an adjustment coefficient. The adjustment system adjusts the file based on its size and type attributes. This method of calculating migration scores has a wider range of applications, resulting in more reasonable and accurate file migration, while also improving the access performance of the hierarchical file system storage system. When migrating across file system layers, this method maintains a unified business IO access path for files, eliminating the need to create linked inodes to redirect to the migrated file path, thereby reducing resource consumption, minimizing IO amplification during file read / write operations, and reducing system complexity.

This invention proposes an annotation file system, device, and storage medium. The annotation file system adds annotations to files for file management. The annotation file system manages annotations within the system and includes: a file module for managing and allocating disk block information and inodes used by files; a tagging module for adding annotations to files and their descriptions; a storage module for storing the tagged information on disk; a file operation module for handling CRUD operations on file content and descriptions; and an annotation operation module for handling CRUD operations on data tags. This invention associates files and their descriptions by first having files and their descriptions, then adding annotations to them. It transforms directory structure information into data and uses a database approach to manage files, improving file query efficiency and unifying the file system and database system.

File names and inodes are indexed directly in a filesystem namespace consisting of a layered set of key value store (KVS) database logical tables. The KVS is used directly for organizing file and directory metadata on a cluster file system instead of traditional Btree structures. Embodiments provide file deletion in the file system namespace, and other features include snapshot deletion and cleaning the namespace. A name hash index is used to speed filename lookup in the file system namespace. Two key types of INODE and DIRENT are used to encode namespace metadata using self-identify variable length records in the namespace. Point-in-time snapshot copies are also formalized using KVS database operations as well operations on sets of snapshots for operations including garbage collection and data replication.

One example method includes monitoring a file access request made by a process, searching, in a pointer array that corresponds to the process, for a pointer to the file that was requested by the process, when the pointer is found in the pointer array, incrementing a score associated with the process, performing an anomaly detection check, and when an anomaly is detected, blocking access to the file by the process, and when no anomaly is detected, updating the pointer array to include a pointer to the file to which access was requested by the process.

A system for managing the storage of metadata includes a client device, a network, and two or more file system servers, including a first file system server and a second file system server. The first file system server is programmed to: receive a request for a file system object from the client device, in response to the request: make a determination that the second file system server is to manage the file system object, based on the determination, generate an index node (I-node), and send the I-node to the second file system server. The I-node includes an object type field corresponding to a type of the file system object, a server base index field corresponding to the second file system server, and a directory index field corresponding to an index in a file system object metadata of the second file system server.

The embodiment of the invention provides a metadata log replay processing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a metadata log event, performing local verification on the metadata log event, if the metadata log event is a non-local operation, extracting a metadata field, comparing the metadata field with a preset local index node, and performing data processing according to an operation type when the creation time of the metadata field is updated; updating a file version number based on a preset version incremental rule, and generating dimension control information in combination with the creation time; according to the dimension control information, the data processing process is packaged into a transaction of a preset key value database to be executed, and whether the instruction fails or not is detected; and if the execution result and the dimension control information are successfully written, the preset local index node is updated, and if the execution result and the dimension control information are failed, the operation is revoked in a reverse order and recovered to a transaction pre-encapsulation state. The method is used for guaranteeing the atomicity of metadata log replay, avoiding data residues, solving the problem of metadata synchronization inconsistency and improving the reliability of metadata log replay.

The present disclosure provides an operation processing method and device for a file system, equipment and medium, relating to the technical field of computers, in particular to the technical field of cloud storage. The specific implementation scheme is: in response to a creation request of a target object, determining a parent directory to which the target object belongs in a file system directory tree, wherein the target object includes a directory or a file; creating a directory entry for the target object in the parent directory, and returning a message to the user layer that the target object is created successfully, wherein the directory entry at least includes an inode of the parent directory and metadata information of the target object; in response to a target operation of changing the target object, judging whether an inode of the target object exists; if not, creating the inode of the target object according to the inode of the parent directory and the metadata information.

The embodiment of the application discloses a file deduplication method for terminal equipment, a terminal equipment and a storage medium, which are used for eliminating redundant files generated by a user when using the terminal equipment and reducing the time required by the deduplication process. The method comprises the following steps: obtaining a write request; obtaining a foreground hash value and metadata of a write page according to the write request; obtaining idle information of the terminal equipment and determining a trigger timing; when the trigger timing is met, calculating a file identical to the foreground hash value, and outputting inode information of the file identical to the foreground hash value; calculating a background hash value according to the foreground hash value, the metadata and the inode information; and using a hard link method to deduplicate the same file in the background hash value.

Improving extent-based read performance using pre-fetches by utilizing synthesized sequential extent files in a deduplication storage system. Synthesized content for the extents is created on a server. The synthesized content is headed by a content handle and stored in a stateful context data structure for lookup by the client. This enables an extent read workflow to be supported on a storage system without an added cache required to achieve sufficient random read performance by synthesizing content into a inode-less file. It improves extent reads at random offsets by synthesizing the extents to leverage the sequential restore performance, and leverages delta files to improve read performance for extent reads.

This disclosure relates to methods, apparatus, and program products for managing indexes in a storage system. In one method, a purge period is received for clearing expired events in a storage system, the storage system comprising multiple events in a data stream. Based on the purge period, time shards for managing the indexes in the storage system are determined. In response to determining that the occurrence time of a first event entering the storage system from the data stream is within the time shard, a sharded index node is created for the time shard, the sharded index node including the index node of the first event in the storage system. The sharded index node is added to the indexes of the storage system. Corresponding apparatus and computer program products are provided. Using exemplary implementations of this disclosure, indexes of a large number of events entering the storage system can be managed according to the occurrence time of the events. Furthermore, the storage system can be queried and updated accurately and efficiently.

This invention discloses a data synchronization method and related components for a distributed file system, relating to the field of data replication. The method includes acquiring a file replication packet sent by the master end, wherein the file replication packet includes the file content and metadata information of files under each inode of the distributed file system. Then, each file content is copied to a corresponding copy file on the slave end. Furthermore, the basic attributes of the copied files are set to the metadata information corresponding to the file content stored within the copied files. Therefore, this application, when synchronizing files from the master end to the slave end, not only copies the file content but also the file attributes, i.e., the metadata information. Thus, there is no need to manually set file attributes for the copied files on the slave end, improving the reliability and efficiency of data replication.

The application discloses a local file system management method, electronic equipment and a readable storage medium, relates to the technical field of file management, each object in the local file system comprises a plurality of file storage areas storing index node data, target file metadata of a target file is stored in a file storage area corresponding to idle index node data obtained based on an idle index node queue, and a target directory entry is constructed according to a target file name, a target file type and a target index node number; a target object corresponding to the target file name is determined based on a linear hash function; if the target file name cannot be found in the target object based on a multi-path cuckoo hash function, and no directory entry slot in an idle state exists in the target object, the target object is split based on the linear hash function to create an idle directory entry slot required to be inserted in the target object. The application solves the problem of poor expansion convenience of the local file system in the cloud storage environment.

The invention provides a file system-oriented capacity expansion method and device, and relates to the field of data processing, in particular to the field of artificial intelligence. According to the specific implementation scheme, a plurality of file sets contained in the file system are determined, wherein the file sets comprise a root file set created by the system in a default mode and a user file set created by a user autonomously; collecting index node related information corresponding to each file set, and for any file set, judging whether the file set has an expansion demand related to the index node based on the index node related information; if the file set has the capacity expansion requirement, generating a capacity expansion task corresponding to the file set in combination with the index node related information; and storing the capacity expansion tasks into the corresponding task queues, and executing the capacity expansion tasks according to the task sequence in the task queues.

The invention relates to a file processing method and device, a storage medium and electronic equipment, and relates to the technical field of data processing.The method comprises the steps that firstly, in response to a file copying instruction, a first inode structural body corresponding to a source file in an F2FS file system is obtained, and a second inode structural body corresponding to a target file in the F2FS file system is obtained; then, file index data of the source file are obtained from the first inode structural body through query of the array elements; copying the file index data to an index area in the second inode structural body, so that the address of the data block of the target file index is consistent with the address of the data block of the source file index; in response to a processing instruction for a to-be-processed file, the to-be-processed file is processed based on the first inode structural body or the second inode structural body, and the to-be-processed file comprises a source file or a target file. By applying the technical scheme of the invention, the space of the storage medium is saved, and meanwhile, the file processing speed in the F2FS file system is greatly improved.

The invention provides a file metadata storage method, device and equipment, and relates to the technical field of databases and file systems. The method comprises the steps that an index node number of a first file is generated into a combination of a resource group identifier, a metadata group identifier and a use position, a resource group comprises a first area, the first area comprises a plurality of metadata groups, each metadata group comprises a plurality of blocks, each block is used for storing metadata of a plurality of files, and each block is used for storing metadata of a plurality of files; using the position to indicate the sequence of storing the file metadata in the block, wherein N, M and X are integers greater than or equal to 2; and storing the file metadata of the first file according to the resource group, the metadata group and the use position indicated by the index node number. According to the method, the storage space requirement of the file metadata can be reduced.

The present application provides a data storage method, apparatus and device. The method comprises: acquiring a data storage request, the data storage request comprising directory information; determining a target index structure on the basis of the directory information, the target index structure comprising multi-level index nodes corresponding to multi-level directories in the directory information, wherein for each index node, if the index node is not the last-level index node among the multi-level index nodes, metadata of the index node comprises storage location information of a next-level index node of the index node, and if the index node is the last-level index node among the multi-level index nodes, the metadata of the index node comprises storage location information of data to be stored; for each index node, storing metadata of the index node on the basis of storage location information of the index node; and on the basis of storage location information of data to be stored, storing the data to be stored. The solution of the present application enables interoperability between an object storage protocol and a distributed file storage protocol.

The invention discloses a file processing method and device based on a distributed file system, equipment and a medium. When a client receives an access request of a user for a target file, index node information of the file is obtained from a metadata server of the distributed file system; the client side inquires a mapping table maintained locally in advance according to the first user hash information, and the table records the corresponding relation between the user hash value of the local system and the user identifier. And replacing the first user identifier in the index node information with the queried second user identifier to generate updated index node information. And the updated information is provided for a local operating system, and the operating system performs access permission control based on the mapped user identifier, so that users with the same user name have consistent file access permissions on different clients, and the permission problem caused by inconsistent user identifiers is improved. The file processing method and device can be widely applied to the technical field of file processing based on the distributed file system.

The application discloses a unified file transparent access method and device based on heterogeneous backend segmented mapping and a storage medium, and relates to the technical field of data processing. The method segments a logical file according to a segmented configuration rule and sets a capacity threshold, specifies a main segment, creates a main file carrying a logical file identifier in a corresponding backend of the main segment, generates a physical file name with unique pointing based on an inode number of the main file, a backend path of a subsequent segment and a segment serial number, and determines a logical offset interval according to the capacity threshold of each segment, thereby establishing a mapping relationship between a logical address space and a physical storage segment. When a vector read / write request is received, the logical file identifier and the pre-constructed segmented mapping relationship can be directly used to accurately locate the segment where target data is located and the corresponding heterogeneous backend, and when a non-vector read / write request is received, metadata in the main file is directly operated, so that the access response efficiency is improved.

The application discloses an inode maintenance method of a file system and an electronic device, and relates to the technical field of computers, and comprises the following steps: when it is monitored that a file or a directory is created, triggering a packet filter function pre-mounted in a kernel state program to obtain a target device number corresponding to a target file system; taking the target device number as a key, querying a current inode usage, an alarm threshold and a growth rate threshold from a packet filter mapping table; if the current inode usage is less than the alarm threshold, calculating a node growth rate according to the current inode usage and a historical inode usage; when the node growth rate exceeds a preset growth rate threshold, sending context information when the packet filter function is triggered to a user state program, so as to clean up the inode according to a preset node cleaning strategy and the context information. The application can improve the efficiency of inode release and reduce the consumption of system resources.

The application provides a unified metadata layer hybrid storage method and system, the method comprises receiving a storage access request of each node, parsing the protocol type and the target data path corresponding to the request; based on the preset directory table, the index node table and the object table, the metadata operation corresponding to the protocol type is executed; the metadata operation is encapsulated as an atomic transaction submission, and the corresponding records of the directory table, the index node table and the object table are synchronously updated, and the read and write operations of the target data are completed. The application not only solves the problems of semantic fragmentation, poor consistency and performance bottleneck existing in the traditional hybrid storage system for a long time, but also solves the problems of resource contention, business insecurity and poor cluster stability in the multi-training task parallel scene according to the special needs of the AI large model training scene.

The invention provides a data storage method, device and equipment, and the method comprises the steps: obtaining a data storage request which comprises directory information; determining a target index structure based on the directory information, wherein the target index structure comprises multiple levels of index nodes corresponding to multiple levels of directories in the directory information; for each index node, if the index node is not the last-level index node of the multi-level index nodes, the metadata of the index node comprises the storage position information of the next-level index node of the index node, and if the index node is the last-level index node of the multi-level index nodes, the metadata of the index node comprises the storage position information of the next-level index node of the index node. The metadata of the index node comprises storage position information of to-be-stored data; for each index node, storing the metadata of the index node based on the storage position information of the index node; and storing the to-be-stored data based on the storage position information of the to-be-stored data. Through the scheme of the invention, the intercommunication between the object storage protocol and the distributed file storage protocol is realized.

The application provides a method, device and equipment and computer storage medium related to a data processing method, which relates to the technical field of data processing and is used to improve the system response speed of query outbound data. The method comprises the following steps: acquiring, based on a rule identifier, a data extraction rule and a data loading rule corresponding to the rule identifier from a configuration file; acquiring, in a database, original data at a data storage address included in the data extraction rule; splitting the original data to obtain corresponding index data and field data; storing the index data to each index node in a target data set with a tree structure indicated by the data loading rule, and storing the field data to a leaf node in the target data set; determining, based on a query index, a target index node in the target data set corresponding to the query index, and returning all target field data corresponding to the target index node to a query instruction source.

The invention discloses a new file creation acceleration method based on an FAT file system, and the method comprises the steps: converting the file names of all existing files into hash values, generating a unique index node, constructing a red-black tree containing a linked list for each directory, and combining the red-black tree into a nested directory tree according to the parent-child relationship of the directories in a mounting stage of the FAT file system. According to the method, the target parent directory red-black tree is quickly searched through the hash value, whether name duplication exists or not is judged by comparing file names in combination with a linked list, so that the time complexity of name duplication search is reduced to O (logN) from O (n) of an existing mechanism, the more the files are, the more significant the search efficiency advantage is, and the new file creation speed is greatly increased.