56 results about "Inode" patented technology

The invention relates to a distributed file system recycle bin performance optimization method, which comprises the following steps: in the execution process of an unlink interface of a Linux file system, sequentially completing recycle bin function starting judgment, file deletability verification and recycle bin storage suitability judgment, and obtaining metadata of a target deleted file parent directory from a server; when a file system is created, a global unified root recycle bin directory is synchronously generated, and when a file is deleted, a corresponding recycle bin subdirectory is created in a target MDS based on a parent directory inode number; an atomic operation is executed through a Linux standard rename interface, the file is moved to a corresponding recycle bin subdirectory, and the operation is limited to the interior of a single MDS; when the file is recovered, extracting an inode number of a parent directory, and iteratively querying metadata to restore an original complete path; and during batch recovery, if the original catalogue does not exist or needs to be recovered to a new catalogue, directly and integrally moving the catalogue corresponding to the recycle bin, otherwise, executing recovery operation one by one. The file deletion and recovery performance is remarkably improved, the resource overhead is reduced, and the data consistency is guaranteed.

User process to directly access a file in a file system. A user process first opens a file in the file system for access. In the process of opening the file, a file handle for the file is returned to the user process by an operating system kernel. The user process then makes a read request to a special function unit for one or more blocks of the file in the file system using the file handle. In response, the special function unit, which is coupled to the processor, bypasses the operating system kernel and returns the requested data directly to the user process. A write by the user process may be refused by the computer system or allowed on a selective basis based on a flag in a file system inode corresponding to the block.

This invention proposes an annotation file system, device, and storage medium. The annotation file system adds annotations to files for file management. The annotation file system manages annotations within the system and includes: a file module for managing and allocating disk block information and inodes used by files; a tagging module for adding annotations to files and their descriptions; a storage module for storing the tagged information on disk; a file operation module for handling CRUD operations on file content and descriptions; and an annotation operation module for handling CRUD operations on data tags. This invention associates files and their descriptions by first having files and their descriptions, then adding annotations to them. It transforms directory structure information into data and uses a database approach to manage files, improving file query efficiency and unifying the file system and database system.

File names and inodes are indexed directly in a filesystem namespace consisting of a layered set of key value store (KVS) database logical tables. The KVS is used directly for organizing file and directory metadata on a cluster file system instead of traditional Btree structures. Embodiments provide file deletion in the file system namespace, and other features include snapshot deletion and cleaning the namespace. A name hash index is used to speed filename lookup in the file system namespace. Two key types of INODE and DIRENT are used to encode namespace metadata using self-identify variable length records in the namespace. Point-in-time snapshot copies are also formalized using KVS database operations as well operations on sets of snapshots for operations including garbage collection and data replication.

The embodiment of the invention provides a metadata log replay processing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a metadata log event, performing local verification on the metadata log event, if the metadata log event is a non-local operation, extracting a metadata field, comparing the metadata field with a preset local index node, and performing data processing according to an operation type when the creation time of the metadata field is updated; updating a file version number based on a preset version incremental rule, and generating dimension control information in combination with the creation time; according to the dimension control information, the data processing process is packaged into a transaction of a preset key value database to be executed, and whether the instruction fails or not is detected; and if the execution result and the dimension control information are successfully written, the preset local index node is updated, and if the execution result and the dimension control information are failed, the operation is revoked in a reverse order and recovered to a transaction pre-encapsulation state. The method is used for guaranteeing the atomicity of metadata log replay, avoiding data residues, solving the problem of metadata synchronization inconsistency and improving the reliability of metadata log replay.

This disclosure relates to methods, apparatus, and program products for managing indexes in a storage system. In one method, a purge period is received for clearing expired events in a storage system, the storage system comprising multiple events in a data stream. Based on the purge period, time shards for managing the indexes in the storage system are determined. In response to determining that the occurrence time of a first event entering the storage system from the data stream is within the time shard, a sharded index node is created for the time shard, the sharded index node including the index node of the first event in the storage system. The sharded index node is added to the indexes of the storage system. Corresponding apparatus and computer program products are provided. Using exemplary implementations of this disclosure, indexes of a large number of events entering the storage system can be managed according to the occurrence time of the events. Furthermore, the storage system can be queried and updated accurately and efficiently.

This invention discloses a data synchronization method and related components for a distributed file system, relating to the field of data replication. The method includes acquiring a file replication packet sent by the master end, wherein the file replication packet includes the file content and metadata information of files under each inode of the distributed file system. Then, each file content is copied to a corresponding copy file on the slave end. Furthermore, the basic attributes of the copied files are set to the metadata information corresponding to the file content stored within the copied files. Therefore, this application, when synchronizing files from the master end to the slave end, not only copies the file content but also the file attributes, i.e., the metadata information. Thus, there is no need to manually set file attributes for the copied files on the slave end, improving the reliability and efficiency of data replication.

The application discloses a local file system management method, electronic equipment and a readable storage medium, relates to the technical field of file management, each object in the local file system comprises a plurality of file storage areas storing index node data, target file metadata of a target file is stored in a file storage area corresponding to idle index node data obtained based on an idle index node queue, and a target directory entry is constructed according to a target file name, a target file type and a target index node number; a target object corresponding to the target file name is determined based on a linear hash function; if the target file name cannot be found in the target object based on a multi-path cuckoo hash function, and no directory entry slot in an idle state exists in the target object, the target object is split based on the linear hash function to create an idle directory entry slot required to be inserted in the target object. The application solves the problem of poor expansion convenience of the local file system in the cloud storage environment.

The invention provides a file system-oriented capacity expansion method and device, and relates to the field of data processing, in particular to the field of artificial intelligence. According to the specific implementation scheme, a plurality of file sets contained in the file system are determined, wherein the file sets comprise a root file set created by the system in a default mode and a user file set created by a user autonomously; collecting index node related information corresponding to each file set, and for any file set, judging whether the file set has an expansion demand related to the index node based on the index node related information; if the file set has the capacity expansion requirement, generating a capacity expansion task corresponding to the file set in combination with the index node related information; and storing the capacity expansion tasks into the corresponding task queues, and executing the capacity expansion tasks according to the task sequence in the task queues.

The invention provides a file metadata storage method, device and equipment, and relates to the technical field of databases and file systems. The method comprises the steps that an index node number of a first file is generated into a combination of a resource group identifier, a metadata group identifier and a use position, a resource group comprises a first area, the first area comprises a plurality of metadata groups, each metadata group comprises a plurality of blocks, each block is used for storing metadata of a plurality of files, and each block is used for storing metadata of a plurality of files; using the position to indicate the sequence of storing the file metadata in the block, wherein N, M and X are integers greater than or equal to 2; and storing the file metadata of the first file according to the resource group, the metadata group and the use position indicated by the index node number. According to the method, the storage space requirement of the file metadata can be reduced.

The present application provides a data storage method, apparatus and device. The method comprises: acquiring a data storage request, the data storage request comprising directory information; determining a target index structure on the basis of the directory information, the target index structure comprising multi-level index nodes corresponding to multi-level directories in the directory information, wherein for each index node, if the index node is not the last-level index node among the multi-level index nodes, metadata of the index node comprises storage location information of a next-level index node of the index node, and if the index node is the last-level index node among the multi-level index nodes, the metadata of the index node comprises storage location information of data to be stored; for each index node, storing metadata of the index node on the basis of storage location information of the index node; and on the basis of storage location information of data to be stored, storing the data to be stored. The solution of the present application enables interoperability between an object storage protocol and a distributed file storage protocol.

The invention discloses a file processing method and device based on a distributed file system, equipment and a medium. When a client receives an access request of a user for a target file, index node information of the file is obtained from a metadata server of the distributed file system; the client side inquires a mapping table maintained locally in advance according to the first user hash information, and the table records the corresponding relation between the user hash value of the local system and the user identifier. And replacing the first user identifier in the index node information with the queried second user identifier to generate updated index node information. And the updated information is provided for a local operating system, and the operating system performs access permission control based on the mapped user identifier, so that users with the same user name have consistent file access permissions on different clients, and the permission problem caused by inconsistent user identifiers is improved. The file processing method and device can be widely applied to the technical field of file processing based on the distributed file system.

The application discloses a unified file transparent access method and device based on heterogeneous backend segmented mapping and a storage medium, and relates to the technical field of data processing. The method segments a logical file according to a segmented configuration rule and sets a capacity threshold, specifies a main segment, creates a main file carrying a logical file identifier in a corresponding backend of the main segment, generates a physical file name with unique pointing based on an inode number of the main file, a backend path of a subsequent segment and a segment serial number, and determines a logical offset interval according to the capacity threshold of each segment, thereby establishing a mapping relationship between a logical address space and a physical storage segment. When a vector read / write request is received, the logical file identifier and the pre-constructed segmented mapping relationship can be directly used to accurately locate the segment where target data is located and the corresponding heterogeneous backend, and when a non-vector read / write request is received, metadata in the main file is directly operated, so that the access response efficiency is improved.

The application provides a unified metadata layer hybrid storage method and system, the method comprises receiving a storage access request of each node, parsing the protocol type and the target data path corresponding to the request; based on the preset directory table, the index node table and the object table, the metadata operation corresponding to the protocol type is executed; the metadata operation is encapsulated as an atomic transaction submission, and the corresponding records of the directory table, the index node table and the object table are synchronously updated, and the read and write operations of the target data are completed. The application not only solves the problems of semantic fragmentation, poor consistency and performance bottleneck existing in the traditional hybrid storage system for a long time, but also solves the problems of resource contention, business insecurity and poor cluster stability in the multi-training task parallel scene according to the special needs of the AI large model training scene.

The invention provides a data storage method, device and equipment, and the method comprises the steps: obtaining a data storage request which comprises directory information; determining a target index structure based on the directory information, wherein the target index structure comprises multiple levels of index nodes corresponding to multiple levels of directories in the directory information; for each index node, if the index node is not the last-level index node of the multi-level index nodes, the metadata of the index node comprises the storage position information of the next-level index node of the index node, and if the index node is the last-level index node of the multi-level index nodes, the metadata of the index node comprises the storage position information of the next-level index node of the index node. The metadata of the index node comprises storage position information of to-be-stored data; for each index node, storing the metadata of the index node based on the storage position information of the index node; and storing the to-be-stored data based on the storage position information of the to-be-stored data. Through the scheme of the invention, the intercommunication between the object storage protocol and the distributed file storage protocol is realized.

The application provides a method, device and equipment and computer storage medium related to a data processing method, which relates to the technical field of data processing and is used to improve the system response speed of query outbound data. The method comprises the following steps: acquiring, based on a rule identifier, a data extraction rule and a data loading rule corresponding to the rule identifier from a configuration file; acquiring, in a database, original data at a data storage address included in the data extraction rule; splitting the original data to obtain corresponding index data and field data; storing the index data to each index node in a target data set with a tree structure indicated by the data loading rule, and storing the field data to a leaf node in the target data set; determining, based on a query index, a target index node in the target data set corresponding to the query index, and returning all target field data corresponding to the target index node to a query instruction source.

The invention discloses a new file creation acceleration method based on an FAT file system, and the method comprises the steps: converting the file names of all existing files into hash values, generating a unique index node, constructing a red-black tree containing a linked list for each directory, and combining the red-black tree into a nested directory tree according to the parent-child relationship of the directories in a mounting stage of the FAT file system. According to the method, the target parent directory red-black tree is quickly searched through the hash value, whether name duplication exists or not is judged by comparing file names in combination with a linked list, so that the time complexity of name duplication search is reduced to O (logN) from O (n) of an existing mechanism, the more the files are, the more significant the search efficiency advantage is, and the new file creation speed is greatly increased.

This invention provides a method and system for incremental time-series data updates using adaptive hierarchical indexes, belonging to the field of database indexing technology. The method includes: constructing a three-level hierarchical index structure comprising a hot layer, a warm layer, and a cold layer; the hot layer using an in-memory skip list, the warm layer using an LSM tree, and the cold layer using a segmented B+ tree; configuring an incremental merge scheduler to dynamically trigger inter-layer data sinking and index merging based on data volume, query frequency, and write rate; employing a lock-free concurrency strategy to avoid merging blocking foreground read / write operations; embedding time-series feature summaries in index nodes to implement query pruning; and introducing a pre-write log and checkpoint mechanism to ensure atomicity and crash recovery. This invention can maintain stable query latency under continuous high-throughput write conditions.

Migration of information, comprising inode numbers, from source filesystem to destination filesystem can be performed and managed. Migrator component can partition inode subspace of destination filesystem into first inode subspace region, comprising first subgroup of inode slices, and second inode subspace region, comprising second subgroup of inode slices. Provisioner manager component can provision first inode slices of the first subgroup in a first order. Inode numbers associated with user files or directories contained in source slices of the source filesystem can be migrated to and allocated for corresponding first inode slices of the destination filesystem. Provisioner manager component can provision second inode slices of the second subgroup in a second order that can be different from the first order. Inodes associated with inode numbers associated with internal files or directories can be stored in the second inode slices. Source filesystem can be active during the migration process.

This invention discloses a semantic tracing and lazy reclamation method for orphan files in a distributed file system, belonging to the field of distributed file system technology. This method persists directory entry change operations through full-link logging and establishes an independent operation log index for each inode. It monitors changes in the inode hard link count in real time, marks files to be classified as orphan files when the hard link count reaches zero, and triggers the semantic tracing process. It traces back the file's entire lifecycle relationships to complete the classification, matches the corresponding hierarchical reclamation strategy based on the classification results, and generates a read-only snapshot before releasing space when the reclamation trigger conditions are met. This method can accurately distinguish different types of orphan files, reduce resource consumption during the reclamation process, avoid accidental deletion of business data, balance storage space reclamation efficiency and data security, and improve the operational stability of the distributed file system.

Techniques are described for a hierarchical caching mechanism enabling efficient cross-region replications. In some embodiments, replication-related information (e.g., key-value pairs) is stored in a particular layout in a binary tree (B-tree) of a file system for replication processing. A hierarchy of caches storing a first type of information (e.g., crypto keys associated with iNodes) may be arranged to match the particular layout in the B-tree to enable efficient parallel processing of a second type of information (e.g., files, file data, or symbolic links), where the replication-related information in the B-tree is partitioned into multiple key ranges for parallel processing. In some embodiments, the caches in different hierarchies may be shared by different parallel-processing key ranges and replication jobs in a file system.

Migration of information, comprising inode numbers, from source filesystem to destination filesystem can be performed and managed. Migrator component can partition inode subspace of destination filesystem into first inode subspace region, comprising first subgroup of inode slices, and second inode subspace region, comprising second subgroup of inode slices. Provisioner manager component can provision first inode slices of the first subgroup in a first order. Inode numbers associated with user files or directories contained in source slices of the source filesystem can be migrated to and allocated for corresponding first inode slices of the destination filesystem. Provisioner manager component can provision second inode slices of the second subgroup in a second order that can be different from the first order. Inodes associated with inode numbers associated with internal files or directories can be stored in the second inode slices. Source filesystem can be active during the migration process.

A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more of a plurality of storage devices. A plurality of failure resilient address spaces are distributed across the plurality of storage devices such that each of the plurality of failure resilient address spaces spans a plurality of the storage devices. The plurality of computing devices maintains metadata that maps each failure resilient address space to one of the plurality of computing devices. The metadata is grouped into buckets. Each bucket is stored in the backend of a computing device. Data may be migrated from an external file system to the plurality of storage devices using inode stubs to represent directories and files of the external file system. As the contents of the external file system are copied, the inode stubs are replaced with real inodes.

In a data processing method, an electronic device detects that a first file and a second file are duplicate files, with a first index node of the first file pointing to file data of the first file, and a second index node of the second file pointing to file data of the second file. The electronic device generates a target index node and configures the target index node to point to the file data of the first file. The electronic device associates the first index node with the target index node, and separately associates the second index node with the target index node. The electronic device then deletes the file data of the second file.

This invention relates to the field of distributed file system technology, and discloses a method, apparatus, computer device, and storage medium for selecting new source files. Upon receiving a deletion operation instruction, the invention determines the target file to be deleted based on the instruction. When the target file to be deleted is a source file, the first inode number of the source file is obtained from its directory entries. Based on the first inode number, the first metadata of the source file is obtained. Hard link data for at least one hard link file corresponding to the source file is obtained from the first metadata. Based on each hard link data, the directory entry corresponding to each hard link file is determined. Based on the target entry of each hard link file, it is determined whether there are any hard link files not currently used by the client. When at least one hard link file not currently used by the client exists, a new source file can be selected using different methods. This invention can avoid the problem of business processing failures caused by the inability to select a new source file.

The application provides a file system processing method, system, server, storage medium and program product. The method is suitable for a container migration scene. In a container saving stage, a source end reads original device numbers of mounting points of original tmpfs and original inode numbers of files in a source container, and transmits the original device numbers and the original inode numbers to a target end through a container management end. In a container recovery stage, the target end explicitly allocates corresponding original device numbers to created mounting points when the mounting points are created, so that consistency of the device numbers of the tmpfs before and after saving and recovery can be ensured. When files are created in the mounting points, corresponding original inode numbers are explicitly allocated to the created files, so that consistency of inode numbers of the files before and after saving and recovery can be ensured, abnormality / failure caused by inconsistency of the device numbers and the inode numbers of the tmpfs before and after migration can be avoided, and safety and stability of a system are improved.

Embodiments manage file system transaction dependencies in distributed file systems. A transaction log containing log entries is obtained, where each log entry represents a transaction with file system operations, execution times, and associated inodes. Dependency graphs are constructed based on shared inode references between log entries. Each inode stores the last modified transaction log key, which serves as a root into the dependency graph of all dependent transactions. Depth-first search traversal identifies leaf log entries ready for application. These leaf entries are communicated to target file systems for replay. Independent subgraphs are identified and processed in parallel to maximize throughput. In-flight windows with bounded capacity prevent deadlock conditions by ensuring dependencies fit within available buffer space. Log entries from multiple subgraphs may be combined into single communication messages to optimize network efficiency across geographically distributed file system infrastructure.

A computer-implemented method (900) for creating or modifying an image file to produce a new image (520) for implementation as a container (523), wherein the computer-implemented method (900) comprises: Check (907, 909), by a daemon (509), files or directories of a container file system of the image file for changes to the files or directories; in response to checking (907, 909) the container file system containing changes to the files or directories, filtering (607, 913), by the daemon (509), an updated file that has been copied from the image file to a read / write layer; Compare (609, 915), by the daemon (509), a hash value of an original file of the container file system from a lower read-only layer (709) of the image file with a hash value of the updated file; as a response to comparing (609, 915) the hash value of the original file with the hash value of the updated file, where the hash value of the original file and the hash value of the updated file are identical (611, 917) or a difference between the hash value of the original file and the hash value of the updated file exceeds a standard difference value (613, 921), generating (919, 923) by the daemon (509) an overwrite and merge data file, OMD file (620) containing an overwrite block and a merge block; Update (615, 617, 919, 923), by the daemon (509), from overwrite block metadata from the inode metadata (721) of the original file, stored in the lower layer (709) of the image file, to metadata (621) of the updated file; and Remove (615, 617, 919, 923), by the daemon (509), the updated file from the container file system.

A method for advertising a forwarding table index, a node, and a storage medium are provided. The method for advertising the forwarding table index includes: determining, according to first information, a specified path-based bit indexed explicit replication-traffic engineering (BIER-TE) forwarding table index of a current node, where the first information includes a sub-domain (SD) and a bit string length (BSL); and advertising the BIER-TE forwarding table index of the current node to another node, where the BIER-TE forwarding table index of the current node is encoded in a preset encoding mode.