231 results about "Clustered file system" patented technology

Methods for operating a network as a clustered file system is disclosed. The methods involve client load rebalancing, distributed Input and Output (I / O) and resource load rebalancing. Client load rebalancing refers to the ability of a client enabled with processes in accordance with the current invention to remap a path through a plurality of nodes to a resource. Distributed I / O refers to the methods on the network which provide concurrent input / output through a plurality of nodes to resources. Resource rebalancing includes remapping of pathways between nodes, e.g. servers, and resources, e.g. volumes / file systems. The network includes client nodes, server nodes and resources. Each of the resources couples to at least two of the server nodes. The method for operating comprising the acts of: redirecting an I / O request for a resource from a first server node coupled to the resource to a second server node coupled to the resource; and splitting the I / O request at the second server node into an access portion and a data transfer portion and passing the access portion to a corresponding administrative server node for the resource, and completing at the second server nodes subsequent to receipt of an access grant from the corresponding administrative server node a data transfer for the resource. In an alternate embodiment of the invention the methods may additionally include the acts of: detecting a change in an availability of the server nodes; and rebalancing the network by applying a load balancing function to the network to re-assign each of the available resources to a corresponding available administrative server node responsive to the detecting act.

A state management system preserves a state for a cluster of file servers in a cluster file system in the presence of load balancing, failover, and fail-back events. The system provides a file and record locking solution for a clustered network attached storage system running on top of a cluster file system. The system employs a lock ownership scheme in which ownership identifiers are guaranteed to be unique across clustered servers and across various protocols the clustered servers may be exporting. The system supports multi-protocol clustered NAS gateways, NAS gateway server failover and fail-back, and load-balancing architectures. The system further eliminates a need for a lock migration protocol, resulting in improved efficiency and simplicity.

A computing method and system is presented that allows multiple heterogeneous computing systems containing file storage mechanisms to work together in a peer-to-peer fashion to provide a fault-tolerant decentralized highly available clustered file system. The file system can be used by multiple heterogeneous systems to store and retrieve files. The system automatically ensures fault tolerance by storing files in multiple locations and requires hardly any configuration for a computing device to join the clustered file system. Most importantly, there is no central authority regarding meta-data storage, ensuring no single point of failure.

A cluster file system is disclosed. A plurality of disk servers, operating as a single distributed disk server layer, are connected to the one or more physical disks. Each disk server stores metadata for each of the files. A plurality of lock servers, having one or more locks for each file and associated file system metadata operates as a single centralized distributed lock server to coordinate access to the files stored by the disk server layer. A plurality of asynchronous file servers, operating as a single distributed file server layer in response to a request for a file from a user program: (i) retrieves the requested file from the disk server layer and at least one lock, associated with the retrieved file, from the lock server, and (ii) retrieves metadata for at least one file that is related to the requested files, and at least one lock, for each file that has had its metadata retrieved.

A data management system or “DMS” provides data services to data sources associated with a set of application host servers. The data management system typically comprises one or more regions, with each region having one or more clusters. A given cluster has one or more nodes that share storage. When providing continuous data protection and data distribution, the DMS nodes create distributed object storage to provide the necessary real-time data management services. The objects created by the DMS nodes are so-called active objects. The distributed object store can be built above raw storage devices, a traditional file system, a special purpose file system, a clustered file system, a database, and so on. According to the present invention, the DMS active object store provides an indexing service to the active objects. In an illustrative embodiment, any object property that has a given attribute is indexed and, as a result, the attribute becomes searchable. The DMS provides hierarchical distributed indexing using index trees to facilitate searching in a highly efficient manner.

Load sharing clusters in which each node is responsible for one or more non-overlapping subset(s) of the cluster namespace and will process only those requests that access file or directory objects in the partitioned namespace that the node controls while redirecting requests designated for other nodes. A non-overlapping subset may be migrated from a source node to a destination node, for example, for load sharing or hotspot mitigation. Migration typically includes maintaining a file virtualization link from the destination node to the source node to permit forwarding of requests by the destination node to the source node during migration of metadata and then migration of data from the source node. After migration is complete, the file virtualization link is broken and the destination node services requests for the non-overlapping subset from the migrated metadata and data.

A system and method proxies data access commands across a cluster interconnect between storage appliances in a cluster. Each storage appliance activates two ports for data access, a local port for data access requests directed to clients of the storage appliance and a proxy port for data access requests directed to the partner storage appliance. Clients utilizing multi-pathing software may send data access requests to either the local port of the storage appliance or the proxy port of the storage appliance. The system and method improve high availability especially during a loss of connectivity due to non-storage appliance hardware failure.

The invention discloses a distributed hadoop cluster fault automatic diagnosis and restoration system which comprises a cluster file system monitoring module for collecting and obtaining cluster node information and a database file; a work and task monitoring module for collecting information of work and tasks; a physical node monitoring module for monitoring resource consumption information of each physical node; a data storage and analysis and processing module for storing monitoring data to a database, setting monitoring alarm rules and configuring alarm ID, level and reasons in advance; and an automatic restoration module for defining and configuring various common alarm faults in advance and making a preprocessing script for each alarm fault, matching the fault happened at present with the alarm faults defined and configured in advance when monitoring a fault, and calling the corresponding preprocessing script to finish automatic restoration of the fault. The method can diagnose and restore system fault automatically to allow maintenance to become easier, and performance data and node state to be clearer and more obvious.

A data management system or “DMS” provides data services to data sources associated with a set of application host servers. The data management system typically comprises one or more regions, with each region having one or more clusters. A given cluster has one or more nodes that share storage. When providing continuous data protection and data distribution, the DMS nodes create distributed object storage to provide the necessary real-time data management services. The objects created by the DMS nodes are so-called active objects. The distributed object store can be built above raw storage devices, a traditional file system, a special purpose file system, a clustered file system, a database, and so on. According to the present invention, the DMS active object store provides an indexing service to the active objects. In an illustrative embodiment, any object property that has a given attribute is indexed and, as a result, the attribute becomes searchable. The DMS provides hierarchical distributed indexing using index trees to facilitate searching in a highly efficient manner.

A method effectively preventing a requesting node from unfencing and mounting a file system subsequent to a failure in a cluster file system having a plurality of active nodes. The method comprising first upgrading one active node in the cluster to function as a cluster manager node. The cluster manager is in communication with all nodes. The cluster manager is assigned manager responsibilities, in part, comprising first receiving an active status request from the node requesting to mount a file system. The cluster manager first queries the quorum nodes to determine whether each node considers the cluster manager to still have cluster management responsibilities for the file system. If a majority of quorum nodes consider the cluster manager to still have cluster management responsibilities for the file system then the cluster manager responds to the requesting node's active status request. Thereafter, the requesting node proceeds with mounting the file system.

A method may involve: a node included in a cluster looking up a file in a metadata name cache; in response to a miss for the file in the metadata name cache, the node creating a metadata name cache entry for the file in the metadata name cache, where the metadata name cache entry includes a copy of a cookie associated with metadata allocated to the file; in response to a lock on the metadata changing state for the node, the node modifying the cookie associated with the metadata, where the lock changes state for the node in response to a request to a global lock manager for the lock on the metadata; and comparing the copy of the cookie stored in the metadata name cache entry to a current value of the cookie associated with the metadata to determine if the metadata name cache entry is valid.

A cluster of computer system nodes share direct read / write access to storage devices via a storage area network using a cluster filesystem and operating system implementing DMAPI. Threads executing on a metadata client know when a DMAPI event is required, and generate the DMAPI event on their own initiative when necessary. A metadata server maintains DMAPI queues. If the metadata server relocates to another host, the DMAPI events in the DMAPI queues are moved transparently to users.

Systems and methods for transactional processing within a clustered file system wherein user defined transactions operate on data segments of the file system data. The users are provided within an interface for using a transactional mechanism, namely services for opening, writing and rolling-back transactions. A distributed shared memory technology is utilized to facilitate efficient and coherent cache management within the clustered file system based on the granularity of data segments (rather than files).

The invention provides a cluster file system, which is characterized by comprising a directory server, a layout server cluster and a memory device array, wherein the directory server is used for storing directory metadata and providing a directory service; the layout server cluster is used for storing layout metadata and providing a layout service; and the memory device array is used for storing file content data. The invention also provides a corresponding file service method. The file service method comprises a file accessing step, namely, a client first accesses the directory server to acquire the directory information of a file, then accesses the layout sever to acquire the distribution information of the file content data on physical memory devices, and finally directly accesses the physical memory devices to acquire the data information. Compared with the prior art, the invention can further improve the expansion capability of the file cluster system, and is highly compatible and short in accessing delay.

A system and method for performing a distributed consistency check of a clustered file system. File system functions for loading an inode and / or buffer tree are modified so that in response to either of these functions being invoked, a consistency check is performed. The consistency check verifies both local consistency on a node of the clustered file and a distributed check across the nodes of the storage system

A cluster of computer system nodes share direct read / write access to storage devices via a storage area network using a cluster filesystem. Access to files on the storage devices is arbitrated by the cluster filesystem using tokens. Upon detection of a change in location of the metadata server, client nodes waiting for a token are interrupted to check on the status of at least one of data and node availability.

The invention provides a method and system concurrently accessing a large amount of small documents in cluster storage. The method comprises the following steps: the written small documents are buffered, a plurality of buffered small documents are merged into one temporary document, the metadata and the data object of the temporary document are stored into the backend storage of a metadata server node and a data server node. The method can effectively improve the response time and speed of the cluster document system service and the whole data read and write time and transaction capacity per unit time of the data.

A method for avoiding directory conflicts across a file system having multiple nodes with one node representing a token manager. For each directory, at least one metanode responsible for directory updates. The method involving obtaining a name-based lock on a filename. Querying the token manager for a lock token. If a new token is obtained, sending a Remote Procedure Call (RPC) to the metanode to obtain matching directory entries, and caching the result of the RPC so that subsequent operations on that filename can be satisfied without additional messages sent to the metanode. Updating a cached directory block in memory by creating a new directory entry in the block or deleting an existing directory entry from the block. Information regarding the updating is sent to the metanode on one of: periodic schedule, a subsequent synchronization event, or when the name-based lock token is revoked.

A client computer system of a cluster may send a request to create a file in a cluster file system. A server may create a file in response to the information and allocate space in a storage to the file. If a request to write to the file is received within a predetermined amount of time, the write may complete without requiring that additional operations be performed to allocate space to the file. If a write to the file is not received within the predetermined amount of time, the space allocated to the file when it was created may be de-allocated. The file system may additionally or alternatively perform a method for opening a file while delaying an associated truncation of space allocated to the file. If a request to write to the file is received within a predetermined amount of time, the write may be performed in the space already allocated to the file.

A cluster of computer system nodes share direct read / write access to storage devices via a storage area network using a cluster filesystem. Version information about subsystems is acquired by a leader node when forming a cluster membership and distributed to all nodes in the cluster to enable proper messaging during operation.

The invention discloses a method for managing metadata for a cluster unit file system, wherein the file system is divided into a plurality of metadata subareas, and each metadata subarea is divided into a plurality of metadata blocks for maintaining the unit information of the metadata. The method comprises the following steps: the client analyzes the generated metadata requirement to acquire the metadata subarea which the metadata belongs to, the request is sent to a metadata server corresponding to the metadata subarea; the metadata server determines the type of the request according to the type value in the received metadata request; according to the type of the request of the metadata server, the metadata server inquires the data structure where the metadata is stored and increases, positions and modifies the metadata. The method adopts the mode of staying the metadata in the memory to improve the processing capability of the metadata, organizes and manages the metadata according to the characteristics of memory access, effectively utilizes the memory space, and improves the processing performance of the metadata.

A method of providing shared objects and node-specific objects in a cluster file system is described and provided. The method includes providing to a particular shared object an attribute, whereas the particular shared object can be a container-type shared object. The attribute indicates any object created in the particular shared object (including its sub-divisions or sub-containers) from this point in time will be designated as node-specific. Furthermore, when a node causes a file system operation that is node-specific to be performed by accessing the particular shared object, the file system operation is performed in an alternate directory corresponding to the node. The alternate directory supports a node-specific object. The shared objects and the node-specific objects can be directories. Additionally, the shared objects and the node-specific objects can be files.

A method for file lock recovery in a distributed computer system. The method includes executing a distributed computer system having a plurality of nodes comprising a cluster, and initiating a network file system server fail over from one node of the cluster to an adoptive node of the cluster. File lock services are then stopped at the adoptive node. File lock services are subsequently restarted at the adoptive node, wherein the restart causes the adoptive node to commence a grace period for other network file system clients to connect to the adoptive node and reclaim file locks. After restarting file lock services, a cluster file system is updated on the adoptive node with process identifiers, and file lock services are resumed at the adoptive node. The cluster file system can be simultaneously exposing the file lock services to other clients as well, like CIFS, etc.

The invention provides a method for ensuring the operation consistence of distributed metadata in a cluster file system in combination with a local log. The method comprises the following steps of: making a coordinator and a participant record a distributed metadata sub-operation update generated by using distributed metadata sub-operation in the local log as a local transaction, wherein the record of the distributed metadata sub-operation update includes a distributed transaction and / or a distributed metadata update; after the local transaction is submitted, writing the distributed transaction into a distributed log, and writing the distributed metadata update into a metadata disk; and before starting the distributed metadata sub-operation, forcibly submitting a local log transaction. According to the method, the local log is multiplexed effectively, and the implementation complexity of the distributed log is lowered; and moreover, cascading aborts can be avoided.

A cluster file system is disclosed. A plurality of disk servers, operating as a single distributed disk server layer, are connected to the one or more physical disks. Each disk server stores metadata for each of the files. A plurality of lock servers, having one or more locks for each file and associated file system metadata operates as a single centralized distributed lock server to coordinate access to the files stored by the disk server layer. A plurality of asynchronous file servers, operating as a single distributed file server layer in response to a request for a file from a user program: (i) retrieves the requested file from the disk server layer and at least one lock, associated with the retrieved file, from the lock server, and (ii) retrieves metadata for at least one file that is related to the requested files, and at least one lock, for each file that has had its metadata retrieved.

Embodiments of the invention relate to coordinated access to a clustered file system's shared storage subsystem using a shared-lock architecture. A particular file server is configured to coordinate the file system's file access layout particular server, in response to a client's file access layout request for byte-ranges of a file stored in the storage subsystem being received by a file server in cluster and sent to said particular server, generates a file access layout of a set of file servers for a parallel file access protocol to use and byte ranges of the file that particular individual file servers to use, to service an I / O request for the file. The file server (that received the request), sends the generated file access layout to said client, and requests coordinated access to said file from the particular server in response to receiving an I / O request to byte-ranges of said file.

A system and method proxies data access commands across a cluster interconnect between storage appliances in a cluster. Each storage appliance activates two ports for data access, a local port for data access requests directed to clients of the storage appliance and a proxy port for data access requests directed to the partner storage appliance. Clients utilizing multi-pathing software may send data access requests to either the local port of the storage appliance or the proxy port of the storage appliance.

Systems and methods for providing more efficient handling of I / O requests for clustered file system data subject to data migration or the like. For instance, exemplary systems can more quickly determine if certain files on primary storage represent actual file data or stub data for recalling file data from secondary storage. Certain embodiments utilize a driver cache on each cluster node to maintain a record of recently accessed files that represent regular files (as opposed to stubs). A dual-locking process, using both strict locking and relaxed locking, maintains consistency between driver caches on different nodes and the data of the underlying clustered file system, while providing improved access to the data by the different nodes. Moreover, a signaling process can be used, such as with zero-length files, for alerting drivers on different nodes that data migration is to be performed and / or that the driver caches should be flushed.