1155 results about "Distributed File System" patented technology

The disclosed embodiments disclose techniques for executing a cloud command for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a cloud controller presents a distributed-filesystem-specific capability to a client system as a file in the distributed filesystem (e.g., using a file abstraction). Upon receiving a request from the client system to access and/or operate upon this file, the client controller executes an associated cloud command. More specifically, the cloud controller initiates a specially-defined operation that accesses additional functionality for the distributed filesystem that exceeds the scope of individual reads and writes to a typical data file.

A NAS (Network Attaches Storage) switch authenticates a client on multiple file servers for proxy services. The NAS switch enables proxy services by successively authenticating the client on the file servers using referrals. The NAS switch further comprises a connection manager to establish connections to the client and the file servers, a referral manager to redirect the client for successive authentications, and a transaction manager to perform data transfers with the file servers on behalf of the client. The system components support DFS (Distributed File System), and communicate using a protocol dialect that supports referral mechanisms such as NFSv4 (Network File Server version 4) or CIFS (Common Internet File System). The transaction manager also performs a protocol dialect translation service when the connection manager negotiates one protocol dialect with the client, and a different protocol dialect with the file server.

A map-reduce compatible distributed file system that consists of successive component layers that each provide the basis on which the next layer is built provides transactional read-write-update semantics with file chunk replication and huge file-create rates. A primitive storage layer (storage pools) knits together raw block stores and provides a storage mechanism for containers and transaction logs. Storage pools are manipulated by individual file servers. Containers provide the fundamental basis for data replication, relocation, and transactional updates. A container location database allows containers to be found among all file servers, as well as defining precedence among replicas of containers to organize transactional updates of container contents. Volumes facilitate control of data placement, creation of snapshots and mirrors, and retention of a variety of control and policy information. Key-value stores relate keys to data for such purposes as directories, container location maps, and offset maps in compressed files.

In one embodiment, a user or client device is connected to a distributed file system comprised of one or more physical nodes. The data on each of the physical nodes store metadata about files and directories within the file system. Some of the embodiments permit a user to take a snapshot of data stored on the file system. The snapshot may include a single file, a single directory, a plurality of files within a directory, a plurality of directories, a path on the file system that includes nested files and subdirectories, or more than one path on the file system that each includes nested files and directories.

A computer implemented method for executing an ANSI SQL expression belonging to the SELECT-WHERE-equi-JOIN class on data residing in a distributed file system, said method comprising the steps of entering the ANSI SQL expression into a user interface; converting the ANSI SQL expression into a map-reduce program; running the map-reduce program on the distributed file system; storing the result set of the program in the distributed file system; and presenting the result set through a user interface.

The embodiment of the disclosure relates to block chain-based distributed storage, and discloses a block chain-based distributed storage method. The method includes: obtaining a file which needs to be stored, and generating a hash value and an index of the hash value for the file. The method also includes: recording the hash value in a block chain network, and storing the file in a distributed network, wherein the file is located on the basis of the hash value in the distributed network. Therefore, according to the embodiment of the disclosure, tampering proofing and the distributed storage of the file can be realized through combining a block chain and a distributed file system, thus the problem of the limited capacity of block chain nodes is solved, and the authenticity and the availability of the stored file are effectively guaranteed.

A distributed cache data system includes a cache adapter configured to reserve a designated portion of memory in at least one node in a networked cluster of machines as a contiguous space. The designated portion of memory forms a cache and the designated portion of memory includes data cells configured to store data. The cache adapter is configured to interface with the data and a distributed file system and the cache adapter is further configured to provide an interface for external clients to access the data. The cache adapter is configured to communicate to clients via a web server, and the cache adapter is further configured to direct data access requests to appropriate the data. A related process of distributing cache data is disclosed as well.

A logical file system is described that distributes copies of files across various different physical storage resources yet provides a consistent view to the user of his or her data, regardless of which machine the user is accessing the files from, and even when the user's computer is offline. The distributed file system uses smart data redundancy to enable a virtually infinite amount of storage as long as additional storage resources are made available to the distributed file system. The result is a reliable storage system that does not necessarily tie the user's data to the user's particular computer. Instead, the user's data is associated with the user—for life—or for however long the user would like the data to be maintained, regardless of whether the user's computer or data storage components are replaced or destroyed.

The invention relates to the technical field of Internet information, in particular to a cloud storage system and an implementation method thereof. The cloud storage system comprises more than one metadata server sets and a plurality of data block storage node sets, wherein metadata server sets manage the whole distributed file system, respond an access of a client to a file, receive a writing request of a writing client and distribute the writing request to a storage block or receive a request of a reading client for reading a file and return to a position of a block corresponding to the file; and the reading client reads file blocks from corresponding blocks and combines the file blocks. In a system, the file is divided into one or more blocks; the blocks are scatteredly stored in the data block storage node sets; and storage nodes create, delete and copy the file blocks under the command of metadata servers. In the cloud storage system and the implementation method, the safety and the transmission efficiency are improved due to block redundancy distribution and block balance distribution, and the congestion probability is reduced. The cloud storage system and the implementation method are widely applied to cloud storage systems of enterprises.

In accordance with an embodiment of the present invention, a security module may be configured to provide an owner the capability to differentiate between users. In particular, the security module may be configured to generate an asymmetric read/write key pair for respectively decrypting/encrypting data for storage on a disk. The owner of the file may distribute the read key of the asymmetric key pair to a group of users that the owner has assigned read-permission for the encrypted data, i.e., a group that has read-only access. Moreover, the owner of the file may distribute the write key of the asymmetric pair to another group of users that the owner has assigned write-permission for the encrypted data, i.e., users in the write-permission group may modify the data. Alternatively, the security module may be configured to throw away the write key and not allow further re-use of the key. The security modules may also be configured to encrypt the read key for with a further key for additional protection while stored. The security module may be also configured to generate a first set of read/write key pairs for fragments of a file. Each file fragment is encrypted with a different write key from the set of read/write key pairs. The respective read keys may then be encrypted with a second long-lived key pair chosen by the owner of the file. The security module may then configured to store the encrypted file fragment and the associated encrypted read key in a storage area of a shared computer system accessible to the users of the shared computer system. The security module may also be configured to provide distribution of the required keys-either the read/write keys for direct use, or the long-lived keys for indirect use-either by means of the data owners themselves, or through use of a key distribution center.

In one embodiment, a user or client device communicates with a distributed file system comprised of one or more physical nodes. The data on each of the physical nodes store metadata about files and directories within the file system. Some of the embodiments permit a user to take a snapshot of data stored on the file system. The snapshot may include a single file, a single directory, a plurality of files within a directory, a plurality of directories, a path on the file system that includes nested files and subdirectories, or more than one path on the file system that each includes nested files and directories. In some embodiments, systems and methods intelligently choose whether to use copy-on-write or point-in-time copy when saving data in a snapshot version of a file whose current version is being overwritten. In some embodiments, systems and methods allow snapshot users to return from a snapshot directory to the immediate parent directory from which the user entered into the snapshot.

A method of managing segments in a distributed-file system implemented by a plurality of file servers includes determining a segment of the distributed-file system controlled by a first file server for which control is to be migrated, selecting a second file server, that is different from the first file server, to take control of the segment, and moving control of the segment from the first file server to the second file server.

The consistency callback mechanisms employed by local file systems such as NTFS and distributed file systems such as DDS, NFS and CIFS are extended to provide a shared memory foundation for efficiently broadcasting real-time high definition video from a source object to large numbers of viewers via the Internet. Distributed applications such as video viewing client applications establish connections to a common distributed file system object, and then each application registers with the underlying distributed file system to receive notifications whenever the video source modifies the source object. The data required to update images maintained by viewing clients is included in notification messages. The distributed file system employs a network of proxy cache nodes. Proxy cache nodes receive notification messages (complete with image update data) and update their cached images of the source object and then retransmit the notification messages towards the viewing clients using IP multicast techniques. In this manner, the distributed file system's consistency mechanism efficiently employs network resources to enable the real-time distribution of video content streams.

A distributed file system is disclosed. A plurality of disk servers, operating as a single distributed disk server layer, are connected to the one or more physical disks. Each disk server stores meta-data for each of the files. A plurality of lock servers, having one or more locks for each file operates as a single distributed lock server to coordinate access to the files stored by the disk server layer. A plurality of asynchronous file servers, operating as a single distributed file server layer in response to a request for a file from a user program: (i) retrieves the requested file from the disk server layer and at least one lock, associated with the retrieved file, from the lock server, and (ii) retrieves meta-data for at least one file that is related to the requested files, and at least one lock, for each file that has had its meta-data retrieved.

A storage caching method and system manages shared access to real time data files while maintaining data file coherency and consistency in a computer network including a plurality of remote computer workstations and at least one file server. The storage caching system is implemented by storage caches, which are associated with workstations, and a cache server, which is associated with a file server, where the storage caches and the cache server interface with a distributed file system to provide shared access to real time data files by remote workstations.

A virtual distributed file system for executing applications from a remote location comprises an application server for remotely storing an original copy of the applications, and a client computing device which contains an operating system for executing the applications, a communications interface configured for communicating with the application server, a virtual file system mapped to a remote file system located on the application server, and a ring buffer cache for storing the applications on the computing device. If the modules are not found locally, the modules are transferred from the application server. The computing device is from a group consisting of a thin client, personal computer, laptop or PDA. The applications are segmented into compressed modules when transferred. The ring buffer is pre-seeded with compressed modules. The computing device is usable online or offline a network. A movable profile is accessible by a user from a plurality of computing devices.

An architecture and implementation for losslessly restarting subsystems in a distributed file system is described. By partitioning functionality and logging appropriately across the kernel and user-level boundaries on a client, the user-level subsystem may be made losslessly restartable. A particular use of transactions achieves efficiency while retaining simplicity. Practical mechanisms for supporting state-based recovery in replicated state machines and like replica are described. In particular, a map assisted state transfer may include receiving one or more state updates, marshaling one or more active data-structures into a marshaled shadow, applying the received state updates to the marshaled shadow and re-instantiating the active data-structures by unmarshaling the marshaled shadow. While active data-structures may include invariance relationships, the marshaled shadow may be structured to support independence from invariance relationships between the shadows of the active data-structures, as well as efficient incremental state update application and unmarshaling to re-instantiate the active data-structures.

The invention discloses a distributed file system metadata management method facing to high-performance calculation. The method comprises the following steps of: 1) establishing a catalogue metadata server cluster, a file metadata server cluster and an object storage server cluster; 2) establishing a global counting-based bloom filter in the catalogue metadata server cluster; 3) when the operation request of a client side arrives, skipping to execute step 4) or 5); 4) enabling the catalogue metadata server cluster to respond to the catalogue operation request of the client side to manage the catalogue metadata; and 5) enabling the file metadata server cluster to respond to the file operation request of the client side to manage the file metadata data. According to the distributed file system metadata management method disclosed by the invention, the metadata transferring problem brought by catalogue renaming can be effectively solved, and the distributed file system metadata management method has the advantages of high storage performance, small maintenance expenditure, high load, no bottleneck, good expansibility and balanced load.

The invention relates to a distributed file system and method for distributing files between computing devices (10-22) of at least one group of computing devices (10-22) in which files are distributed automatically upon the networking of two or more computing devices (10-22). In the preferred embodiment, the user is not required to invoke any file distribution application as this is effected automatically. Advantageously, all files in a shared file directory are distributed to all computers (10-22) in the group and stored in a usable format such that every computer (10-22) is able to read and write to a shared file even when not networked to other computers (10-22) in the group. File distribution is preferably carried out in a non-specific manner and in a one-for-all manner. The system is very suitable for networks with low connectivity patterns between the computers.

The disclosed embodiments provide a system that archives data for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers cache and ensure data consistency for the stored data. During operation, a cloud controller determines that a cloud file in a previously stored data snapshot is no longer being actively referenced in the distributed filesystem. The cloud controller transfers this cloud file from the (first) cloud storage system to an archival cloud storage system, thereby reducing storage costs while preserving the data in the cloud file in case it is ever needed again.

Disclosed are novel methods and apparatus for persistent queuing in distributed file systems. In an embodiment, an apparatus is disclosed. The apparatus includes a distributed file system including a plurality of remote systems. The plurality of remote systems includes a sender site and a receiver site. The apparatus further includes a local queue accessible by the sender site; a remote queue accessible by the receiver site; a next attempt time indicator; and an attempt counter. The next attempt time indicator may specify a next time to install a transferred file on the receiver site. The attempt counter indicates how many attempts have been made to install the transferred file on the receiver site.

The disclosed embodiments provide a system that distributes data for a distributed filesystem across multiple cloud storage systems. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers cache and ensure data consistency for the stored data. Whenever each cloud controller receives new data from a client, it outputs an incremental metadata snapshot for the new data that is propagated to the other cloud controllers and an incremental data snapshot containing the new data that is sent to a cloud storage system. During operation, a backup cloud controller associated with the distributed filesystem is also configured to receive each (incremental) metadata snapshot, such that, upon determining the failure of a cloud controller, the backup cloud controller can immediately begin receiving data requests from clients associated with the failed cloud controller.

A hierarchical distributed search mechanism is integrated into a distributed file system. Traditional file system APIs (create, open, close, read, write, link, rename, delete, . . . ) and the over-the-wire protocols employed to project these APIs into remote client sites (CIFS, NFS, DDS, Appletalk) are extended to enable the dynamic creation of temporary directories containing links to objects identified by search engines (executing at sites “close” to “their” data) as meeting the search criteria specified by the first parameter of a search function call. The search function, derived from the standard file system API function create, is added to the file system API.

Techniques for cloning, writing to, and reading from file system metadata. Cloning involves identifying a first set of pointers included in a first root node in a file system metadata tree structure that stores file system metadata in leaf nodes of the tree structure, creating a first copy of the first root node that includes the first set of pointers, creating a second copy of the first root node that includes the first set of pointers, associating the first copy with a first view, and associating the second copy with a second view. Reading generally involves traversing the tree structure towards a target leaf node that contains data to be read. Writing generally involves traversing the tree structure in the same manner, but also creating copies of any nodes to be modified if those nodes are deemed to have a different treelD than a particular root node.

Described are techniques for creating a snapshot of a distributed file system or a file thereof. The distributed file system includes metadata file systems storing metadata about files in the distributed file system and includes data file systems storing file content data. Quiescing is performed for requests affecting the distributed file system, or a file thereof, for which a snapshot is being created. A snapshot of metadata is obtained. For the distributed file system, this includes a snapshot of each metadata file system. For a file, this includes a snapshot of the file's metadata. A snapshot of file data is obtained. For distributed file systems, this includes a snapshot of each data file system. For a file, this includes a snapshot of the file's data. Unquiescing is performed for requests affecting the distributed file system, or the file thereof, upon successful completion of obtaining snapshots of metadata and file data.

The disclosed embodiments disclose techniques for synchronizing file updates between two cloud controllers of a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in the cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a cloud controller receives a request from a client to access a file in the distributed filesystem. The cloud controller sends a synchronization update request for the file to a second cloud controller and in response receives a synchronization update for the file from the second cloud controller.

A widelink directive provides an enhanced level of indirection with respect to a resource, such as a unit of storage, attached to a destination, such as a storage system. The widelink has a scope of indirection that is “wider” than a conventional symbolic link (“symlink”). The novel widelink directive is similar to the distributed file system (DFS) facility that allows DFS-enabled common internet file system clients to resolve uniform naming convention paths to locations that may or may not be on an original storage system, such as a multi-protocol filer, or original protocol server of the filer. By taking advantage of clients that support the DFS facility, the widelink directive is quite flexible in that it can be used to resolve symlinks that “leave” a share.

The invention relates to a method and a system for dynamically managing metadata of a distributed file system in order to solve the problems that a metadata management method cannot dynamically adjust the distribution of the metadata according to the load change, smoothly extend a metadata server, process the access to a hot content and well deal with the access to a single hot file. The method comprises that: the metadata server dynamically divides the distribution of the metadata on each metadata server according to the access condition of the metadata and manages the dynamically divided metadata according to the operation request of a client. The method and the system solve the problems of extensibility and load balance of the metadata server through dynamic division and the problem of consistency of the metadata through the management of the dynamically divided metadata.

A shared storage distributed file system is presented that provides applications with transparent access to a storage area network (SAN) attached storage device. This is accomplished by providing clients read access to the devices over the SAN and by requiring most write activity to be serialized through a network attached storage (NAS) server. Both the clients and the NAS server are connected to the SAN-attached device over the SAN. Direct read access to the SAN attached device is provided through a local file system on the client. Write access is provided through a remote file system on the client that utilizes the NAS server. A supplemental read path is provided through the NAS server for those circumstances where the local file system is unable to provide valid data reads.

Consistency is maintained by comparing modification times in the local and remote file systems. Since writes occur over the remote file systems, the consistency mechanism is capable of flushing data caches in the remote file system, and invalidating metadata and real-data caches in the local file system. It is possible to utilize unmodified local and remote file systems in the present invention, by layering over the local and remote file systems a new file system. This new file system need only be installed at each client, allowing the NAS server file systems to operate unmodified. Alternatively, the new file system can be combined with the local file system.

In one embodiment, a user or client device communicates with a distributed file system comprised of one or more physical nodes. The data on each of the physical nodes store metadata about files and directories within the file system. Some of the embodiments permit a user to take a snapshot of data stored on the file system. The snapshot may include a single file, a single directory, a plurality of files within a directory, a plurality of directories, a path on the file system that includes nested files and subdirectories, or more than one path on the file system that each includes nested files and directories. In some embodiments, systems and methods intelligently choose whether to use copy-on-write or point-in-time copy when saving data in a snapshot version of a file whose current version is being overwritten. In some embodiments, systems and methods allow snapshot users to return from a snapshot directory to the immediate parent directory from which the user entered into the snapshot.