1926 results about "Distributed memory systems" patented technology

Data is secured in a computer network to transparently establish and manage a separation of user-based communities of interest based upon crypto-graphically separated, need to know, security levels. Data from a source document, data object or data stream is filtered to form subsets of extracted data and remainder data based upon security levels for the communities. Extracts are stored in assigned memories. Full or partial plaintext reconstruction is permitted only in the presence of assigned security clearance for the community of the inquiring party. Encryption, corresponding to security levels, establishes separation of secured data. The information processing system uses a data filter to extract security sensitive words, data objects, etc., a distributed storage system and a compiler is used to reconstruct plaintext based on security clearance. Multiple level encryption in one document is also available.

A distributed, web-services based storage system. A system may include a web services interface configured to receive, according to a web services protocol, a given client request for access to a given data object, the request including a key value corresponding to the object. The system may also include storage nodes configured to store replicas of the objects, where each replica is accessible via a respective unique locator value, and a keymap instance configured to store a respective keymap entry for each object. For the given object, the respective keymap entry includes the key value and each locator value corresponding to replicas of the object. A coordinator may receive the given client request from the web services interface, responsively access the keymap instance to identify locator values corresponding to the key value and, for a particular locator value, retrieve a corresponding replica from a corresponding storage node.

A system and method for distributed query processing may compile and optimize query plans for incoming query requests independent of hardware configurations and / or physical locations of data partitions in a distributed storage system (e.g., a data grid). The query plan may be divided into segments, and each segment may be instantiated on a remote query processing node of the distributed system by a query coordinator node according to metadata accessed at runtime by remote sub-query operators in the query plan. The metadata may include an indication of the physical locations of data partitions in the system and may be stored on one or more of the query processing nodes. The remote query processing nodes may execute the query plan segments and return results to the requestor. Cached query plans may be re-executed without recompilation, according to current metadata, even in the event of a node failure or data partition move.

A keymap service architecture for a distributed storage system. A system may implement storage nodes configured to store replicas of data objects, where each of the replicas is accessible via a respective locator value, and keymap instances each configured to store keymap entries corresponding respectively to the data objects. A given keymap entry may indicate a mapping from a given key value corresponding to a given data object to each respective locator value of its replicas. Each of the keymap instances may store a replica of the given keymap entry and may index its respective stored keymap entries within a respective index data structure including hierarchically arranged index nodes corresponding to keymap entries. For a given keymap entry having a given corresponding index node, each tag value associated with each ancestor of the given corresponding index node may be a prefix of the given key value.

A data storage subsystem in a distributed storage system having a plurality of predictive failure analyzing data storage devices. The subsystem furthermore has a circuit that is responsive to a predicted failure indication by a data storage device in relation to predetermined rules stored in memory for deterministically initiating a preventive recovery either by a data recovery procedure in the data storage device or by a fault tolerance storage arrangement in the subsystem.

A distributed storage system for the long-term storage of data objects that is implemented utilizing one or more distinct storage sites that may be comprised of system controllers and object storage systems that act in concert to embody a single distributed storage system. A system may include a one or more types and / or instances of object storage systems in which replicas of objects are stored in at least two of the said object storage systems. A system may further include system controllers associated with logical and / or physical sites that coordinate object, user, device, and system management functionally.

Embodiments of the invention provide methods and apparatus for improving the performance of file transfer to a client from a distributed storage system which provides single name space to clients. In one embodiment, a system for providing access to files in a distributed storage system comprises a plurality of storage nodes and at least one computer device connected via a network. Each storage node is configured, upon receiving a file access request for a file from one of the at least one computer device as a receiver storage node, to determine whether or not to inform the computer device making the file access request to redirect the file access request to an owner storage node of the file according to a preset policy. The preset policy defines conditions for whether to redirect the file access request based on at least one of file type or file size of the file.

Approaches for a distributed storage system that comprises a plurality of nodes. Each node, of the plurality of nodes, executes one or more application processes which are capable of accessing persistent shared memory. The persistent shared memory is implemented by solid state devices physically maintained on each of the plurality of nodes. Each the one or more application processes, maintained on a particular node, of the plurality of nodes, communicates with a shared data fabric (SDF) to access the persistent shared memory. The persistent shared memory comprises a scoreboard implemented in shared DRAM memory that is mapped to a persistent storage. The scoreboard provides a crash tolerant mechanism for enabling application processes to communicate with the shared data fabric (SDF).

A distributed, web-services based storage system. A system may include a web services interface configured to receive, according to a web services protocol, a given client request for access to a given data object, the request including a key value corresponding to the object. The system may also include storage nodes configured to store replicas of the objects, where each replica is accessible via a respective unique locator value, and a keymap instance configured to store a respective keymap entry for each object. For the given object, the respective keymap entry includes the key value and each locator value corresponding to replicas of the object. A coordinator may receive the given client request from the web services interface, responsively access the keymap instance to identify locator values corresponding to the key value and, for a particular locator value, retrieve a corresponding replica from a corresponding storage node.

Methods and systems described herein implement an SLA-based dynamic provisioning of storage for virtualized applications or virtual machines (VMs) on shared storage. The shared storage can be located behind a storage area network (SAN) or on a virtual distributed storage system that aggregates storage across direct attached storage in the server or host, or behind the SAN or a WAN.

A control device for preparing redundant data, dividing the data into a plurality of volumes, and distributing and storing volumes in a plurality of storage units scattered through a network includes a route management unit and a storage set management unit. The route management unit computes an evaluation value indicating the preferability of a use target on each of the scattered storage unit based on the bandwidth, the communications cost, the physical distance between a node requesting a write and a storage unit. The storage set management unit selects a plurality of storage units as the optimum storage set from among the above mentioned scattered storage units based on the evaluation value.

Distributed storage system having content-based deduplication function and object storing method. The distributed storage system may include a plurality of data nodes and a server coupled with the plurality of data nodes. Each one of the plurality of data nodes may be configured to store at least one object. The server may be configured to perform a deduplication function based on a content-specific index of a target object and content-specific indexes of objects stored in the plurality of data nodes in response to an object storage request from a client, and configured to store the target object in one of the plurality of data nodes based on a result of the deduplication function performed by the server.

A system and method for distributing data over a plurality of remote storage nodes. Data are split into segments and each segment is encoded into a number of codeword chunks. None of the codeword chunks contains any of the segments. Each codeword chunk is packaged with at least one encoding parameter and identifier, and metadata are generated for at least one file and for related segments of the at least one file. The metadata contains information to reconstruct from the segments, and information for reconstructing from corresponding packages. Further, metadata are encoded into package(s), and correspond to a respective security level and a protection against storage node failure. A plurality of packages are assigned to remote storage nodes to optimize workload distribution. Each package is transmitted to at least one respective storage node as a function iteratively accessing and retrieving the packages of metadata and file data.

A distributed storage system that dispatches an input / output request is described. In an exemplary embodiment, a storage controller client receives the input / output request, wherein the distributed storage system includes the storage controller client, a plurality of storage controller servers, a plurality of virtual nodes distributed among a plurality of physical nodes, and each of the plurality of physical nodes is hosted on one of the plurality of storage controller servers. The storage controller client further computes a target virtual node for the input / output request, where the target virtual node is one of the plurality of virtual nodes. Using the computed target virtual node, the storage controller client determines a target physical node that corresponds to the target virtual node, where the target physical node is one of the plurality of physical nodes. In addition, the storage controller client dispatches the input / output request to a target storage controller that corresponds to the target physical node, wherein the target storage controller server is one of the plurality of storage controller servers. In addition, the virtual node includes a set of one or more mirrored copies across different fault domains for ensuring resiliency and high availability.

An embodiment of a method of operating a distributed storage system includes reading m data blocks from a distributed cache. The distributed cache comprises memory of a plurality of independent computing devices that include redundancy for the m data blocks. The m data blocks and p parity blocks are stored across m plus p independent computing devices. Each of the m plus p independent computing devices stores a single block selected from the m data blocks and the p parity blocks.

Provided are a distributed data storage method, device and system. The method comprises: splitting a data file, generating K data slices, splitting each data slice of the K data slices, generating M data blocks for each data slice, performing check encoding on the M data blocks using a redundancy algorithm, and generating N check blocks (110); determining a first physical storage node corresponding to a block of the M data blocks and the N check blocks using a random algorithm, and determining at least M+1 different physical storage nodes according to a first ranking manner conforming to a rule, based on the first determined physical storage node (120); and respectively storing at least M+1 blocks of the M data blocks and the N check blocks on the at least M+1 different storage nodes, with K, M and N being integers (130). The method, device and system can improve the reliability of distributed storage systems.

A systematic distributed storage system (DSS) comprising: a plurality of storage nodes, wherein each storage node configures to store a plurality of sub blocks of a data file and a plurality of coded blocks, a set of repair pairs for each of the storage nodes, wherein the system is configured to use the respective repair pair of storage nodes to repair a lost or damaged sub block or coded block on a given storage node. Also a distributed storage system DSS comprising h non-empty nodes, and data stored non homogenously across the non-empty nodes according to the storing codes (n,k). Further a method for determining linear erasure codes with local repairability comprising: selecting two or more coding parameters including r and δ; determining if an optimal [n, k, d] code having all-symbol (r, δ)-locality (“(r, δ)a”) exists for the selected r, δ; and if the optimal (r, δ)a code exists performing a local repairable code using the optimal (r, δ)a code.

A system and method for distributing and accessing files in a distributed storage system uses an ordered list of the storage nodes in the system to determine the storage node on which a file is stored. The distributed storage system includes a cluster of storage nodes and may also include one or more client nodes that participate in the system as storage resources. Each node (client and storage) stores an ordered list of the storage nodes in the system, allowing any of the nodes to access the file. The list is updated whenever a new storage node is added to the system, an existing storage node is removed from the system, or a new storage node is swapped with an existing storage node. Each one of the nodes may independently compute a new mapping of files to the storage nodes when the ordered list is changed.

A method includes dividing a dataset into partitions by hashing a specified key, selecting a set of distributed file system nodes as a primary node group for storage of the partitions, and causing a primary copy of the partitions to be stored on the primary node group by a distributed storage system file server such that the location of each partition is known by hashing of the specified key.

An in-memory storage manager represents XML-compliant documents as a collection of objects in memory. The collection of objects allows the storage manager to manipulate the document, or parts of the document with a consistent interface and to provide for features that are not available in conventional XML documents, such as element attributes with types other than text and documents that contain binary rather than text information. In addition, in the storage manager, the XML-compliant document is associated with a schema document which defines the arrangement of the document elements and attributes. The schema data associated with a document can contain a mapping between document elements and program code to be associated with each element. The storage manager further has methods for retrieving the code from the element tag. The retrieved code can then be invoked using attributes and content from the associated element and the element then acts like a conventional object. Further, the storage manager allows real-time access by separate process operating in different contexts. The objects that are used to represent the document are constructed from common code found locally in each process. In addition, the data in the objects is also stored in memory local to each process. The local memories are synchronized by means of a distributed memory system that continually equates the data copies of the same element in different processes. Client-specified collections are managed by a separate collection manager. The collection manager maintains a data structure called a “waffle” that represents the XML data structures in tabular form. A record set engine that is driven by user commands propagates a set of updates for a collection to the collection manager. Based on those updates, the collection manager updates index structures and may notify waffle users via the notification system.

The invention relates to a method, apparatus and system for data reconstruction in a distributed storage system. A main storage node in the distributed storage system performs EC coding on the data to be written to generate an EC stripe; each EC block on the EC stripe is stored on each storage node; when part of the storage nodes cause failure of writing of EC blocks because of faults, the main storage node stores the EC blocks distributed to the storage nodes which fails to write in the local, and generates metadata information required by data reconstruction; and when the faults of the storage nodes recover, the main storage node retransmit the stored EC blocks distributed to the storage nodes and the metadata information corresponding to the EC blocks to the storage nodes so as to enable the storage nodes after recovering the faults to complete data reconstruction. The scheme for data reconstruction in the distributed storage system has the advantages that when part of the storage nodes malfunction, EC reversal coding is not required to recover the data on the fault nodes, but the buffer of main storage node is distributed to the EC blocks with fault nodes; after the fault nodes recover, the buffered EC blocks are retransmitted to the fault nodes to perform data reconstruction, thus avoiding computing resource consumption because of performing EC reversal coding when the storage nodes recover fault to perform data reconstruction, and avoiding consumption of network resource because large amount of data is transmitted during the EC reversal coding process at the same time.

A distributed storage system including support for distinct storage classes. A system may implement a web services interface, a number of storage nodes and a coordinator. The web services interface may receive, according to a web services protocol, client requests to store data objects. A request for a given object may include an indication of a particular storage class to be associated with the given object, and where each storage class respectively specifies one or more service level agreement (SLA) characteristics. The storage nodes may store replicas of the data objects. The coordinator may receive the client requests to store the data objects from the web services interface, where in response to the given client request, the coordinator may store one or more replicas of the given data object among the storage nodes according to the SLA characteristics specified by the particular storage class.

A server computer at a first storage sub-system of a distributed storage system receives from a client a first client request for an object. If the object is not present in the first storage sub-system, the server computer identifies a second storage sub-system of the distributed storage system as having a replica of the requested object, the requested object including content and metadata. The server computer submits an object replication request for the requested object to the second storage sub-system and independently receives the content and metadata of the requested object from the second storage sub-system. The server computer generates a new replica of the object at the first storage sub-system using the received metadata and content and returns the metadata of the new replica of the object to the client.

The present invention relates to an apparatus and method for eliminating duplication of a file in a distributed storage system. The apparatus and method for eliminating duplication of a file in a distributed storage system according to the present invention calculates a hash value of each chunk for an active file; calculates a secondary hash value by adding the hash values calculated for respective chunks; examines duplication of the file using the hash value of each chunk and the secondary hash value; and eliminates a duplicated file depending on a result of the examination.

The invention discloses a data storage method, data storage equipment and a distributed storage system. The method, the equipment and the system belong to the technical field of computers. The data storage method comprises the steps that access nodes of the distributed storage system cut a data block which is written in by a user into a plurality of data fragments with preset sizes; a plurality of check fragments corresponding to the data fragments which are obtained by cutting are calculated through a redundancy check algorithm; the data fragments and the check fragments are stored into the data nodes of the distributed storage system, and each data fragment and each check fragment only store one copy. According to the data storage method, the data storage equipment and the distributed storage system, the proportion of redundant data can be reduced on the premise of ensuring data reliability, so as to save the storage space.

A method for providing access to data items from a distributed storage system is provided. Each data item is replicated across a plurality of storage nodes. Data items are read from the distributed storage system by selecting between a first reading mode, comprising attempting to read the data item from a set of the storage nodes to check for data item consistency across at least a quorum of the set of nodes, and a second reading mode, comprising reading the data item from at least one of the storage nodes. The reading mode is selected according to at least one detected characteristic of system status of the distributed storage system. The second reading mode is selected when the detected characteristic indicates a higher likelihood of data item consistency, and the first reading mode is selected when the detected characteristic indicates a lower likelihood of data item consistency.

A client device or other processing device separates a file into blocks and distributes the blocks across multiple servers for storage. In one aspect, subsets of the blocks are allocated to respective primary servers, a code of a first type is applied to the subsets of the blocks to expand the subsets by generating one or more additional blocks for each subset, and the expanded subsets of the blocks are stored on the respective primary servers. A code of a second type is applied to groups of blocks each including one block from each of the expanded subsets to expand the groups by generating one or more additional blocks for each group, and the one or more additional blocks for each expanded group are stored on respective secondary servers. The first and second codes are advantageously configured to provide security against an adversary that is able to corrupt all of the servers over multiple periods of time but fewer than all of the servers within any particular one of the periods of time.

The invention discloses a method for data fragmentation of a distributed storage system. The method comprises the steps that a physical node state is monitored through a cluster controller and the correctness of data transmission is guaranteed; physical spaces of storage nodes are managed in a unified mode through the consistent Hash algorithm, so that a logic storage space is formed; the reliability of the system is guaranteed in multi-transcript fragment mode. According to the method, the storage capacity is lager and flexible and the storage capacity virtualized by a host computer depends on resources of a native machine. Due to the limitation on hardware design of the host computer, the storage capacity cannot be infinitely increased; the storage capacity cannot be limited to the storage capacity of the host computer after the distributed system is adopted, storage resources can be increased or reduced at will according to the requirement of a client, and the storage capacity and the flexibility of the system are improved to the greatest extent.