70 results about "Distributed memory systems" patented technology

This application discloses a load balancing method and related equipment for distributed storage links, which can effectively improve the overall load utilization of a distributed storage system. The method includes: obtaining initial link grouping information corresponding to the distributed storage, wherein the distributed storage includes M storage links; determining T data acquisition time points corresponding to the current processing cycle; acquiring load data corresponding to each of the M storage links based on the T data acquisition time points; determining a first maximum load value corresponding to the M storage links based on the initial link grouping information and the load data; determining a second maximum load value corresponding to target link grouping information based on the load data and the initial link grouping information; and adjusting the M storage links according to the target link grouping information if the difference between the second maximum load value and the first maximum load value satisfies an adjustment condition.

This application provides a data scheduling method, a distributed storage system, a computing device, a storage medium, and a program product. The data scheduling method includes: responding to a data scheduling request, determining at least one piece of data to be scheduled corresponding to a target object; determining at least two media types corresponding to the target object from a plurality of media types; determining a target storage medium corresponding to each of the at least one piece of data to be scheduled and a target data node providing the target storage medium according to the at least two media types; and storing the at least one piece of data to be scheduled in the target storage medium of its corresponding target data node. The technical solution provided by this application can improve the service availability of a distributed storage system.

This application discloses a file read operation method, apparatus, and medium for a distributed storage system, relating to the field of communications. To improve the efficiency of read operations in a distributed storage system, this application involves receiving a read operation request; allocating memory space based on the read operation request; packaging the read operation request and the memory space address information into a read operation message; and sending the read operation message to an object storage device using remote direct memory access (RDA) communication, so that the object storage device writes the requested data into the memory space according to the read operation message. The client sends a read operation message containing control data for the read operation request and the memory space address information to the object storage device using RDA communication. The object writes the requested data into the specified memory space, achieving separation of control and data transfer, reducing the number of data copies between memory locations, and improving network transmission efficiency through the application of RDA technology.

An apparatus in an illustrative embodiment comprises at least one processing device configured to identify data units stored on storage nodes of a storage system, wherein different subsets of the data units comprise data sub-units stored utilizing different data striping schemes. The at least one processing device is also configured to maintain a data structure characterizing relationships between the storage nodes, the data structure comprising, for a given data unit, edge weights that are proportional to a given number of data sub-units that are part of the given data unit. The at least one processing device is further configured to receive an additional data unit to be stored, to determine, utilizing the data structure, a subset of the storage nodes on which to store data sub-units of the additional data unit, and to store the data sub-units of the additional data unit on the determined subset of the storage nodes.

A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more of a plurality of storage devices. The computing devices may push a point in time to a backend for a distributed storage system.

This invention provides a method, apparatus, and electronic device for batch generation of short drama works, belonging to the field of film and television work generation technology. The method includes: a master node acquiring the script of the short drama to be generated; decomposing the script into multiple atomic tasks and determining the dependencies between these tasks; based on the dependencies, distributing atomic tasks without dependencies to multiple slave nodes in a distributed generation system in parallel via a message queue; each slave node executing the received atomic tasks in parallel, writing the intermediate data generated as execution results into a distributed storage system; and the master node reading the execution results of each atomic task from the distributed storage system, merging them according to the dependencies and the time sequence of the script to generate the short drama video. This invention enables high-concurrency batch production of short dramas, significantly improving the efficiency and stability of short drama generation.

The present application relates to the technical field of computer storage service high availability, and discloses a data distribution method, device and equipment of a distributed storage system and a storage medium, the method comprising: configuring a plurality of object storage devices for a distributed storage system cluster, and calculating a total number of expected PGs of the cluster based on the expected number of PGs of each object storage device; calculating a number of non-repeating PGs that can be accommodated within a polynomial time based on the number of object storage devices; distributing the cluster PGs based on the size relationship between the number of non-repeating PGs that can be accommodated and the total number of expected PGs of the cluster and using mathematical graph theory, until all of the total number of expected PGs of the cluster are distributed. If two disks are damaged, at most only one PG cannot be written, and if three disks are damaged, at most only one PG cannot be read, thereby reducing the risk of data loss. If more disks are damaged, the amount of data that can be lost will also be greatly reduced, thereby reducing the range of PGs that can be affected when multiple data disks are damaged.

The application discloses a data reconstruction method and device, electronic equipment and storage medium, and relates to the technical field of data storage and recovery. In the case that a to-be-reconstructed disk exists in a distributed storage system, the redundancy of the to-be-reconstructed disk is determined based on state data of the to-be-reconstructed disk; a target redundancy level corresponding to the redundancy of the to-be-reconstructed disk is determined from a plurality of preset redundancy levels; different redundancy levels correspond to different erasure code types; the erasure code types corresponding to different redundancy levels represent different combinations of data recovery rates and storage resource occupation ratios; the erasure code type of the to-be-reconstructed disk is switched to a target erasure code type corresponding to the target redundancy level, and the target erasure code type is used to transmit a fault data block in the to-be-reconstructed disk to a specified disk for data reconstruction, thereby solving the problem that data recovery performance, resource consumption and network bandwidth resource occupation cannot be dynamically balanced in the data reconstruction process in the related art.

A data processing method applicable to database systems or distributed storage systems, particularly suitable for systems with high memory usage, such as in-memory databases, is disclosed. The method includes: during a snapshot period, writing at least one first object from memory to a snapshot file; wherein the snapshot file includes a first version number and first data content for each first object; writing a second version number and second data content of a second object corresponding to a write request within the snapshot period to a log file; wherein the second object is contained within at least one first object; and wherein the first version number of the first object in the snapshot file and the second version number of the second object in the log file are used to restore the data corresponding to the memory object. Because generating a snapshot does not require blocking write requests, database access latency is reduced. Furthermore, restoring data by comparing the current version number ensures data consistency after restoration.

This specification discloses a distributed data processing method, apparatus, storage medium, and electronic device. The method includes: acquiring streaming change events and node baseline dimension data for target business data nodes in an online database through a preset non-intrusive log data capture method; transmitting the streaming change events to a stream-batch integrated distributed storage system; using a real-time computing engine to perform dynamic time sharding processing on the streaming change events according to a preset time window strategy; calculating the multi-dimensional incremental aggregation value of each target business node in the current time window based on the baseline dimension data and superimposing it with the underlying baseline state snapshot to obtain a real-time aggregated state value; and updating the data of the target business nodes based on this.

The application discloses an omap operation performance test method and device and a medium, and relates to the field of distributed storage. Test parameters are initialized, the test parameters at least including a running time and a concurrency number; an operation queue is generated, the length of the operation queue being equal to the concurrency number; and an omap request is generated, the omap request including an omap write request or an omap read request, the number of the omap request being equal to the concurrency number, each omap request corresponding to a slot of the operation queue; the omap request is processed asynchronously through the operation queue, it is judged whether the processing process meets a preset condition, if yes, test statistical data is output. It can be seen that the above scheme can test the write operation and the read operation of the omap operation in the distributed storage system. Since the concurrency number is set and the asynchronous processing mode is adopted, the test pressure intensity can be controlled through multi-concurrency, so that the limit value of the omap operation performance is obtained, and the method is very reliable.

This application discloses a block storage task processing method, apparatus, device, and medium, relating to the field of distributed storage. It involves creating an initial block scenario for a storage cluster; obtaining the running status of each node in the storage cluster and information about the storage pool to which each node belongs; wherein each storage pool contains multiple nodes; selecting a corresponding master node in each storage pool; and when a task is received, the master node processes or assigns the task according to its type. Therefore, the above scheme achieves direct task processing by selecting a master node from the storage pool where each node in the storage cluster resides; and when a task is received, the master node processes or assigns the task according to its type. Simultaneously, task assignment avoids the problem of unhandled task interruptions due to long execution times or single-point failures, thus achieving efficient task processing in the distributed storage system and improving system reliability.

This invention discloses a distributed, block-based, multi-dimensional storage method and apparatus for seismic data volumes. The method includes: separating the header information from the seismic traces in the acquired seismic volume file and storing the header information in a preset storage space; determining multiple dimensions based on the keywords of the seismic traces to construct a multi-dimensional model; dividing the multi-dimensional model into multiple multi-dimensional data cubes and determining the name of each multi-dimensional data cube, as well as the extreme values ​​of operational values, the total number of location numbers, and the total number of multi-dimensional data cubes in each dimension; storing the multi-dimensional data cubes on a distributed storage system using a mapping algorithm; calculating the physical storage location of the target data based on the acquired target data information, and extracting the target data from the distributed storage system and / or the preset storage space. This method improves data storage flexibility and query efficiency by optimizing the seismic data storage structure and index calculation.

An apparatus in an illustrative embodiment includes at least one processing device comprising a processor coupled to a memory. The at least one processing device is configured to obtain, for a first storage system, information characterizing input-output (IO) processing loads of respective storage targets of a second storage system, and to configure a data service involving transfer of data from the first storage system to the second storage system based at least in part on the obtained information, wherein configuring the data service comprises selecting a particular subset of the storage targets of the second storage system for use in implementing the data service based at least in part on the obtained information. The first and second storage systems illustratively comprise respective first and second clusters of storage nodes of a distributed storage system. The storage targets of the second storage system each illustratively comprise one or more NVMe controllers.

The application relates to a node automatic scaling method and system of a distributed storage system, and the method comprises the following steps: obtaining network flow data and making the network flow data into a data set; building a deep learning network model for regression prediction; training the deep learning network model by using the data set to obtain a flow prediction model for network flow prediction; obtaining real-time network flow, obtaining future network flow by the flow prediction model, calculating storage data demand corresponding to the future network flow, calculating the demand node number of hadoop according to the storage data demand, if the current node number of hadoop is adapted to the demand node number, the node number of hadoop remains unchanged, otherwise, the current node number of hadoop is adjusted to adapt to the demand node number. Compared with the prior art, the application is based on hadoop for distributed storage and processing, the number of nodes is automatically scaled by learning and predicting the flow trend through a deep learning algorithm, the storage service quality is guaranteed, and the storage resources of the distributed storage system are effectively saved.

An example method for automated resource management in different environments for a container system by a storage management system includes providing, by a storage management system, a first storage resource for use in a test environment by an application on a container system, the first storage resource satisfying a first storage profile of a storage provision request; analyzing, by the storage management system, usage of the first storage resource by the application in the test environment; and providing, by the storage management system and based on the analyzing, a second storage resource that satisfies a second storage profile different from the first storage profile for use in a production environment by the application on the container system.

The application provides a distributed storage system online upgrading method and device and a storage medium, relates to the technical field of distributed storage, and comprises the following steps: acquiring node pool information of each node pool; wherein the node pool information comprises redundancy configuration information, node pool type, total node quantity and node state; performing data redundancy check on each node pool based on a preset redundancy check rule to determine a node pool to be upgraded; determining a node to be upgraded and a redundant node in the node pool to be upgraded based on the node pool information of the node pool to be upgraded; performing operation and maintenance offline operation, upgrading processing and operation and maintenance online operation on the node to be upgraded in parallel between each node pool to be upgraded and in each node pool to be upgraded until all nodes in the node pool to be upgraded are upgraded. The method improves the upgrading concurrency, shortens the upgrading period and reduces the risk of business interruption and data unavailability in the upgrading process under the premise of guaranteeing the data redundancy safety.

This application discloses a disk processing method, a distributed storage system, a device, and a storage medium. The method includes: analyzing the health status of each disk based on its operating parameters; if a disk is determined to be in a sub-healthy state, determining a target backup disk based on the attribute information of the sub-healthy disk; constructing a mapping relationship between the logical address of the sub-healthy disk and the physical address of the target backup disk; updating the address mapping information according to the mapping relationship; synchronizing the data stored in the sub-healthy disk to the target backup disk; and performing targeted processing on the IO requests corresponding to the sub-healthy disk according to the updated address mapping information; and removing the sub-healthy disk from the distributed storage system after data synchronization is complete. This method enables silent disk replacement before the sub-healthy disk is removed, avoiding impact on process and upper-layer business continuity, and avoiding impact on the performance of the distributed storage system.

This application discloses a hard disk optimization method, apparatus, electronic device, and storage medium for storage services, applied in the field of distributed storage system technology. It addresses the problem in related technologies where hard disk storage space cannot be reorganized, affecting normal system operation and user experience. The proposed method includes: identifying data to be deleted in each storage layer of the disk; deleting the data to be deleted from the corresponding storage layer; merging and reorganizing the remaining data in each storage layer; and storing the merged and reorganized data in the corresponding storage layer. This application can reorganize disk space, freeing up storage space to ensure normal system operation and improve user experience.

The application provides a load balancing method for a distributed storage system, comprising the following steps: in response to acquisition of a node expansion instruction, detecting whether current expansion association information meets a predetermined expansion condition; if the current expansion association information meets the predetermined expansion condition, acquiring original routing information of the distributed storage system; setting a slot weight for a hash slot of each original node; determining a slot block number after expansion of the distributed storage system based on a first number and a second number; determining expansion routing information of the distributed storage system according to a predetermined load balancing algorithm based on the original routing information, the slot weight of the hash slot of each original node and the slot block number after expansion; determining expansion state metadata based on the original routing information and the expansion routing information; and re-determining a node for managing each hash slot in the original node according to the expansion state metadata. The scheme improves the stability level of the distributed storage system when facing node changes.

Disclosed are a distributed storage system control method and apparatus, and a readable storage medium, relating to the technical field of information. According to the distributed storage system control method, after configuration is completed, replica data of a master replica and replica data of a slave replica in a node are written into a log. Meanwhile, transmission of a data storage instruction and writing of storage data are completed by means of the log. After storage data of the master replica is written, an acknowledgment character is fed back to a client. And after the client receives the acknowledgment character, writing of storage data of the slave replica is performed.

Disclosed are methods, systems and computer readable media for managing host access in a distributed storage system. A first array and a second array are part of a distributed storage system in which each array is a remote data facility-duplicate array in an active-active configuration. Input / Output (I / O) requests are transmitted to each array. Response times from the I / O requests are determined at the arrays a priority is assigned to one of the first array and the second array based on a comparison of the response times.

This disclosure discloses a garbage collection method and apparatus. In the method, a client of a distributed storage system may send a garbage collection request to a server of the distributed storage system. The server may determine, based on the garbage collection request, that a target unit data block is a valid data block, and send a move command to a storage device storing the target unit data block, to cause the storage device to move the target unit data block from a first storage position to a second storage position. As a server moves data between storage positions of storage devices accessed by the server, instead of a client moving data between servers, network bandwidth between the client and the server is reduced.

This application discloses a data processing method applied to a business gateway device. The method includes: if a data update operation is received from a first client device, sending the data update operation to a first distributed storage system, so that the first distributed storage system updates the data based on the data update operation, and the first distributed storage system provides a data point read service; receiving primary key information of the data corresponding to the data update operation sent by the first distributed storage system; and storing the primary key information in a second distributed storage system, so that the second distributed storage system provides a batch data query service. This application also discloses a data processing system and a storage medium.

The application discloses a data processing method and device based on distributed storage. The method comprises the following steps: a node in a distributed storage system receives a data write request, wherein the write request comprises to-be-written data and a first index; the to-be-written data is written into a target data container in a plurality of data containers, to obtain written data and a container position descriptor of the written data; the first index is updated based on the container position descriptor and the first index, to obtain a second index, and the state of a state machine corresponding to the node is updated according to the second index. The application solves the technical problem of write amplification caused by the fact that, in the write-ahead log mechanism of the related technology, both log files and data files need to be written into a storage area.

The application provides a buffer capacity processing method and device, electronic equipment, medium and product. It relates to the field of distributed storage. The method comprises the following steps: determining the request type of a network request, the request type comprising an internal request and an external request; according to the request type, adopting a preset corresponding buffer allocation strategy to allocate an initial buffer for the network request; in the data receiving process, detecting the usage state of the initial buffer; if it is detected that the capacity of the initial buffer is insufficient, dynamically adjusting the capacity of the initial buffer. Based on the buffer capacity processing method provided by the application, the buffer size can be dynamically adjusted, thereby significantly improving the performance and reliability of the distributed storage system.