69 results about "IOPS" patented technology

The invention relates to the technical field of computer storage, and particularly provides a large model training storage resource dynamic allocation method, device and system, and the method comprises the steps: collecting a workload index and a storage system state index of a large model training task; inputting the real-time monitoring data into a convolutional neural network (CNN) model, and outputting a feature identifier of a current training stage; inputting the real-time data and the stage identifier into a long short-term memory network LSTM model, and predicting future bandwidth, IOPS and storage space requirements; and generating a resource allocation scheme through multi-objective optimization according to a prediction result in combination with a system state, and executing load allocation, data layering and bandwidth reservation operations. According to the method, training stage perception and resource demand prediction are realized through cooperation of the CNN and the LSTM, and dynamic allocation and advanced scheduling of storage resources are realized, so that the resource utilization rate, the training efficiency and the system stability are improved.

The invention relates to the technical field of data processing, in particular to an enterprise data asset management method, system and device based on cloud data and a storage medium, an activity sensing engine is deployed to collect indexes such as data access frequency, business key weight and cross-department calling popularity, and the indexes are processed into multi-dimensional activity factors through a rule engine; and dividing hot, warm and cold grades after calculating scores. Hot data is controlled by a trusted domain and stored in a high-IOPS cloud disk, warm data generates role self-adaptive desensitization view retention standard cloud disk, and cold data starts full mask and manual approval transfer object storage. And if the cold data is overdue, starting compliance review and physically erasing the cold data. According to the method, dynamic collaboration of data value evaluation, security policy and storage scheduling is realized, traditional static management contradictions are solved, full-life-cycle intelligent management of data is achieved, the value utilization rate is improved, the resource cost is optimized, and compliance risks are reduced.

The invention relates to the technical field of large model reasoning, in particular to a reasoning performance optimization method and system based on a large model, and the method comprises the following steps: initializing a reasoning performance optimization agent; collecting hardware environment indexes in real time, wherein the hardware environment indexes comprise a video memory utilization rate, a CPU (Central Processing Unit) exchange number, residual video card resources, storage IOPS (Input / Output Per Second) and network throughput; the method has the beneficial effects that related statistical indexes, including model types, model weight file total volume, model average sequence length, the number of tokens per second output by the model, first token time of the model, a display card list occupied by the model, the size of a KVcache block, the size of a KVcache sliding window and the like, of each model in a service system are comprehensively collected and analyzed; the system performance is comprehensively evaluated, and the defects that in the prior art, performance evaluation is not comprehensive, and a real-time monitoring mechanism for key indexes such as memory occupation and network bandwidth is lacked are overcome.

The invention relates to the technical field of data storage, and discloses a solid state disk compatibility and reliability testing method which comprises the steps of initializing a testing environment, installing an operating system, compiling and executing a testing script and configuring a compiling environment. Basic information of the solid state disk is recognized and classified, an automatic script is written, the classified basic information of the solid state disk is created into file clusters of different sizes according to the automatic script, continuous / random reading and writing are conducted, and file integrity is verified; and performing performance test on the classified basic information of the solid state disk by using a Fio tool and recording performance indexes of the solid state disk, wherein the performance indexes comprise sequential read-write speed, random IOPS and response delay, compiling an automatic script, and simulating a database load to perform small block random read-write operation. According to the invention, a plurality of modules for basic function and compatibility test, performance test, power management and reliability test, safety characteristic test and the like are integrated into a complete and automatic test scheme.

The invention discloses a hard disk temperature prediction method and device, electronic equipment and a storage medium, and relates to the technical field of computer hardware management, and the method comprises the steps: obtaining a PCIe real-time bandwidth utilization rate, a hard disk IOPS and an environment parameter, inputting the PCIe real-time bandwidth utilization rate, the hard disk IOPS and the environment parameter into a pre-trained deep learning network prediction model, the initial temperature prediction result of the hard disk is obtained and then corrected, a target temperature prediction result of the hard disk is obtained, a temperature heat dissipation strategy of the hard disk is generated based on the target temperature prediction result, and the temperature of the hard disk is dynamically regulated and controlled according to the temperature heat dissipation strategy. The problem that the heat dissipation risk of the hard disk is increased due to the fact that the obtained hard disk temperature information is lagged and the read-write performance is poor is solved, the hard disk temperature is predicted through the PCIe bandwidth actual utilization rate and the hard disk IOPS, and therefore the hard disk temperature change in the future time period can be predicted in advance, and the heat dissipation safety of the hard disk is guaranteed.

The present invention discloses a data storage method and device based on address redirection. The method comprises receiving a write request, extracting a data block identifier, data offset, and data length from the write request, determining whether the write is a large block or a small block based on the data length, querying a metadata database based on the data block identifier and data offset, and determining whether the write is a new write operation or an overwrite write operation; obtaining the bandwidth, IOPS, and latency of the HDD device within the current sliding time window to determine the load of the HDD device; writing data based on the load of the HDD device; and persisting data distribution information and address space usage information to the metadata database to complete data storage. The present invention utilizes the characteristics of the disk and the current load to perform IO scheduling between the SSD device and the HDD device, and combines the address redirection and small block write aggregation characteristics of the storage system to improve the performance of the storage system.

Systems and methods for using object storage as a primary storage tier for a file system of a virtual storage system are provided. According to one embodiment, ephemeral storage of a host on which a virtual storage system is running in the cloud is used as an external cache in which all or some subset of file system metadata may be stored, for example, to boost read IOPS and reduce latency of the read path. In various examples, external cache policies may be defined to trigger adding to, updating, and / or re-warming the external cache to accelerate access to file system metadata, thereby making a virtual storage solution that maintains all file system data and file system metadata within object storage a suitable and cost-effective platform for certain types of workloads. In some example, the working set of a workload may also be kept in the external cache to further accelerate read performance of the virtual storage system.

This invention discloses an edge-side out-of-order write reordering method for consumer-grade partitioned storage, belonging to the field of computer storage systems and flash memory management technology. Addressing the problem of degraded multi-threaded parallel write performance in consumer-grade partitioned storage due to kernel-forced sequential writes, and the insufficient capacity of the edge-side SRAM write buffer to meet large-scale out-of-order reordering requirements, this invention proposes a dynamic processing method with a two-layer reordering architecture. By dividing the high-performance SLC flash memory region in the device into page-level SLC and zone-level SLC as an extended write buffer, when processing write requests, the continuity and temperature label of the data in the SRAM buffer are judged; if the data is continuous and hot data, a direct flushing strategy is executed to the zone-level SLC; if out-of-order data occurs and the buffer is insufficient, a degradation strategy is executed to temporarily store the data in the page-level SLC; and continuity is dynamically restored when subsequent data is supplemented. This invention overcomes the limitations of small edge-side caches while releasing the parallel write potential of the device, significantly improving multi-threaded write IOPS.

A method, computer program product, and computing system for defining a plurality of IO processing queues within a storage system coupled to a storage platform, wherein the plurality of IO processing queues have differing priority levels; receiving one or more IO requests on the storage system coupled to the storage platform; assigning the one or more IO requests to one or more of the plurality of IO processing queues based, at least in part, upon a priority level of the one or more IO requests; and servicing the plurality of IO processing queues in a distributed fashion.

The invention provides a state self-repairing method and device for a thick backup volume in distributed storage, and relates to the technical field of distributed storage system state self-repairing technology.The method comprises the steps that metadata state information of the thick backup volume is obtained, and whether the volume is in a formatted state or not is judged; if the volume is in the formatting state, performing layered random sampling verification on the physical data of the volume, and identifying that the volume is a false abnormal volume or a true abnormal volume; if the volume is the false abnormal volume, only updating the metadata state into a formatting completion state; if the volume is a true abnormal volume, determining a breakpoint by adopting a binary search method; the method comprises the steps of collecting IOPS and bandwidth data of a cluster in real time, dynamically calculating a formatting quota, pausing formatting operation and storing current breakpoint coordinates when a cluster load exceeds a preset threshold value, and recovering the formatting operation from the stored breakpoint coordinates when the load is reduced to be below the threshold value. By means of the method and device, separation of metadata and physical data is effectively avoided, the repairing efficiency is remarkably improved, and storage resource consumption is reduced.

Embodiments of the present application disclose a kind of SSD equipment, the steady-state detection device, method, medium and program product of SSD equipment, the device includes: acquisition unit, acquisition unit and determination unit, wherein acquisition unit collects the IOPS data of pre-set time period, obtains multiple IOPS data;Acquisition unit is divided into multiple time periods according to time sequence in pre-set time period, obtains the IOPS data corresponding to multiple time periods from multiple IOPS data, obtains multiple IOPS data sets;Determination unit determines the average IOPS of each time period in multiple time periods according to multiple IOPS data sets, obtains multiple average IOPS values;Determine current SSD steady-state evaluation value according to multiple average IOPS values;When current SSD steady-state evaluation value is greater than pre-set threshold, determine that SSD is in steady-state data state.

The application discloses a kind of block storage performance automation test method, system, storage medium based on FIO, it is related to cloud computing and storage performance test technical field, it is applied to the cloud platform environment including management node and multiple test nodes.By running environment deployment script on management node, automatically complete test node IP management, SSH password-free configuration, firewall processing and according to X86 / ARM architecture adaptive deployment FIO;Run test execution script, precheck is carried out to test disk and time synchronization, start FIO service on multiple nodes according to preset task queue, in turn complete the stress test of covering 4K / 32K / 128K / 1M block size and random, order, 7:3 mixed read-write scene;Run result arrangement script, automatically parse FIO log, extract and uniformly output the IOPS, bandwidth and latency index of each scene.The method reduces manual operation while improving the consistency and result reliability of block storage performance test.

A storage node, a storage cluster and a method for achieving accelerating data access thereof are provided. The storage cluster includes a plurality of storage nodes. The method is performed by a processor of each of the storage nodes in the storage cluster. Each of the storage nodes includes the processor and a plurality of storage devices. The storage devices have at least two different input / output operations per second (IOPS) speeds. The method evaluates each storage device according to the IOPS speed thereof to perform an optimized sorting of the storage devices, and then migrates metadata previously stored in non-optimized storage devices to optimized storage devices so as to accelerate data access. The method further migrates the metadata stored in an optimized storage device when the IOPS speed of the optimized storage device has dropped to a threshold value.

A system is described. The system includes a processing resource and a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause the processing resource to collect telemetry data of a distributed storage system associated with a client device, monitor a first set of the IOPS values, select a first IOPS value in the first set of the IOPS values as a highest IOPS value, determine whether the first IOPS value is unequal to a current Max-IOPS parameter value and adjust the Max-IOPS parameter value to be equal to the first IOPS value upon a determination that the first IOPS value is unequal to the current Max-IOPS parameter value.

The present invention provides an SSD write performance regulation method, device, equipment and medium, which relates to the technical field of flash memory storage control. The method includes: determining a first running time according to SSD particle parameters; determining a second running time according to a preset IOPS threshold; determining a time slice coefficient according to the first running time and the second running time; determining the number M of garbage collection time slices and the number N of data writing time slices according to the time slice coefficient; judging whether it is necessary to allocate data writing time slices to a data writing module according to the number N of data writing time slices. If so, performing a data writing operation on the SSD through the data writing module; if not, judging whether it is necessary to allocate garbage collection time slices to a garbage collection module according to the number M of garbage collection time slices. If so, performing a garbage collection operation on the SSD through the garbage collection module. The present invention improves the minimum write performance of the SSD.

A shared file system performance test method, tool, device and storage medium, comprising: obtaining the startup parameters of the test node in the shared file system to start the test program to perform write operation and shift operation on the target file system; when the write file flag is 1, a data file containing pseudo-random number initialization is generated and written to a subdirectory composed of year, serial day and hour; when the shift flag is 1, the data file is shifted from the write directory to the shift target directory by directly calculating the file path and file name in the memory; the success and failure times and time consumption of writing and shifting in the write operation and the shift operation are recorded, the test result is generated and output, the technical problem that the related technology generally starts from IOPS and cannot truly reflect the business application performance is solved, the performance test data truly reflecting the business scenario is generated, and the IOPS utilization rate is improved.

The invention relates to the field of flash memory storage control, in particular to a delay monitoring device based on streaming timestamp sampling. Comprising a timestamp acquisition module which is arranged in the SSD controller and is used for acquiring a timestamp on an IO path; the delay calculation and classification module is used for calculating the collected timestamps to obtain sub-path delays; the streaming statistical unit is used for calculating average delay, maximum delay and minimum delay in real time and supporting generation of delay distribution percentile data; and the threshold value alarm and PMU linkage module is used for configuring a delay threshold value by firmware, triggering the threshold value alarm and PMU linkage module when the I / O delay exceeds the threshold value, giving an alarm and storing execution snapshots of N instructions before and after the abnormal I / O. On the premise that the area and the power consumption of the SSD controller are not obviously increased, delay monitoring with the precision of ten nanoseconds is achieved. The influence of the device on the IOPS performance is lower than 1%, and high-reliability delay diagnosis and QoS guarantee can be provided in a data center scene.

Data storage devices with an open-ended protocol can be limited in terms of input-output operations per second (IOPS) due to time spent on stop-transmission clean-up activities. A data storage device is disclosed that can increase IOPS by using command length estimation logic that stores / remembers the command length of a previous write / read command and restricts the next open-ended read / write command to the predicted / remembered length. This increases IOPS by eliminating a need to perform a stop-transmission clean-up activity.

A method of optimizing the scheduling of storage resource for a server used in cloud computing obtains a first input / output operations per seconds (IOPS) value of a virtual disk in an instance. A warning message is generated when the first IOPS value is larger than a predefined threshold value, which represents an overloaded state of the virtual disk. An predicted IOPS value is obtained and the predicted IOPS value is set as the maximum IOPS value for limiting the IOPS value of the virtual disk, thus other virtual disks are not affected by a virtual disk being overloaded, and storing efficiency of the server is improved. A server and a computer readable storage medium applying the method are also disclosed.

The present application discloses a data reading and writing method, apparatus, device, and medium, which includes: using a bilateral interface RDMA recv to receive a data reading and writing request sent by a storage client on a second node in the distributed storage system; obtaining a unilateral memory address from the data reading and writing request; performing an RDMA unilateral data reading and writing operation on the memory corresponding to the unilateral memory address in response to the data reading and writing request; using a bilateral interface RDMA send to send a reply message corresponding to the data reading and writing request to the storage client on the second node, so that the storage client on the second node receives the reply message using the bilateral interface RDMA recv, wherein the reply message is used to indicate that the operation corresponding to the data reading and writing request has been completed. This is compatible with existing network communication architectures, reduces data reading and writing latency, improves IOPS, and saves communication overhead.

The invention discloses a 6G full cloudization-oriented multi-dimensional virtualization resource intelligent allocation method and system, and belongs to the technical field of artificial intelligence, and the method is implemented by the following steps: multi-dimensional resource perception: collecting bandwidth, GPU utilization rate and computing power in real time, storing IOPS and generating feature vectors; resource demand grading: dividing the business into multiple levels of demands based on a clustering algorithm; rAG + MCP intelligent decision making: connecting a 6G full-clouding heterogeneous computing power virtualization platform through a retrieval enhancement generation technology in combination with MCP, and dynamically reasoning an optimal resource allocation strategy; 6G full-clouding heterogeneous computing power platform dynamic and static resource allocation: binding high-level demands to straight-through hardware resources, and elastically allocating low-level demands to virtualized resources; and continuous learning optimization: finely adjusting the RAG knowledge ability according to real-time feedback. According to the method, the resource scheduling real-time performance can be improved, the edge computing resource utilization rate is improved, and intelligent scheduling of application granularity is realized.

Data storage devices with an open protocol may be limited in terms of input / output operations per second (IOPS) due to the time taken to stop the transmission cleanup activity. A data storage device is disclosed that may increase IOPS by using command length estimation logic that stores / memorizes a command length of a previous write / read command and limits a next open read / write command to a predicted / memorized length. This increases IOPS by eliminating the need to perform a stop transmission cleaning activity.

Methods and systems for a networked storage system are provided. One method includes predicting an IOPS limit for a plurality of storage pools based on a maximum allowed latency of each storage pool, the maximum allowed latency determined from a relationship between the retrieved latency and a total number of IOPS from a resource data structure; identifying a storage pool whose utilization has reached a threshold value, the utilization based on a total number of IOPS directed towards the storage pool and a predicted IOPS limit; detecting a bully workload based on a numerical value determined from a total number of IOPS issued by the bully workload for the storage pool and a rising step function; and implementing a corrective action to reduce an impact of the bully workload on a victim workload.

The invention relates to the technical field of intelligent resource scheduling, in particular to an intelligent microhost virtualization resource scheduling method and system.The method comprises the following steps that a demand vector of each virtual machine in a to-be-scheduled virtual machine set is obtained, mirror image metadata is extracted to determine the architecture and instruction set demand of a central processing unit, and the demand of the central processing unit is determined; and analyzing the configuration parameters to obtain a memory demand, a central processing unit core number demand, a storage per-second input and output operation number demand and a network bandwidth demand. According to the method, virtual machine mirror image metadata and configuration parameters are extracted, a demand vector covering a CPU architecture, an instruction set, a memory, a core number, IOPS and network bandwidth is constructed, meanwhile, hardware information and real-time load data of a heterogeneous micro host are collected, a quantitative resource portrait set containing capacity and availability is generated, and the capacity and availability of the heterogeneous micro host are improved. And dynamic quantitative alignment between the resource demand and the capability is realized.

The invention belongs to the related technical field of storage system parameter adjustment, and discloses a storage system parameter adjustment and optimization method and adjustment and optimization equipment based on reinforcement learning, and the adjustment and optimization method comprises the steps: carrying out the random configuration of global parameters in an initial parameter space, and constructing a random sample set; on the basis of the random sample set, performing importance sorting on the parameters in the initial parameter space, and obtaining important parameters which have the greatest influence on the performance index IOPS to form a new parameter space; taking {important parameter configuration, IOPS} as an individual, executing a genetic algorithm to optimize and configure parameters in a new parameter space, and obtaining a genetic sample set; taking configuration of important parameters as an action, taking IOPS of a storage system after the action as a state, firstly obtaining initial network parameters of a depth deterministic strategy gradient algorithm model based on a genetic sample set, then exploring configuration of the important parameters in a new parameter space, and outputting final key parameters. Through the method, the parameters of the storage system can be automatically and accurately adjusted.

The application relates to the technical field of distributed storage, and discloses a cache data flushing flow control method and device, equipment and a storage medium, which comprises the following steps: acquiring the maximum service bandwidth and the maximum IOPS that can be carried by an HDD device at the same time; before each service thread and / or flushing thread writes data to the HDD device, calculating the bandwidth resource amount and the IOPS resource amount required by the current write operation; judging whether the bandwidth resource amount required is not more than the maximum service bandwidth and whether the IOPS resource amount required is not more than the maximum IOPS; if yes, executing the write operation and releasing the occupied bandwidth resource and IOPS resource after the write operation is executed; and if no, waiting for the release of the bandwidth resource and the IOPS resource. In this way, the HDD disk service bandwidth and IOPS can be simply and effectively controlled, the disk performance can be fully utilized, and the flushing performance of the cache data of an ICFS distributed layered storage system can be effectively improved.

Disclosed are systems, computer-readable mediums, and methods for managing client performance in a storage system. According to one embodiment, a total Input / Output Operations per Second (IOPS) pool and a read / write IOPS pool are managed for clients to ensure their write requests can be accommodated by both pools. In one example, a write request is received from a client by the storage system. A requested number of write IOPS is determined for a time period to accommodate the request. Based on the requested number of write IOPS exceeding a number of allocated write IOPS to the client for the time period, a target total IOPS for the client during the time period is determined by subtracting the number of allocated write IOPS from a number of allocated total IOPS to the client. At least a portion of the request is performed by executing the target total IOPS during the time period.

The invention provides an mClock metadata flow control method of a file system, which belongs to the technical field of cloud computing storage and comprises the following steps of: declaring an mClock structure, defining limit logic for using the mClock, and adding flow control logic to metadata operation. According to the method, the upper limit of the IOPS of the metadata of the cloud file system of the single client is limited, and on the premise of maintaining a multi-thread scene, by limiting the IOPS of the metadata operation of the single client, the occupation of storage resources of each client is effectively controlled, and resource waste caused by scrambling is avoided, so that the user experience is improved, and the user experience is improved. Meanwhile, the load pressure of the cluster is controlled to a certain extent, and the problem triggering probability and the equipment loss rate are reduced.

The invention discloses a hard disk garbage collection method and device. The method comprises the following steps of: dividing a to-be-recycled storage object into a plurality of continuous intervals with fixed sizes, and determining starting and ending intervals of to-be-migrated data according to the offset and length of the to-be-migrated data; dividing a reading scene into a single-interval scene, a cross-double-interval scene and a cross-multi-interval scene based on the interval position relationship: in the single-interval scene, if the interval is cached, directly copying data, otherwise, when the reading IOPS does not reach the upper limit, reading complete interval data to the cache; executing the same logic on each interval in a cross-double-interval scene; directly reading data in a cross-multi-interval scene and converting IOPS concurrency according to an actual length; and migrating the valid data to the new storage object and releasing the original object space. According to the method, the number of hard disk read requests is effectively reduced through an interval pre-reading mechanism, the addressing overhead of a mechanical disk magnetic head is reduced, meanwhile, the IOPS flow control conflict is relieved through cache hit, and the garbage collection efficiency and cluster capacity stability of a storage system are remarkably improved.

The application discloses a hard disk garbage collection method and device. The method comprises the following steps: dividing a storage object to be collected into a plurality of fixed-size continuous intervals, determining the starting and ending intervals of the data to be migrated according to the offset and length of the data to be migrated; dividing a reading scene into three categories, i.e., a single-interval, a cross-double-interval and a cross-multiple-interval, based on the interval position relationship; in the single-interval scene, if the interval has been cached, the data is directly copied, otherwise, the complete interval data is read to the cache when the reading IOPS does not reach the upper limit; the same logic is executed for each interval in the cross-double-interval scene; in the cross-multiple-interval scene, the data is directly read and the IOPS concurrency is converted according to the actual length; and the effective data is migrated to a new storage object and the original object space is released. The application effectively reduces the number of hard disk reading requests, reduces the mechanical disk head addressing overhead, simultaneously reduces the IOPS flow control conflict by using cache hits, and significantly improves the garbage collection efficiency of the storage system and the stability of the cluster capacity.