363 results about "Write amplification" patented technology

The invention discloses a data storage method and a related device, which can be applied to the technical field of storage. The method comprises the steps that when the load of the storage system is low, data received by the storage system is stored in a small check group mode, and the small check group comprises a data block and a check block; when the load of the storage system is high, data received by the storage system is stored in a large verification group mode, the large verification group comprises data blocks and verification blocks, and the number of the data blocks in the large verification group is larger than that of the data blocks in the small verification group. The problem of extra overhead caused by zero padding due to the fact that the data cannot be fully split is avoided, write amplification is reduced, and the performance and the service life of the disk are improved.

The embodiment of the invention provides a data processing method and device and electronic equipment. In the embodiment, when a data writing request is received, whether a target cache region corresponding to a target file indicated by the request exists locally or not is checked firstly, if not, the target cache region is created in a local memory, and the size of the target cache region is the same as that of a storage page in an NAND Flash medium; and then, before to-be-written target data is written into the NAND Flash medium, the target data is written into the target cache region based on the size of the available storage space of the target cache region, and on this basis, when the size of the available storage space in the target cache region is a preset value, namely when the target cache region is full, the data in the target cache region is written into the NAND Flash medium. In this way, write amplification caused by the fact that the size of the target data to be written is smaller than that of the storage page can be effectively avoided, and therefore the PE service life of the NAND Flash medium is prolonged.

The invention belongs to the technical field of computer storage, and particularly relates to a solid state disk data management and efficient storage distribution method, which comprises a data characteristic sensing and classifying unit, a data characteristic sensing and classifying unit, a data processing unit, a data storage unit and a data distribution unit, wherein the data characteristic sensing and classifying unit is configured for analyzing write-in request data flow entering a solid state disk in real time or periodically, extracting various characteristic parameters associated with data blocks and storing the data blocks; classifying the data on the basis of the characteristic parameters, and distributing one or more data classification labels for each data block to be written or stored; and the self-adaptive storage allocation strategy unit is configured to be used for dynamically selecting an optimal physical block for data writing according to the data classification label and in combination with the physical block state information of the current flash memory medium. According to the invention, the solid state disk can better adapt to various dynamic and variable workloads, and especially shows more excellent continuous write-in performance, lower delay jitter and obviously prolonged service life when processing a large amount of random write-in, small block write-in and write amplification sensitive applications.

System-level garbage collection is performed in an SSD-based data storage system employing log-structured writing of data. Respective reclaim unit sizes of the storage devices are identified, wherein each reclaim unit is a unit of programming and erasure used by device-level garbage collection to generate erased reclaim units for allocation for physical data storage. The data is arranged across the storage devices in first and second data units, each first data unit being of a first size U equal to the product of R and a stripe width W, each second data unit including a respective set of first data units which occupy a maximum-size reclaim unit of a storage device among those storing the first data units. The system-level garbage collection is operated with second-data-unit granularity to regularly generate empty second data units for allocation in the log-structured writing of data, reducing write amplification and device wear.

A storage device may carve out redundant array of independent disks (RAID) and reduce write amplification when writing data to the RAID. The storage device includes multiple flexible data placement drives that are configured to execute RAID techniques. A controller on the storage device may receive data from a host. The controller obtains reclaim unit handles from the data to determine locations where the data is to be stored on the flexible data placement drives. During storage, the controller stripes the data in parallel across reclaim units within different reclaim groups and endurance groups across a single flexible data placement drive or multiple flexible data placement drives. The storage device leverages the write amplification of the flexible data placement drives to reduce a write amplification factor on the RAID.

An apparatus is disclosed that includes a network interface device comprising processors to implement network interface device functionality and communication protocol engine circuitry, wherein the network interface device is to: receive a request to write data to a memory node communicably coupled to the network interface device; identify network information corresponding to the request, wherein the network information includes at least one of quality of service (QoS), physical function (PF), virtual function (VF), name space identifier (NSID), flow ID, service level objectives (SLOs), or process address space ID (PASID); identify characteristics of the memory node, wherein the characteristics include at least page size of the memory node; and cause the data to be coalesced with other data on the memory node based on the network information and the characteristics.

The invention discloses a FPGA (Field Programmable Gate Array) accelerated parallel computing method for a Key-Value storage system, which comprises the following steps of: triggering a compaction operation, and transmitting an SSTable with a repeated key range into an FPGA memory; decoding and sorting the SSTtables in parallel by using an FPGA (Field Programmable Gate Array) to obtain an ordered and non-repeated key value pair set; the set is encoded in parallel by using an FPGA (Field Programmable Gate Array) to form new SSTtables, the SSTtables are directly written into an SSD (Solid State Disk) from an FPGA memory, and invalid key value pairs in the log file are recycled by bypassing a CPU (Central Processing Unit) and parallel garbage recycling operation. According to the parallel computing method for FPGA acceleration of the key value storage system, a key value separation method is adopted to reduce the write amplification problem of key value storage, comprehensive optimization is provided for computing load and data transmission delay in key value storage, and the pressure of a CPU is reduced.

The invention discloses a dynamic self-adaptive NAND Flash reading method and system based on artificial intelligence, a medium and equipment, and relates to the technical field of solid-state storage, the method comprises the following steps: S1, initiating a reading request, reading page data by using default voltage, decoding the page data through an LDPC algorithm, constructing a feature vector and performing preprocessing, inputting the voltage into a pre-trained neural network model, and carrying out voltage prediction; s2, carrying out fine sampling on the predicted voltage, reconstructing accurate threshold voltage distribution, and calculating a log-likelihood ratio; s3, designing a shared bottom layer backbone network, enabling a neural network model to have independent output branches, predicting the voltage, and evaluating the long-term health state of the block; and S4, designing an online transfer learning mechanism, and enabling the neural network model to quickly adapt to a new environment by using a small amount of new data. According to the method, the delay is greatly reduced, the power consumption is remarkably saved, data loss is prevented through early warning of bad blocks, and the write amplification factor is reduced to 1.26 through dynamic read interference suppression.

The invention relates to a fault prediction method for a solid state disk, which relates to the field of hard disk life prediction, and comprises the following steps of: acquiring a redistribution sector count to correct a plurality of erasure equalization parameters, and performing erasure equalization change analysis to obtain erasure equalization change parameters; fault time prediction is carried out according to the erasure equilibrium change parameters and the latest erasure feature map, and fault remaining time is obtained; according to the multiple redistribution amplification coefficients, the multiple write amplification coefficients and the multiple common write coefficients, an amplified write wear coefficient is obtained through calculation, the fault remaining time is corrected, and the corrected fault remaining time is obtained and serves as a fault prediction result. According to the method, the technical problems that the dynamic use condition and the abrasion condition of the solid state disk are not comprehensively considered in a traditional solid state disk service life prediction method, and the service life prediction accuracy and reliability are insufficient can be solved; precision, accuracy and reliability of hard disk residual life prediction can be significantly improved, and potential fault risks can be identified in time.

The invention relates to the technical field of hybrid granularity mapping, in particular to a hybrid granularity mapping system based on data compression. Comprising a data preprocessing module, a compressed sensing block division module, a dynamic compression execution module, a mixed granularity mapping module, an entropy balance metadata module, a dynamic wear distribution module, a compressed sensing recovery triggering module and a self-adaptive strategy optimization module. The data preprocessing module is used for data stream fragmentation and hot and cold data identification; and the compressed sensing block division module is used for entropy-driven block division and compressed residual prediction. Through an entropy-driven dynamic blocking mechanism and a compressed sensing recovery strategy, data features are quantized through information entropy, a compressed residual space is predicted through LSTM, block granularity and a GC strategy are optimized in real time in combination with a genetic algorithm, optimal matching of compression efficiency and block allocation is achieved, compared with a traditional scheme, write amplification is reduced, effective storage density is improved, and the compression efficiency is improved. Meanwhile, the service life of equipment is prolonged through wear entropy balance, and the heterogeneous storage load adaptability is higher.

The embodiment of the invention provides a key value data processing method, a database platform, equipment, a medium and a product, and the method comprises the steps that in response to a disk refreshing instruction of a target storage unit in a target partition, a target key value pair in a memory buffer area of the target storage unit is obtained, the target key value pair is any key value pair in the memory buffer area, and the target storage unit in the target partition is stored in the memory buffer area; the target key value pair comprises a target index key and a corresponding target data value; if the target data value is greater than the set size threshold value, writing the target data value into a data file, and writing a target index key and a target file path of the target data value in the data file into a disk storage file of a target storage unit, the structure of the data file is the same as that of the disk storage file, the data file and the disk storage file are stored in the same path, the recovery mechanisms of the data file and the disk storage file are different, and the data file and the disk storage file are distinguished through the set file mark added in the metadata area. The invasion to the database is small, and the write amplification is reduced.

The invention provides a data processing method and related equipment, and relates to the field of database application, the method comprises the following steps: when a write request of target data is received, a storage space is applied for the target data, the storage space comprises at least one range, and the range comprises a data page and a verification page; the verification information of the first range is written into a verification page corresponding to the first range, the verification information of the first range is the verification information obtained through calculation when the target data is written into a data page corresponding to the first range, the verification information is used for conducting redundancy verification, and the first range is any range in the at least one range; and writing the first range into persistent storage. Whether the data is damaged or not can be detected by using the verification information, so that the reliability of data storage is improved, and the risk of data damage caused by a silent error and the like is reduced; and meanwhile, the problem of write amplification is reduced, the storage space occupied by configuration of additional copies is reduced, and the storage overhead and the data reliability can be effectively balanced.

Monitoring write amplification metrics for a flash-based storage system, including: generating, by a controller of a storage system, data describing write activity for each storage device of a plurality of storage devices of the storage system, wherein the data describing the write activity comprises data describing application write activity for each storage device of the plurality of storage devices; calculating, by the controller of the storage system, for each storage device of the plurality of storage devices, a corresponding write amplification metric; and providing, by the controller of the storage system, to each storage device of the plurality of storage devices, the corresponding write amplification metric.

The invention discloses a garbage collection method, device and equipment suitable for FDP and a medium, and relates to the technical field of storage technologies. Whether the number of the source virtual blocks corresponding to the low-effective-data-frame-number water level virtual blocks in the RUH linked list is larger than a certain number or not is judged in turn; under different conditions, different garbage collection strategies are adopted, and then different garbage collection results are obtained, so that the technical problems that the performance such as write reduction, write amplification and random write is prone to being affected, the overhead is large, the utilization rate of the storage space is low, the garbage collection difficulty is large, and the stability is poor are solved; according to the method, the distribution of the data on the SSD disk is more effectively managed, garbage collection is started at a proper time, operation is only carried out on the source virtual blocks really needing to be collected, dynamic adjustment is carried out according to the number of the source virtual blocks and the number of the target virtual blocks, and garbage collection is more flexible and efficient.

The disclosure configures a memory sub-system controller to perform scalable write amplification (WAF) measurement. The controller receives a request to place a memory sub-system into a WAF mode and, in response, reduces a full capacity of the memory sub-system to a reduced capacity by restricting write operations to an individual portion of a set of memory components. The controller generates a log that tracks a number of write operations performed to store data to the individual portion of the set of memory components relative to a plurality of requests to program the data to the individual portion of the memory sub-system. The controller computes, based on the log, the WAF for the full capacity of the memory sub-system based on the WAF of the reduced capacity of the memory sub-system.

The invention relates to the technical field of solid state disks, and discloses an optimization method and device for SSD open data block timeout check, computer equipment and a storage medium. The method comprises the following steps: traversing all open blocks one by one to obtain a next open block; obtaining the temperature of the current SSD, and querying a corresponding data retention coefficient; calculating according to the data retention coefficient to obtain a timeout accumulated record value of the open block; judging whether the overtime accumulated record value of the open block exceeds a processing threshold value or not; and if the overtime accumulated record value of the open block exceeds a processing threshold, triggering open block processing to obtain a processing result. According to the method, the SSD temperature is introduced to correct the timeout time of the open block, the time accuracy of timeout processing can be improved, and write amplification caused by open block processing is reduced as much as possible on the premise of avoiding the data retention problem.

System-level garbage collection is performed in an SSD-based data storage system employing log-structured writing of data. Respective reclaim unit sizes of the storage devices are identified, wherein each reclaim unit is a unit of programming and erasure used by device-level garbage collection to generate erased reclaim units for allocation for physical data storage. The data is arranged across the storage devices in first and second data units, each first data unit being of a first size U equal to the product of R and a stripe width W, each second data unit including a respective set of first data units which occupy a maximum-size reclaim unit of a storage device among those storing the first data units. The system-level garbage collection is operated with second-data-unit granularity to regularly generate empty second data units for allocation in the log-structured writing of data, reducing write amplification and device wear.

The invention provides a storage device, an electronic device, a data storage method and a storage medium, and relates to the technical field of data processing, the storage device comprises a CXL interface module, a dynamic memory module, a flash memory module and a flash memory controller module; wherein the dynamic memory module comprises a plurality of device side periodic regions, the flash memory module comprises a plurality of data placement units, and the device side periodic regions are in one-to-one correspondence with the data placement units; the CXL interface module is used for receiving host data from the host side periodic region and temporarily storing the host data in the equipment side periodic region corresponding to the host side periodic region in the dynamic memory module; the dynamic memory module is used for packaging the host data in the equipment side periodic region after preset time; and the flash memory controller module is used for storing the encapsulated host data into a data placement unit corresponding to the equipment side periodic region in the flash memory module. The storage device effectively improves the write amplification phenomenon.

The invention relates to the technical field of storage garbage recycling algorithms, in particular to a garbage recycling system based on a genetic algorithm, which comprises an environment sensing module, a dynamic excitation pool module, a gene coding module, a genetic operation module, a strategy evaluation module, a cold and hot data verification bridge module, a wear leveling compensator and an iterative deployment module. The environment sensing module is used for scene feature extraction and load mode classification. Through dynamic parameter optimization and a multi-scene adaptation mechanism, an optimal strategy parameter combination is automatically searched by utilizing a genetic algorithm, and controllable disturbance is introduced by dynamically adjusting sample weight and polynomial variation in combination with an entropy weight method, so that multi-target balance of write amplification, block efficiency and wear balance is realized; meanwhile, through a cold and hot data verification bridge and a wear leveling compensator module, the data state transition trend is predicted, parameter deviation is actively corrected, high robustness and long-term stability can still be kept in a complex load scene, and finally a closed-loop self-adaptive garbage collection strategy generation system is formed.

Disclosed in the present disclosure are a memory coding method, a memory decoding method, an apparatus, an electronic device and a storage medium. The coding method comprises: acquiring a data block to be written and a known word string corresponding to said data block; performing error correction coding on said data block and the known word string to obtain error correction information; and splicing the data block and the error correction information to obtain an error-correcting code codeword, and storing same into a memory. The known word string is introduced to undergone error correction coding together with the data block, so as to increase the length of coded data and ensure the protection capability of error-correcting code codewords, and then the data block and the error correction information are spliced to obtain the error-correcting code codeword to be written into the memory. Since the known word string is introduced for error correction coding, when the error-correcting code codeword protects one data block, the strong error correction capability of the error-correcting code codeword having high code length can still be ensured, and the memory access granularity becomes a basic access unit, such that the problems of read amplification and write amplification can be overcome when the data block is updated, thus weakening the contradiction between the code length of the error-correcting code codeword and the memory access granularity.

According to the QLC medium write amplification optimization method and device, by responding to a data write-in request, metadata of a RocksDB database is written into a metadata pool formed by TLC media, and user data is written into a data pool formed by QLC media. The service load monitoring module is used for collecting operation state data, and the intelligent load analysis module is used for judging whether the current state is in a non-service or low-service load state or not. In a low-load or non-business state, closing a garbage collection (GC) mechanism, and only executing Purge operation to recycle invalid data; and under a normal load, the GC trigger condition is controlled according to the storage space utilization rate and the data block life cycle set by the dynamic threshold configuration module. And obtaining the write-in data volume of the QLC medium through the SMART interface, and calculating the write magnification to evaluate the optimization effect. The method can be applied to electronic equipment such as a server, the write magnification of the QLC medium is effectively reduced, the service life of the QLC medium is prolonged, and the performance and stability of a storage system are improved.

The present application discloses a data storage method and a related apparatus, which may be applied to the technical field of storage. The method comprises: when a storage system has low load, selecting to store data received by the storage system in a small verification group mode, the small verification group comprising data blocks and verification blocks; and when the storage system has high load, selecting to store the data received by the storage system in a large verification group mode, wherein the large verification group comprises data blocks and verification blocks, and the number of data blocks in the large verification group is greater than the number of data blocks in the small verification group. The problem that extra overhead is caused by the need of padding with zeros when data cannot completely fill all required segments is avoided, write amplification is reduced, and the performance and service life of disks is improved.

The invention relates to the technical field of key value storage systems, and particularly discloses an LSM-Tree key value data layout architecture based on a partition namespace solid-state disk. Comprising the following steps: S1, when a new SSTable of an L layer is written, searching a zone with an overlapped key; s2, sorting zones with overlapped keys in the L + 1 layer according to the residual space; s3, selecting the zone with the maximum residual space from the sorted zones for writing, and if no zone containing the overlapping key exists, writing the zone into a new zone; and S4, in the selected zone, if the residual effective SSTable is less than a set threshold value K, writing the residual effective SSTable into a new zone along with the Compaction operation. According to the architecture, the overlapping performance of SSTable data in the same hierarchy is fully considered, so that invalid data residues in the Zone are reduced, overlapping key data in the same Zone can be processed at a time, the data migration amount during garbage collection is reduced, and the double-write amplification effect caused by large-granularity management and multiple data rewriting is reduced.

The invention provides a distributed data storage method, device and equipment and a readable storage medium, and the method comprises the steps: responding to an event created by a storage target, obtaining metadata of the storage target, and submitting the metadata to a key-value pair database in a key-value pair manner; a distributed file system is called to split the specific file into distributed storage objects, and the distributed storage objects obtained through splitting are sent to object storage equipment of a distributed storage cluster; and in response to an event of storage target change, retrieving the corresponding key value pair and modifying the metadata recorded by the corresponding key value pair according to the change condition. Through the technical scheme of the specification, the metadata is stored by the key-value pair granularity, the problem of directory-level read-write amplification is relieved, single file metadata update only needs to modify the corresponding key-value pair, overall directory rewriting is avoided, and IO consumption and network delay are reduced in combination with a write-in merging mechanism of an embedded database and the transparent object splitting capability of a distributed file system.

The invention discloses a pointer-based data filling dynamic control method and system, the number of times of calling a preset interface by a flash memory controller is reduced through a flash memory cache built-in mapping table, in addition, the setting of writing a small data volume into a first flash memory storage area cache and then writing a large data volume into a second flash memory storage area is adopted, and the data filling dynamic control efficiency is improved. Therefore, the quantity of the filling data required to be filled in the data filling operation link of the single write operation is greatly reduced, so that the data volume of the data written into the storage interval of the flash memory by the host cache is reduced, the utilization rate of the bandwidth resource of the preset interface is improved, and the problem of write amplification is effectively controlled.

The invention discloses a data garbage collection method, electronic equipment and a storage medium, and the data garbage collection method comprises the steps: responding to target data on a source block of the storage equipment, including valid data and invalid data, and carrying out feature processing on the valid data to obtain a feature value of the valid data; judging whether a characteristic value which is the same as the characteristic value of the effective data exists in a characteristic value table or not; wherein feature values of various preset data are stored in the feature value table; and if the feature value which is the same as the feature value of the effective data exists in the feature value table, determining that the target data on the source block is erasable data. By means of the mode, normal reading of the effective data can be guaranteed, the process that the effective data are written into the target block again to be stored is reduced, the situation of write amplification is reduced, and the host write-in speed and the access speed of the storage device are improved.

The invention relates to a data storage method and device, equipment, a readable storage medium and a program product. The method comprises the steps of obtaining a predicted cold and hot degree corresponding to to-be-written data, then storing the to-be-written data into a first storage space corresponding to the predicted cold and hot degree according to the predicted cold and hot degree so as to obtain attribute parameters of the data stored in the first storage space, and storing the to-be-written data into a second storage space according to the attribute parameters. Data stored in the first storage space can be transferred to the data blocks included in the target data group in the second storage space. By adopting the method, the write amplification problem can be relieved.

The invention discloses a method, device and equipment for optimizing read interference of a physical block and a medium, and relates to the technical field of solid state hard disks, and the method comprises the following steps: in response to the fact that the read frequency of the physical block is in a preset evaluation range, carrying out quality evaluation on the physical block; and if the result of the quality evaluation shows that the quality of the physical block is superior to a preset standard, performing compensation updating on the number of reading times of the physical block to delay data migration. According to the method, the accurate quality evaluation of the physical block is triggered by setting the preset evaluation range, and the reading times of the physical block with excellent quality is compensated and updated based on the evaluation result. According to the mechanism, unnecessary data migration can be dynamically delayed, so that write amplification and flash memory wear of the solid state disk are effectively reduced. According to the method, on the premise of ensuring the data reliability, the additional write-in overhead generated due to read interference management is remarkably reduced, and finally the technical effect of prolonging the service life of the solid state disk is achieved.

Provided in the embodiments of the present disclosure are a space resource allocation method and apparatus, and a system, a storage medium and a program product. In the embodiments of the present disclosure, for a storage system, a relationship model between target IO access features related to garbage collection, additional spaces and write amplification is pre-learned; and on the basis of the relationship model and respective target IO access features of a plurality of storage devices within a target time period, local additional spaces for garbage collection are allocated to the plurality of storage devices on the basis of a global additional space for garbage collection, which is provided by the storage system. Since a relationship between write amplification and additional spaces is taken into consideration when a local additional space for garbage collection is allocated to each storage device, the problem of write amplification can be alleviated on the whole.

The invention discloses an SSD (Solid State Disk) garbage collection method and device with RAID (Redundant Array of Independent Disk) redundancy and a medium, and relates to the technical field of garbage collection, and the method comprises the following steps: in a garbage collection process, selecting a target NAND flash memory physical block of which the effective data volume meets a collection condition, copying the effective data to a new position, and updating address mapping; querying a first RAID block to which the target block belongs and all member block identifiers thereof according to a first mapping table and a second mapping table which record a mapping relationship between the NAND flash memory physical block and the RAID blocks; and recombining other blocks except the target block in the first RAID block into a second RAID block, recalculating and storing verification data for the second RAID block, synchronously updating the two mapping tables to reflect a new composition relationship, and finally erasing the target block. The physical block of the NAND flash memory is used as the minimum recovery unit to replace the traditional recovery mode taking the whole RAID block as the unit, and the block with the minimum effective data can be preferentially recovered, so that the effective data migration amount is remarkably reduced, the write amplification coefficient is reduced, the abrasion is reduced, and the service life of a hard disk is prolonged.