304 results about "Write amplification" patented technology

The embodiment of the invention provides a data processing method and device and electronic equipment. In the embodiment, when a data writing request is received, whether a target cache region corresponding to a target file indicated by the request exists locally or not is checked firstly, if not, the target cache region is created in a local memory, and the size of the target cache region is the same as that of a storage page in an NAND Flash medium; and then, before to-be-written target data is written into the NAND Flash medium, the target data is written into the target cache region based on the size of the available storage space of the target cache region, and on this basis, when the size of the available storage space in the target cache region is a preset value, namely when the target cache region is full, the data in the target cache region is written into the NAND Flash medium. In this way, write amplification caused by the fact that the size of the target data to be written is smaller than that of the storage page can be effectively avoided, and therefore the PE service life of the NAND Flash medium is prolonged.

The invention belongs to the technical field of computer storage, and particularly relates to a solid state disk data management and efficient storage distribution method, which comprises a data characteristic sensing and classifying unit, a data characteristic sensing and classifying unit, a data processing unit, a data storage unit and a data distribution unit, wherein the data characteristic sensing and classifying unit is configured for analyzing write-in request data flow entering a solid state disk in real time or periodically, extracting various characteristic parameters associated with data blocks and storing the data blocks; classifying the data on the basis of the characteristic parameters, and distributing one or more data classification labels for each data block to be written or stored; and the self-adaptive storage allocation strategy unit is configured to be used for dynamically selecting an optimal physical block for data writing according to the data classification label and in combination with the physical block state information of the current flash memory medium. According to the invention, the solid state disk can better adapt to various dynamic and variable workloads, and especially shows more excellent continuous write-in performance, lower delay jitter and obviously prolonged service life when processing a large amount of random write-in, small block write-in and write amplification sensitive applications.

A storage device may carve out redundant array of independent disks (RAID) and reduce write amplification when writing data to the RAID. The storage device includes multiple flexible data placement drives that are configured to execute RAID techniques. A controller on the storage device may receive data from a host. The controller obtains reclaim unit handles from the data to determine locations where the data is to be stored on the flexible data placement drives. During storage, the controller stripes the data in parallel across reclaim units within different reclaim groups and endurance groups across a single flexible data placement drive or multiple flexible data placement drives. The storage device leverages the write amplification of the flexible data placement drives to reduce a write amplification factor on the RAID.

The invention discloses a FPGA (Field Programmable Gate Array) accelerated parallel computing method for a Key-Value storage system, which comprises the following steps of: triggering a compaction operation, and transmitting an SSTable with a repeated key range into an FPGA memory; decoding and sorting the SSTtables in parallel by using an FPGA (Field Programmable Gate Array) to obtain an ordered and non-repeated key value pair set; the set is encoded in parallel by using an FPGA (Field Programmable Gate Array) to form new SSTtables, the SSTtables are directly written into an SSD (Solid State Disk) from an FPGA memory, and invalid key value pairs in the log file are recycled by bypassing a CPU (Central Processing Unit) and parallel garbage recycling operation. According to the parallel computing method for FPGA acceleration of the key value storage system, a key value separation method is adopted to reduce the write amplification problem of key value storage, comprehensive optimization is provided for computing load and data transmission delay in key value storage, and the pressure of a CPU is reduced.

The invention discloses a dynamic self-adaptive NAND Flash reading method and system based on artificial intelligence, a medium and equipment, and relates to the technical field of solid-state storage, the method comprises the following steps: S1, initiating a reading request, reading page data by using default voltage, decoding the page data through an LDPC algorithm, constructing a feature vector and performing preprocessing, inputting the voltage into a pre-trained neural network model, and carrying out voltage prediction; s2, carrying out fine sampling on the predicted voltage, reconstructing accurate threshold voltage distribution, and calculating a log-likelihood ratio; s3, designing a shared bottom layer backbone network, enabling a neural network model to have independent output branches, predicting the voltage, and evaluating the long-term health state of the block; and S4, designing an online transfer learning mechanism, and enabling the neural network model to quickly adapt to a new environment by using a small amount of new data. According to the method, the delay is greatly reduced, the power consumption is remarkably saved, data loss is prevented through early warning of bad blocks, and the write amplification factor is reduced to 1.26 through dynamic read interference suppression.

The invention relates to the technical field of hybrid granularity mapping, in particular to a hybrid granularity mapping system based on data compression. Comprising a data preprocessing module, a compressed sensing block division module, a dynamic compression execution module, a mixed granularity mapping module, an entropy balance metadata module, a dynamic wear distribution module, a compressed sensing recovery triggering module and a self-adaptive strategy optimization module. The data preprocessing module is used for data stream fragmentation and hot and cold data identification; and the compressed sensing block division module is used for entropy-driven block division and compressed residual prediction. Through an entropy-driven dynamic blocking mechanism and a compressed sensing recovery strategy, data features are quantized through information entropy, a compressed residual space is predicted through LSTM, block granularity and a GC strategy are optimized in real time in combination with a genetic algorithm, optimal matching of compression efficiency and block allocation is achieved, compared with a traditional scheme, write amplification is reduced, effective storage density is improved, and the compression efficiency is improved. Meanwhile, the service life of equipment is prolonged through wear entropy balance, and the heterogeneous storage load adaptability is higher.

The embodiment of the invention provides a key value data processing method, a database platform, equipment, a medium and a product, and the method comprises the steps that in response to a disk refreshing instruction of a target storage unit in a target partition, a target key value pair in a memory buffer area of the target storage unit is obtained, the target key value pair is any key value pair in the memory buffer area, and the target storage unit in the target partition is stored in the memory buffer area; the target key value pair comprises a target index key and a corresponding target data value; if the target data value is greater than the set size threshold value, writing the target data value into a data file, and writing a target index key and a target file path of the target data value in the data file into a disk storage file of a target storage unit, the structure of the data file is the same as that of the disk storage file, the data file and the disk storage file are stored in the same path, the recovery mechanisms of the data file and the disk storage file are different, and the data file and the disk storage file are distinguished through the set file mark added in the metadata area. The invasion to the database is small, and the write amplification is reduced.

The invention provides a data processing method and related equipment, and relates to the field of database application, the method comprises the following steps: when a write request of target data is received, a storage space is applied for the target data, the storage space comprises at least one range, and the range comprises a data page and a verification page; the verification information of the first range is written into a verification page corresponding to the first range, the verification information of the first range is the verification information obtained through calculation when the target data is written into a data page corresponding to the first range, the verification information is used for conducting redundancy verification, and the first range is any range in the at least one range; and writing the first range into persistent storage. Whether the data is damaged or not can be detected by using the verification information, so that the reliability of data storage is improved, and the risk of data damage caused by a silent error and the like is reduced; and meanwhile, the problem of write amplification is reduced, the storage space occupied by configuration of additional copies is reduced, and the storage overhead and the data reliability can be effectively balanced.

Monitoring write amplification metrics for a flash-based storage system, including: generating, by a controller of a storage system, data describing write activity for each storage device of a plurality of storage devices of the storage system, wherein the data describing the write activity comprises data describing application write activity for each storage device of the plurality of storage devices; calculating, by the controller of the storage system, for each storage device of the plurality of storage devices, a corresponding write amplification metric; and providing, by the controller of the storage system, to each storage device of the plurality of storage devices, the corresponding write amplification metric.

The invention discloses a garbage collection method, device and equipment suitable for FDP and a medium, and relates to the technical field of storage technologies. Whether the number of the source virtual blocks corresponding to the low-effective-data-frame-number water level virtual blocks in the RUH linked list is larger than a certain number or not is judged in turn; under different conditions, different garbage collection strategies are adopted, and then different garbage collection results are obtained, so that the technical problems that the performance such as write reduction, write amplification and random write is prone to being affected, the overhead is large, the utilization rate of the storage space is low, the garbage collection difficulty is large, and the stability is poor are solved; according to the method, the distribution of the data on the SSD disk is more effectively managed, garbage collection is started at a proper time, operation is only carried out on the source virtual blocks really needing to be collected, dynamic adjustment is carried out according to the number of the source virtual blocks and the number of the target virtual blocks, and garbage collection is more flexible and efficient.

The disclosure configures a memory sub-system controller to perform scalable write amplification (WAF) measurement. The controller receives a request to place a memory sub-system into a WAF mode and, in response, reduces a full capacity of the memory sub-system to a reduced capacity by restricting write operations to an individual portion of a set of memory components. The controller generates a log that tracks a number of write operations performed to store data to the individual portion of the set of memory components relative to a plurality of requests to program the data to the individual portion of the memory sub-system. The controller computes, based on the log, the WAF for the full capacity of the memory sub-system based on the WAF of the reduced capacity of the memory sub-system.

The invention relates to the technical field of solid state disks, and discloses an optimization method and device for SSD open data block timeout check, computer equipment and a storage medium. The method comprises the following steps: traversing all open blocks one by one to obtain a next open block; obtaining the temperature of the current SSD, and querying a corresponding data retention coefficient; calculating according to the data retention coefficient to obtain a timeout accumulated record value of the open block; judging whether the overtime accumulated record value of the open block exceeds a processing threshold value or not; and if the overtime accumulated record value of the open block exceeds a processing threshold, triggering open block processing to obtain a processing result. According to the method, the SSD temperature is introduced to correct the timeout time of the open block, the time accuracy of timeout processing can be improved, and write amplification caused by open block processing is reduced as much as possible on the premise of avoiding the data retention problem.

System-level garbage collection is performed in an SSD-based data storage system employing log-structured writing of data. Respective reclaim unit sizes of the storage devices are identified, wherein each reclaim unit is a unit of programming and erasure used by device-level garbage collection to generate erased reclaim units for allocation for physical data storage. The data is arranged across the storage devices in first and second data units, each first data unit being of a first size U equal to the product of R and a stripe width W, each second data unit including a respective set of first data units which occupy a maximum-size reclaim unit of a storage device among those storing the first data units. The system-level garbage collection is operated with second-data-unit granularity to regularly generate empty second data units for allocation in the log-structured writing of data, reducing write amplification and device wear.

The invention provides a storage device, an electronic device, a data storage method and a storage medium, and relates to the technical field of data processing, the storage device comprises a CXL interface module, a dynamic memory module, a flash memory module and a flash memory controller module; wherein the dynamic memory module comprises a plurality of device side periodic regions, the flash memory module comprises a plurality of data placement units, and the device side periodic regions are in one-to-one correspondence with the data placement units; the CXL interface module is used for receiving host data from the host side periodic region and temporarily storing the host data in the equipment side periodic region corresponding to the host side periodic region in the dynamic memory module; the dynamic memory module is used for packaging the host data in the equipment side periodic region after preset time; and the flash memory controller module is used for storing the encapsulated host data into a data placement unit corresponding to the equipment side periodic region in the flash memory module. The storage device effectively improves the write amplification phenomenon.

The invention relates to the technical field of storage garbage recycling algorithms, in particular to a garbage recycling system based on a genetic algorithm, which comprises an environment sensing module, a dynamic excitation pool module, a gene coding module, a genetic operation module, a strategy evaluation module, a cold and hot data verification bridge module, a wear leveling compensator and an iterative deployment module. The environment sensing module is used for scene feature extraction and load mode classification. Through dynamic parameter optimization and a multi-scene adaptation mechanism, an optimal strategy parameter combination is automatically searched by utilizing a genetic algorithm, and controllable disturbance is introduced by dynamically adjusting sample weight and polynomial variation in combination with an entropy weight method, so that multi-target balance of write amplification, block efficiency and wear balance is realized; meanwhile, through a cold and hot data verification bridge and a wear leveling compensator module, the data state transition trend is predicted, parameter deviation is actively corrected, high robustness and long-term stability can still be kept in a complex load scene, and finally a closed-loop self-adaptive garbage collection strategy generation system is formed.

Disclosed in the present disclosure are a memory coding method, a memory decoding method, an apparatus, an electronic device and a storage medium. The coding method comprises: acquiring a data block to be written and a known word string corresponding to said data block; performing error correction coding on said data block and the known word string to obtain error correction information; and splicing the data block and the error correction information to obtain an error-correcting code codeword, and storing same into a memory. The known word string is introduced to undergone error correction coding together with the data block, so as to increase the length of coded data and ensure the protection capability of error-correcting code codewords, and then the data block and the error correction information are spliced to obtain the error-correcting code codeword to be written into the memory. Since the known word string is introduced for error correction coding, when the error-correcting code codeword protects one data block, the strong error correction capability of the error-correcting code codeword having high code length can still be ensured, and the memory access granularity becomes a basic access unit, such that the problems of read amplification and write amplification can be overcome when the data block is updated, thus weakening the contradiction between the code length of the error-correcting code codeword and the memory access granularity.

According to the QLC medium write amplification optimization method and device, by responding to a data write-in request, metadata of a RocksDB database is written into a metadata pool formed by TLC media, and user data is written into a data pool formed by QLC media. The service load monitoring module is used for collecting operation state data, and the intelligent load analysis module is used for judging whether the current state is in a non-service or low-service load state or not. In a low-load or non-business state, closing a garbage collection (GC) mechanism, and only executing Purge operation to recycle invalid data; and under a normal load, the GC trigger condition is controlled according to the storage space utilization rate and the data block life cycle set by the dynamic threshold configuration module. And obtaining the write-in data volume of the QLC medium through the SMART interface, and calculating the write magnification to evaluate the optimization effect. The method can be applied to electronic equipment such as a server, the write magnification of the QLC medium is effectively reduced, the service life of the QLC medium is prolonged, and the performance and stability of a storage system are improved.

The invention provides a distributed data storage method, device and equipment and a readable storage medium, and the method comprises the steps: responding to an event created by a storage target, obtaining metadata of the storage target, and submitting the metadata to a key-value pair database in a key-value pair manner; a distributed file system is called to split the specific file into distributed storage objects, and the distributed storage objects obtained through splitting are sent to object storage equipment of a distributed storage cluster; and in response to an event of storage target change, retrieving the corresponding key value pair and modifying the metadata recorded by the corresponding key value pair according to the change condition. Through the technical scheme of the specification, the metadata is stored by the key-value pair granularity, the problem of directory-level read-write amplification is relieved, single file metadata update only needs to modify the corresponding key-value pair, overall directory rewriting is avoided, and IO consumption and network delay are reduced in combination with a write-in merging mechanism of an embedded database and the transparent object splitting capability of a distributed file system.

The invention discloses a pointer-based data filling dynamic control method and system, the number of times of calling a preset interface by a flash memory controller is reduced through a flash memory cache built-in mapping table, in addition, the setting of writing a small data volume into a first flash memory storage area cache and then writing a large data volume into a second flash memory storage area is adopted, and the data filling dynamic control efficiency is improved. Therefore, the quantity of the filling data required to be filled in the data filling operation link of the single write operation is greatly reduced, so that the data volume of the data written into the storage interval of the flash memory by the host cache is reduced, the utilization rate of the bandwidth resource of the preset interface is improved, and the problem of write amplification is effectively controlled.

The invention discloses a method, device and equipment for optimizing read interference of a physical block and a medium, and relates to the technical field of solid state hard disks, and the method comprises the following steps: in response to the fact that the read frequency of the physical block is in a preset evaluation range, carrying out quality evaluation on the physical block; and if the result of the quality evaluation shows that the quality of the physical block is superior to a preset standard, performing compensation updating on the number of reading times of the physical block to delay data migration. According to the method, the accurate quality evaluation of the physical block is triggered by setting the preset evaluation range, and the reading times of the physical block with excellent quality is compensated and updated based on the evaluation result. According to the mechanism, unnecessary data migration can be dynamically delayed, so that write amplification and flash memory wear of the solid state disk are effectively reduced. According to the method, on the premise of ensuring the data reliability, the additional write-in overhead generated due to read interference management is remarkably reduced, and finally the technical effect of prolonging the service life of the solid state disk is achieved.

Provided in the embodiments of the present disclosure are a space resource allocation method and apparatus, and a system, a storage medium and a program product. In the embodiments of the present disclosure, for a storage system, a relationship model between target IO access features related to garbage collection, additional spaces and write amplification is pre-learned; and on the basis of the relationship model and respective target IO access features of a plurality of storage devices within a target time period, local additional spaces for garbage collection are allocated to the plurality of storage devices on the basis of a global additional space for garbage collection, which is provided by the storage system. Since a relationship between write amplification and additional spaces is taken into consideration when a local additional space for garbage collection is allocated to each storage device, the problem of write amplification can be alleviated on the whole.

The invention discloses an SSD (Solid State Disk) garbage collection method and device with RAID (Redundant Array of Independent Disk) redundancy and a medium, and relates to the technical field of garbage collection, and the method comprises the following steps: in a garbage collection process, selecting a target NAND flash memory physical block of which the effective data volume meets a collection condition, copying the effective data to a new position, and updating address mapping; querying a first RAID block to which the target block belongs and all member block identifiers thereof according to a first mapping table and a second mapping table which record a mapping relationship between the NAND flash memory physical block and the RAID blocks; and recombining other blocks except the target block in the first RAID block into a second RAID block, recalculating and storing verification data for the second RAID block, synchronously updating the two mapping tables to reflect a new composition relationship, and finally erasing the target block. The physical block of the NAND flash memory is used as the minimum recovery unit to replace the traditional recovery mode taking the whole RAID block as the unit, and the block with the minimum effective data can be preferentially recovered, so that the effective data migration amount is remarkably reduced, the write amplification coefficient is reduced, the abrasion is reduced, and the service life of a hard disk is prolonged.

The invention discloses a data processing method and device, electronic equipment and a readable storage medium, and relates to the technical field of data read-write.The method comprises the steps that after target data is determined, the data type and the operation frequency of the target data are obtained; determining a target data block according to the data type and the operation frequency, and distributing the target data block to a corresponding performance area in the target hard disk in advance according to the access frequency of the target data block; the data processing operation is executed based on the target data and the target data block, so that the problem that a write amplification effect exists in the data processing process can be solved, the data processing is executed based on the performance advantages of the target data block, the frequently written data is placed in an area with a smaller write amplification effect, and the data processing efficiency is improved. Therefore, the technical effects of reducing the write amplification effect based on flexible placement of the data, improving the data processing efficiency and prolonging the service life of the SSD are achieved.

The invention provides a data processing method for block storage, electronic equipment and a program product. The data processing method for block storage comprises the steps that a mapping index tree is constructed, the mapping index tree is used for uniformly managing the mapping relation between a logic block address and a physical block address, and the mapping index tree is a multi-level cache architecture comprising a cache layer, a segmented memory table and a multi-layer ordered table which are arranged in sequence; according to the received data writing request, the storage priority of the to-be-written data is determined, and the storage priority is used for representing the importance degree of the to-be-written data; according to the storage priority, writing the to-be-written data into a cache layer or a segmented memory table for storage; and when the storage data in the segmented memory table or the multi-layer ordered table meets the merging migration rule, merging migration is carried out on the storage data to the next-level storage space. According to the technical scheme, unified scheduling of cross-layer storage mapping can be achieved, and multi-layer redundancy management and the write amplification effect are reduced.

The application discloses a kind of parallel computing methods of FPGA acceleration for Key-Value storage system, method includes: triggering compaction operation, the SSTable of key range repetition is transferred to FPGA memory;Using FPGA parallel decoding and sorting these SSTable, obtain ordered and non-repeated key-value pair set;Using FPGA parallel coding this set, form new SSTable, and these SSTable are directly written from FPGA memory to SSD, bypass CPU, and the invalid key-value pair in log file is recycled by parallel garbage collection operation.The application parallel computing method of FPGA acceleration for Key-Value storage system, using key-value separation method reduces the write amplification problem of key-value storage, and provides comprehensive optimization for the calculation load and data transmission delay in key-value storage, reduces the pressure of CPU.

Based on the WAF (Write Amplification Factor) value, the efficiency of writing data to non-volatile memory is improved, extending the lifespan of the non-volatile memory. [Solution] The image forming apparatus includes an image forming unit, a non-volatile memory (flash memory) for storing data, a memory controller for controlling the writing of data to the non-volatile memory, a storage device (SSD) including a storage unit that holds size information of the first data written to the non-volatile memory by the memory controller, and a main controller that outputs the second data to be written to the non-volatile memory to the memory controller. The main controller suspends the writing of the second data to the non-volatile memory if the size of the second data to be written to the non-volatile memory is less than or equal to a preset minimum size, and writes the two or more suspended second data to the non-volatile memory as a single second data if the combined size exceeds the minimum size.

Embodiments of the present application provide a log-merge tree key-value storage system and related methods and related devices, and relate to the technical field of storage. The system is based on a hybrid storage architecture, which includes a host and a persistent storage medium and a non-volatile storage medium connected to the host, wherein the persistent storage medium stores a plurality of ordered string table layers, each ordered string table layer includes a plurality of ordered string table groups, and each ordered string table group includes a plurality of ordered string tables; the log-merge tree key-value storage system adopts a grouping compression mechanism and includes two-level filters corresponding to the ordered string table groups, which include group filters corresponding to the ordered string table groups and table filters corresponding to the ordered string tables. The embodiments optimize the read amplification problem and the write amplification problem of the storage system, and improve the read performance and the write performance of the storage system.

The embodiment of the invention provides a mapping table reconstruction method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: performing first scanning processing on page data without recording a mapping snapshot in a four-layer unit QLC data block to obtain first logic conversion physical information; transferring the first logical conversion physical information to the front part of a preset logical conversion physical temporary storage table; second scanning processing is carried out on page data, without recording the mapping snapshots, in the SLC data block, and second logic conversion physical information is obtained; transferring the second logical conversion physical information to the rear part of the logical conversion physical temporary storage table; in the logical conversion physical temporary storage table, the first logical conversion physical information and the second logical conversion physical information are combined in sequence to obtain a reconstruction mapping table. According to the scheme of the embodiment of the invention, the problem of write amplification can be reduced, and the service life of the storage device is prolonged.

The invention discloses a solid-state storage loss balancing method and system, and relates to the technical field of computer storage, and the method comprises the steps: dividing a solid-state storage device into a plurality of logic intervals according to the time sequence characteristics of data access, and calculating a corresponding thermal entropy value based on the data access behavior of each logic interval; constructing a cold and hot intensity matrix, and mapping the cold and hot intensity matrix into a corresponding storage block allocation sequence based on a binary search tree; constructing an idle storage block resource pool, covering a preset erasure interval by applying a gradient pointer mechanism, and executing cross-channel storage block allocation in combination with a channel state mask and a probabilistic selection algorithm; the storage blocks in all the channels are arranged in a descending order according to the thermal entropy values, and cross-channel migration operation is executed according to the sorting result. Storage resource allocation is optimized, and efficient layout of the storage blocks is achieved, so that the data migration frequency and storage fragmentation are reduced, the write amplification effect is reduced, and the service life of solid-state storage equipment is effectively prolonged.

Sub indirection unit (IU) write ordering can be achieved by aggregating log data in local cache. The cache is IU sized. If a write command is analyzed and determined to need to be aggregated, through a host provided hint or perhaps a determination of sequential write commands, the log data is placed in the local cache to aggregate log data. The log data is aggregated in order both sequentially and atomically. The log data is written to a cache and then, when additional log data is to be aggregated with the log data, copied to a new cache along with the additional log data so that the order can be maintained. Sufficient log data is aggregated until IU size is reached at which time the entire log data of a full IU sized cache is sent to the memory device.