224 results about "Write amplification" patented technology

The invention belongs to the technical field of computer storage, and particularly relates to a solid state disk data management and efficient storage distribution method, which comprises a data characteristic sensing and classifying unit, a data characteristic sensing and classifying unit, a data processing unit, a data storage unit and a data distribution unit, wherein the data characteristic sensing and classifying unit is configured for analyzing write-in request data flow entering a solid state disk in real time or periodically, extracting various characteristic parameters associated with data blocks and storing the data blocks; classifying the data on the basis of the characteristic parameters, and distributing one or more data classification labels for each data block to be written or stored; and the self-adaptive storage allocation strategy unit is configured to be used for dynamically selecting an optimal physical block for data writing according to the data classification label and in combination with the physical block state information of the current flash memory medium. According to the invention, the solid state disk can better adapt to various dynamic and variable workloads, and especially shows more excellent continuous write-in performance, lower delay jitter and obviously prolonged service life when processing a large amount of random write-in, small block write-in and write amplification sensitive applications.

A storage device may carve out redundant array of independent disks (RAID) and reduce write amplification when writing data to the RAID. The storage device includes multiple flexible data placement drives that are configured to execute RAID techniques. A controller on the storage device may receive data from a host. The controller obtains reclaim unit handles from the data to determine locations where the data is to be stored on the flexible data placement drives. During storage, the controller stripes the data in parallel across reclaim units within different reclaim groups and endurance groups across a single flexible data placement drive or multiple flexible data placement drives. The storage device leverages the write amplification of the flexible data placement drives to reduce a write amplification factor on the RAID.

The invention discloses a dynamic self-adaptive NAND Flash reading method and system based on artificial intelligence, a medium and equipment, and relates to the technical field of solid-state storage, the method comprises the following steps: S1, initiating a reading request, reading page data by using default voltage, decoding the page data through an LDPC algorithm, constructing a feature vector and performing preprocessing, inputting the voltage into a pre-trained neural network model, and carrying out voltage prediction; s2, carrying out fine sampling on the predicted voltage, reconstructing accurate threshold voltage distribution, and calculating a log-likelihood ratio; s3, designing a shared bottom layer backbone network, enabling a neural network model to have independent output branches, predicting the voltage, and evaluating the long-term health state of the block; and S4, designing an online transfer learning mechanism, and enabling the neural network model to quickly adapt to a new environment by using a small amount of new data. According to the method, the delay is greatly reduced, the power consumption is remarkably saved, data loss is prevented through early warning of bad blocks, and the write amplification factor is reduced to 1.26 through dynamic read interference suppression.

The embodiment of the invention provides a key value data processing method, a database platform, equipment, a medium and a product, and the method comprises the steps that in response to a disk refreshing instruction of a target storage unit in a target partition, a target key value pair in a memory buffer area of the target storage unit is obtained, the target key value pair is any key value pair in the memory buffer area, and the target storage unit in the target partition is stored in the memory buffer area; the target key value pair comprises a target index key and a corresponding target data value; if the target data value is greater than the set size threshold value, writing the target data value into a data file, and writing a target index key and a target file path of the target data value in the data file into a disk storage file of a target storage unit, the structure of the data file is the same as that of the disk storage file, the data file and the disk storage file are stored in the same path, the recovery mechanisms of the data file and the disk storage file are different, and the data file and the disk storage file are distinguished through the set file mark added in the metadata area. The invasion to the database is small, and the write amplification is reduced.

System-level garbage collection is performed in an SSD-based data storage system employing log-structured writing of data. Respective reclaim unit sizes of the storage devices are identified, wherein each reclaim unit is a unit of programming and erasure used by device-level garbage collection to generate erased reclaim units for allocation for physical data storage. The data is arranged across the storage devices in first and second data units, each first data unit being of a first size U equal to the product of R and a stripe width W, each second data unit including a respective set of first data units which occupy a maximum-size reclaim unit of a storage device among those storing the first data units. The system-level garbage collection is operated with second-data-unit granularity to regularly generate empty second data units for allocation in the log-structured writing of data, reducing write amplification and device wear.

The invention provides a storage device, an electronic device, a data storage method and a storage medium, and relates to the technical field of data processing, the storage device comprises a CXL interface module, a dynamic memory module, a flash memory module and a flash memory controller module; wherein the dynamic memory module comprises a plurality of device side periodic regions, the flash memory module comprises a plurality of data placement units, and the device side periodic regions are in one-to-one correspondence with the data placement units; the CXL interface module is used for receiving host data from the host side periodic region and temporarily storing the host data in the equipment side periodic region corresponding to the host side periodic region in the dynamic memory module; the dynamic memory module is used for packaging the host data in the equipment side periodic region after preset time; and the flash memory controller module is used for storing the encapsulated host data into a data placement unit corresponding to the equipment side periodic region in the flash memory module. The storage device effectively improves the write amplification phenomenon.

Disclosed in the present disclosure are a memory coding method, a memory decoding method, an apparatus, an electronic device and a storage medium. The coding method comprises: acquiring a data block to be written and a known word string corresponding to said data block; performing error correction coding on said data block and the known word string to obtain error correction information; and splicing the data block and the error correction information to obtain an error-correcting code codeword, and storing same into a memory. The known word string is introduced to undergone error correction coding together with the data block, so as to increase the length of coded data and ensure the protection capability of error-correcting code codewords, and then the data block and the error correction information are spliced to obtain the error-correcting code codeword to be written into the memory. Since the known word string is introduced for error correction coding, when the error-correcting code codeword protects one data block, the strong error correction capability of the error-correcting code codeword having high code length can still be ensured, and the memory access granularity becomes a basic access unit, such that the problems of read amplification and write amplification can be overcome when the data block is updated, thus weakening the contradiction between the code length of the error-correcting code codeword and the memory access granularity.

According to the QLC medium write amplification optimization method and device, by responding to a data write-in request, metadata of a RocksDB database is written into a metadata pool formed by TLC media, and user data is written into a data pool formed by QLC media. The service load monitoring module is used for collecting operation state data, and the intelligent load analysis module is used for judging whether the current state is in a non-service or low-service load state or not. In a low-load or non-business state, closing a garbage collection (GC) mechanism, and only executing Purge operation to recycle invalid data; and under a normal load, the GC trigger condition is controlled according to the storage space utilization rate and the data block life cycle set by the dynamic threshold configuration module. And obtaining the write-in data volume of the QLC medium through the SMART interface, and calculating the write magnification to evaluate the optimization effect. The method can be applied to electronic equipment such as a server, the write magnification of the QLC medium is effectively reduced, the service life of the QLC medium is prolonged, and the performance and stability of a storage system are improved.

The invention provides a distributed data storage method, device and equipment and a readable storage medium, and the method comprises the steps: responding to an event created by a storage target, obtaining metadata of the storage target, and submitting the metadata to a key-value pair database in a key-value pair manner; a distributed file system is called to split the specific file into distributed storage objects, and the distributed storage objects obtained through splitting are sent to object storage equipment of a distributed storage cluster; and in response to an event of storage target change, retrieving the corresponding key value pair and modifying the metadata recorded by the corresponding key value pair according to the change condition. Through the technical scheme of the specification, the metadata is stored by the key-value pair granularity, the problem of directory-level read-write amplification is relieved, single file metadata update only needs to modify the corresponding key-value pair, overall directory rewriting is avoided, and IO consumption and network delay are reduced in combination with a write-in merging mechanism of an embedded database and the transparent object splitting capability of a distributed file system.

The invention discloses a method, device and equipment for optimizing read interference of a physical block and a medium, and relates to the technical field of solid state hard disks, and the method comprises the following steps: in response to the fact that the read frequency of the physical block is in a preset evaluation range, carrying out quality evaluation on the physical block; and if the result of the quality evaluation shows that the quality of the physical block is superior to a preset standard, performing compensation updating on the number of reading times of the physical block to delay data migration. According to the method, the accurate quality evaluation of the physical block is triggered by setting the preset evaluation range, and the reading times of the physical block with excellent quality is compensated and updated based on the evaluation result. According to the mechanism, unnecessary data migration can be dynamically delayed, so that write amplification and flash memory wear of the solid state disk are effectively reduced. According to the method, on the premise of ensuring the data reliability, the additional write-in overhead generated due to read interference management is remarkably reduced, and finally the technical effect of prolonging the service life of the solid state disk is achieved.

Provided in the embodiments of the present disclosure are a space resource allocation method and apparatus, and a system, a storage medium and a program product. In the embodiments of the present disclosure, for a storage system, a relationship model between target IO access features related to garbage collection, additional spaces and write amplification is pre-learned; and on the basis of the relationship model and respective target IO access features of a plurality of storage devices within a target time period, local additional spaces for garbage collection are allocated to the plurality of storage devices on the basis of a global additional space for garbage collection, which is provided by the storage system. Since a relationship between write amplification and additional spaces is taken into consideration when a local additional space for garbage collection is allocated to each storage device, the problem of write amplification can be alleviated on the whole.

The invention discloses an SSD (Solid State Disk) garbage collection method and device with RAID (Redundant Array of Independent Disk) redundancy and a medium, and relates to the technical field of garbage collection, and the method comprises the following steps: in a garbage collection process, selecting a target NAND flash memory physical block of which the effective data volume meets a collection condition, copying the effective data to a new position, and updating address mapping; querying a first RAID block to which the target block belongs and all member block identifiers thereof according to a first mapping table and a second mapping table which record a mapping relationship between the NAND flash memory physical block and the RAID blocks; and recombining other blocks except the target block in the first RAID block into a second RAID block, recalculating and storing verification data for the second RAID block, synchronously updating the two mapping tables to reflect a new composition relationship, and finally erasing the target block. The physical block of the NAND flash memory is used as the minimum recovery unit to replace the traditional recovery mode taking the whole RAID block as the unit, and the block with the minimum effective data can be preferentially recovered, so that the effective data migration amount is remarkably reduced, the write amplification coefficient is reduced, the abrasion is reduced, and the service life of a hard disk is prolonged.

The invention discloses a data processing method and device, electronic equipment and a readable storage medium, and relates to the technical field of data read-write.The method comprises the steps that after target data is determined, the data type and the operation frequency of the target data are obtained; determining a target data block according to the data type and the operation frequency, and distributing the target data block to a corresponding performance area in the target hard disk in advance according to the access frequency of the target data block; the data processing operation is executed based on the target data and the target data block, so that the problem that a write amplification effect exists in the data processing process can be solved, the data processing is executed based on the performance advantages of the target data block, the frequently written data is placed in an area with a smaller write amplification effect, and the data processing efficiency is improved. Therefore, the technical effects of reducing the write amplification effect based on flexible placement of the data, improving the data processing efficiency and prolonging the service life of the SSD are achieved.

The application discloses a kind of parallel computing methods of FPGA acceleration for Key-Value storage system, method includes: triggering compaction operation, the SSTable of key range repetition is transferred to FPGA memory;Using FPGA parallel decoding and sorting these SSTable, obtain ordered and non-repeated key-value pair set;Using FPGA parallel coding this set, form new SSTable, and these SSTable are directly written from FPGA memory to SSD, bypass CPU, and the invalid key-value pair in log file is recycled by parallel garbage collection operation.The application parallel computing method of FPGA acceleration for Key-Value storage system, using key-value separation method reduces the write amplification problem of key-value storage, and provides comprehensive optimization for the calculation load and data transmission delay in key-value storage, reduces the pressure of CPU.

Based on the WAF (Write Amplification Factor) value, the efficiency of writing data to non-volatile memory is improved, extending the lifespan of the non-volatile memory. [Solution] The image forming apparatus includes an image forming unit, a non-volatile memory (flash memory) for storing data, a memory controller for controlling the writing of data to the non-volatile memory, a storage device (SSD) including a storage unit that holds size information of the first data written to the non-volatile memory by the memory controller, and a main controller that outputs the second data to be written to the non-volatile memory to the memory controller. The main controller suspends the writing of the second data to the non-volatile memory if the size of the second data to be written to the non-volatile memory is less than or equal to a preset minimum size, and writes the two or more suspended second data to the non-volatile memory as a single second data if the combined size exceeds the minimum size.

Embodiments of the present application provide a log-merge tree key-value storage system and related methods and related devices, and relate to the technical field of storage. The system is based on a hybrid storage architecture, which includes a host and a persistent storage medium and a non-volatile storage medium connected to the host, wherein the persistent storage medium stores a plurality of ordered string table layers, each ordered string table layer includes a plurality of ordered string table groups, and each ordered string table group includes a plurality of ordered string tables; the log-merge tree key-value storage system adopts a grouping compression mechanism and includes two-level filters corresponding to the ordered string table groups, which include group filters corresponding to the ordered string table groups and table filters corresponding to the ordered string tables. The embodiments optimize the read amplification problem and the write amplification problem of the storage system, and improve the read performance and the write performance of the storage system.

The embodiment of the invention provides a mapping table reconstruction method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: performing first scanning processing on page data without recording a mapping snapshot in a four-layer unit QLC data block to obtain first logic conversion physical information; transferring the first logical conversion physical information to the front part of a preset logical conversion physical temporary storage table; second scanning processing is carried out on page data, without recording the mapping snapshots, in the SLC data block, and second logic conversion physical information is obtained; transferring the second logical conversion physical information to the rear part of the logical conversion physical temporary storage table; in the logical conversion physical temporary storage table, the first logical conversion physical information and the second logical conversion physical information are combined in sequence to obtain a reconstruction mapping table. According to the scheme of the embodiment of the invention, the problem of write amplification can be reduced, and the service life of the storage device is prolonged.

Sub indirection unit (IU) write ordering can be achieved by aggregating log data in local cache. The cache is IU sized. If a write command is analyzed and determined to need to be aggregated, through a host provided hint or perhaps a determination of sequential write commands, the log data is placed in the local cache to aggregate log data. The log data is aggregated in order both sequentially and atomically. The log data is written to a cache and then, when additional log data is to be aggregated with the log data, copied to a new cache along with the additional log data so that the order can be maintained. Sufficient log data is aggregated until IU size is reached at which time the entire log data of a full IU sized cache is sent to the memory device.

A storage device may carve out redundant array of independent disks (RAID) and reduce write amplification when writing data to the RAID. The storage device includes multiple flexible data placement drives that are configured to execute RAID techniques. A controller on the storage device may receive data from a host. The controller obtains reclaim unit handles from the data to determine locations where the data is to be stored on the flexible data placement drives. During storage, the controller stripes the data in parallel across reclaim units within different reclaim groups and endurance groups across a single flexible data placement drive or multiple flexible data placement drives. The storage device leverages the write amplification of the flexible data placement drives to reduce a write amplification factor on the RAID.

A host Flash Translation Layer (FTL) synchronizes host FTL operations with the drive FTL operations to reduce write amplification and over-provisioning. Embodiments of FTL synchronization map, at the host FTL software (SW) stack level, logical bands in which data is managed, referred to as host bands, to the physical bands on a drive where data is stored. The host FTL tracks validity levels of data managed in host bands to determine validity levels of data stored in corresponding physical bands, and optimizes defragmentation operations (such as garbage collection processes and trim operations) applied by the host FTL SW stack to the physical bands based on the tracked validity levels.

The application relates to the technical field of hybrid solid state disks, and discloses a hybrid solid state disk dynamic optimization method and device based on a large language model, which systematically classifies the parameter space of a hybrid solid state disk, constructs a parameter sensitivity matrix based on the classified parameters, determines the influence difference parameters of the parameters on performance under different workloads, collects system state information of the hybrid solid state disk in real time, generates an optimized parameter configuration scheme based on a large language model according to the system state information, the influence difference parameters and historical configuration information. The application can realize real-time perception of work load changes and dynamic focusing on the most sensitive parameter combination under the current load, realize collaborative reasoning and adaptive adjustment among multiple parameters, significantly reduce the write amplification rate and operation delay, and solve the problem that the response efficiency, system stability and energy consumption performance of the hybrid SSD are poor under a dynamic work load in the prior art.

The invention relates to a data storage method and device and computer equipment. The method is applied to a balanced data storage structure, the balanced data storage structure comprises a plurality of levels used for storing data, and the method comprises the steps that in the data storage process, the cross state between an upper-layer data storage set in a first level and a lower-layer data storage set in a second level in the balanced data storage structure is obtained, and the upper-layer data storage set and the lower-layer data storage set are stored according to the cross state; a single-time merging storage mode corresponding to the data in the upper-layer data storage set is determined, the data in the upper-layer data storage set is stored in the lower-layer data storage set according to the single-time merging storage mode, and the first hierarchy and the second hierarchy are any two adjacent hierarchies in the balanced data storage structure. By adopting the method, the merging times executed in the data storage process can be reduced, so that the write amplification is reduced, and the data storage speed and efficiency are improved.

The invention discloses an in-disk fast copy mapping management method and device for NAND storage, equipment and a medium, and relates to the technical field of solid state hard disks, and the method comprises the steps that a mapping table from a logic address to a physical address is maintained, and each mapping table item stores first address information and an address type identifier, the address type identifier is used for indicating that the first address information is a physical address or a logical address. And in response to a data writing operation, allocating a physical address to the source logic address, setting a mapping table entry of the source logic address to be in a first state, and at the moment, indicating that the first address information is the physical address by the address type identifier. And in response to an in-disk copy command, setting the mapping table item of the destination logic address to be in a second state, at the moment, the address type identifier indicates that the first address information is a source logic address, and establishing indirect reference between the logic addresses. Only the mapping relation is updated during copying, data are prevented from being transmitted back and forth through a bus and being written repeatedly, and therefore the copying speed is remarkably increased, writing amplification is reduced, and flash memory abrasion is reduced.

The invention discloses a duplication data deletion metadata management method based on a persistent memory. The method comprises the following steps: managing a global fingerprint table and a logic address to a fingerprint mapping cache in a DRAM (Dynamic Random Access Memory), and managing an additional redo log and a global mapping table in the persistent memory. The write-in path firstly writes metadata update into the DRAM and forms a transaction record, the transaction record is sequentially added to a persistent memory log in a group submission mode, persistence is aligned according to the read-write granularity of the persistent memory, and random write and lock competition is reduced; in the checkpoint stage, an aggregation strategy is dynamically selected according to the redundancy rate and the coverage write proportion obtained through online load detection, and therefore metadata write amplification is remarkably reduced on the premise that collapse consistency is guaranteed. Meanwhile, three self-adaptive deduplication modes are provided, fingerprint management is bypassed in the low redundancy stage, redundancy is sampled and estimated in the medium redundancy stage, and full deduplication is started in the high redundancy stage, so that the metadata management cost is stably reduced under different loads, and the system throughput is improved.

The invention discloses a data writing method applied to a storage device and the storage device. The data writing method comprises the following steps: writing data to be written into a source physical block in a storage medium according to a single-bit programming mode; backing up the data to be written into a host memory buffer area; writing the to-be-written data into a target physical block in the storage medium according to a multi-bit programming mode to complete rough programming; re-acquiring the data to be written from the host memory buffer area; and writing the to-be-written data into the target physical block in the storage medium according to a multi-bit programming mode to complete fine programming. Compared with an existing scheme, the data writing method has the advantages that intermediate data carrying is reduced, performance is improved, writing amplification of flash memory particles is reduced, and the service life of the whole storage device is prolonged.

A storage device may carve out redundant array of independent disks (RAID) and reduce write amplification when writing data to the RAID. The storage device includes multiple flexible data placement drives that are configured to execute RAID techniques. A controller on the storage device may receive data from a host. The controller obtains reclaim unit handles from the data to determine locations where the data is to be stored on the flexible data placement drives. During storage, the controller stripes the data in parallel across reclaim units within different reclaim groups and endurance groups across a single flexible data placement drive or multiple flexible data placement drives. The storage device leverages the write amplification of the flexible data placement drives to reduce a write amplification factor on the RAID.

A storage controller for writing first data to a first memory cell by performing programming of the first memory cell N-times, where N is a positive integer greater than 1, includes a write amplification manager and a central processing unit. The write amplification manager checks whether the first data is invalid data before an Nth programming of the first memory cell is performed, and the central processing unit does not perform the N-th programming of the first memory cell when the first data is the invalid data.

The present disclosure relates to the technical field of database, and particularly relates to a multi-table association query method, device, storage medium and program product. The method determines the record position of the matching data of the connection field when performing a merge connection on the corresponding data table based on each group of equivalent connection relationships, instead of directly generating and transferring complete intermediate data; and subsequently, the target data is read and the final query result is generated according to the located record position, without landing, reading and writing disk temporary files throughout the process, so that the read-write amplification problem caused by multiple disk interactions is avoided from the root, the system I / O resource consumption is reduced, and the overall execution efficiency of the multi-table association query is effectively improved.

The application provides a flash / SCM hybrid array method based on data reference rate, and relates to the field of computer storage. A RAID4 architecture for storing data in flash memory and storing check information in a new storage medium SCM is constructed, comprising: (1) initialization: the host module first initializes the flash memory and the SCM, and then initializes the metadata; (2) read / write request processing: after the first step of initialization, after receiving a read / write request, entering a read / write request processing module, which processes the read / write request initiated by the user; (3) data recovery: after the first step of initialization, after receiving a data recovery request, entering a data recovery module, which completes the data recovery work; (4) strip-based garbage collection; using the characteristics that the SCM has better durability and read / write performance than the flash medium, the overall performance of the storage system is improved. Using a strip-based garbage collection strategy, data blocks with high reference times are stored in the same strip, reducing write amplification.

The present invention provides a ZNS-based AI large model storage acceleration algorithm. By defining the storage mode and state machine scheduling algorithm of the ZNS-based neural network learning algorithm, the ZNS-based SSD separately stores the parameters of each layer, reducing the device's demand for large-capacity memory storage, while improving the parameter import performance during the neural network algorithm learning process, reducing overall power consumption and whole-disk write amplification. The present invention combines the processing definition of the neural network learning algorithm with the host computer mode coordinated with the ZNS SSD, cooperates with the host computer to improve the computing performance of the model, and facilitates the upper-layer ecological software to mobilize the data of each Zone; the ZNS SSD cooperates with the host computer to define the timing of loading parameters for each layer, reducing the capacity demand for DRAM during the large model processing process. At the same time, the ZNS SSD enters a low-power mode when the model parameters do not need to be loaded to save the power consumption of the entire machine, which is convenient for promoting the application fields and scenarios of large models.