170 results about "Write amplification" patented technology

The invention belongs to the technical field of computer storage, and particularly relates to a solid state disk data management and efficient storage distribution method, which comprises a data characteristic sensing and classifying unit, a data characteristic sensing and classifying unit, a data processing unit, a data storage unit and a data distribution unit, wherein the data characteristic sensing and classifying unit is configured for analyzing write-in request data flow entering a solid state disk in real time or periodically, extracting various characteristic parameters associated with data blocks and storing the data blocks; classifying the data on the basis of the characteristic parameters, and distributing one or more data classification labels for each data block to be written or stored; and the self-adaptive storage allocation strategy unit is configured to be used for dynamically selecting an optimal physical block for data writing according to the data classification label and in combination with the physical block state information of the current flash memory medium. According to the invention, the solid state disk can better adapt to various dynamic and variable workloads, and especially shows more excellent continuous write-in performance, lower delay jitter and obviously prolonged service life when processing a large amount of random write-in, small block write-in and write amplification sensitive applications.

The invention discloses a dynamic self-adaptive NAND Flash reading method and system based on artificial intelligence, a medium and equipment, and relates to the technical field of solid-state storage, the method comprises the following steps: S1, initiating a reading request, reading page data by using default voltage, decoding the page data through an LDPC algorithm, constructing a feature vector and performing preprocessing, inputting the voltage into a pre-trained neural network model, and carrying out voltage prediction; s2, carrying out fine sampling on the predicted voltage, reconstructing accurate threshold voltage distribution, and calculating a log-likelihood ratio; s3, designing a shared bottom layer backbone network, enabling a neural network model to have independent output branches, predicting the voltage, and evaluating the long-term health state of the block; and S4, designing an online transfer learning mechanism, and enabling the neural network model to quickly adapt to a new environment by using a small amount of new data. According to the method, the delay is greatly reduced, the power consumption is remarkably saved, data loss is prevented through early warning of bad blocks, and the write amplification factor is reduced to 1.26 through dynamic read interference suppression.

The embodiment of the invention provides a key value data processing method, a database platform, equipment, a medium and a product, and the method comprises the steps that in response to a disk refreshing instruction of a target storage unit in a target partition, a target key value pair in a memory buffer area of the target storage unit is obtained, the target key value pair is any key value pair in the memory buffer area, and the target storage unit in the target partition is stored in the memory buffer area; the target key value pair comprises a target index key and a corresponding target data value; if the target data value is greater than the set size threshold value, writing the target data value into a data file, and writing a target index key and a target file path of the target data value in the data file into a disk storage file of a target storage unit, the structure of the data file is the same as that of the disk storage file, the data file and the disk storage file are stored in the same path, the recovery mechanisms of the data file and the disk storage file are different, and the data file and the disk storage file are distinguished through the set file mark added in the metadata area. The invasion to the database is small, and the write amplification is reduced.

The invention discloses a method, device and equipment for optimizing read interference of a physical block and a medium, and relates to the technical field of solid state hard disks, and the method comprises the following steps: in response to the fact that the read frequency of the physical block is in a preset evaluation range, carrying out quality evaluation on the physical block; and if the result of the quality evaluation shows that the quality of the physical block is superior to a preset standard, performing compensation updating on the number of reading times of the physical block to delay data migration. According to the method, the accurate quality evaluation of the physical block is triggered by setting the preset evaluation range, and the reading times of the physical block with excellent quality is compensated and updated based on the evaluation result. According to the mechanism, unnecessary data migration can be dynamically delayed, so that write amplification and flash memory wear of the solid state disk are effectively reduced. According to the method, on the premise of ensuring the data reliability, the additional write-in overhead generated due to read interference management is remarkably reduced, and finally the technical effect of prolonging the service life of the solid state disk is achieved.

The invention discloses an SSD (Solid State Disk) garbage collection method and device with RAID (Redundant Array of Independent Disk) redundancy and a medium, and relates to the technical field of garbage collection, and the method comprises the following steps: in a garbage collection process, selecting a target NAND flash memory physical block of which the effective data volume meets a collection condition, copying the effective data to a new position, and updating address mapping; querying a first RAID block to which the target block belongs and all member block identifiers thereof according to a first mapping table and a second mapping table which record a mapping relationship between the NAND flash memory physical block and the RAID blocks; and recombining other blocks except the target block in the first RAID block into a second RAID block, recalculating and storing verification data for the second RAID block, synchronously updating the two mapping tables to reflect a new composition relationship, and finally erasing the target block. The physical block of the NAND flash memory is used as the minimum recovery unit to replace the traditional recovery mode taking the whole RAID block as the unit, and the block with the minimum effective data can be preferentially recovered, so that the effective data migration amount is remarkably reduced, the write amplification coefficient is reduced, the abrasion is reduced, and the service life of a hard disk is prolonged.

The application discloses a kind of parallel computing methods of FPGA acceleration for Key-Value storage system, method includes: triggering compaction operation, the SSTable of key range repetition is transferred to FPGA memory;Using FPGA parallel decoding and sorting these SSTable, obtain ordered and non-repeated key-value pair set;Using FPGA parallel coding this set, form new SSTable, and these SSTable are directly written from FPGA memory to SSD, bypass CPU, and the invalid key-value pair in log file is recycled by parallel garbage collection operation.The application parallel computing method of FPGA acceleration for Key-Value storage system, using key-value separation method reduces the write amplification problem of key-value storage, and provides comprehensive optimization for the calculation load and data transmission delay in key-value storage, reduces the pressure of CPU.

Based on the WAF (Write Amplification Factor) value, the efficiency of writing data to non-volatile memory is improved, extending the lifespan of the non-volatile memory. [Solution] The image forming apparatus includes an image forming unit, a non-volatile memory (flash memory) for storing data, a memory controller for controlling the writing of data to the non-volatile memory, a storage device (SSD) including a storage unit that holds size information of the first data written to the non-volatile memory by the memory controller, and a main controller that outputs the second data to be written to the non-volatile memory to the memory controller. The main controller suspends the writing of the second data to the non-volatile memory if the size of the second data to be written to the non-volatile memory is less than or equal to a preset minimum size, and writes the two or more suspended second data to the non-volatile memory as a single second data if the combined size exceeds the minimum size.

Embodiments of the present application provide a log-merge tree key-value storage system and related methods and related devices, and relate to the technical field of storage. The system is based on a hybrid storage architecture, which includes a host and a persistent storage medium and a non-volatile storage medium connected to the host, wherein the persistent storage medium stores a plurality of ordered string table layers, each ordered string table layer includes a plurality of ordered string table groups, and each ordered string table group includes a plurality of ordered string tables; the log-merge tree key-value storage system adopts a grouping compression mechanism and includes two-level filters corresponding to the ordered string table groups, which include group filters corresponding to the ordered string table groups and table filters corresponding to the ordered string tables. The embodiments optimize the read amplification problem and the write amplification problem of the storage system, and improve the read performance and the write performance of the storage system.

The embodiment of the invention provides a mapping table reconstruction method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: performing first scanning processing on page data without recording a mapping snapshot in a four-layer unit QLC data block to obtain first logic conversion physical information; transferring the first logical conversion physical information to the front part of a preset logical conversion physical temporary storage table; second scanning processing is carried out on page data, without recording the mapping snapshots, in the SLC data block, and second logic conversion physical information is obtained; transferring the second logical conversion physical information to the rear part of the logical conversion physical temporary storage table; in the logical conversion physical temporary storage table, the first logical conversion physical information and the second logical conversion physical information are combined in sequence to obtain a reconstruction mapping table. According to the scheme of the embodiment of the invention, the problem of write amplification can be reduced, and the service life of the storage device is prolonged.

Sub indirection unit (IU) write ordering can be achieved by aggregating log data in local cache. The cache is IU sized. If a write command is analyzed and determined to need to be aggregated, through a host provided hint or perhaps a determination of sequential write commands, the log data is placed in the local cache to aggregate log data. The log data is aggregated in order both sequentially and atomically. The log data is written to a cache and then, when additional log data is to be aggregated with the log data, copied to a new cache along with the additional log data so that the order can be maintained. Sufficient log data is aggregated until IU size is reached at which time the entire log data of a full IU sized cache is sent to the memory device.

The application relates to the technical field of hybrid solid state disks, and discloses a hybrid solid state disk dynamic optimization method and device based on a large language model, which systematically classifies the parameter space of a hybrid solid state disk, constructs a parameter sensitivity matrix based on the classified parameters, determines the influence difference parameters of the parameters on performance under different workloads, collects system state information of the hybrid solid state disk in real time, generates an optimized parameter configuration scheme based on a large language model according to the system state information, the influence difference parameters and historical configuration information. The application can realize real-time perception of work load changes and dynamic focusing on the most sensitive parameter combination under the current load, realize collaborative reasoning and adaptive adjustment among multiple parameters, significantly reduce the write amplification rate and operation delay, and solve the problem that the response efficiency, system stability and energy consumption performance of the hybrid SSD are poor under a dynamic work load in the prior art.

The invention relates to a data storage method and device and computer equipment. The method is applied to a balanced data storage structure, the balanced data storage structure comprises a plurality of levels used for storing data, and the method comprises the steps that in the data storage process, the cross state between an upper-layer data storage set in a first level and a lower-layer data storage set in a second level in the balanced data storage structure is obtained, and the upper-layer data storage set and the lower-layer data storage set are stored according to the cross state; a single-time merging storage mode corresponding to the data in the upper-layer data storage set is determined, the data in the upper-layer data storage set is stored in the lower-layer data storage set according to the single-time merging storage mode, and the first hierarchy and the second hierarchy are any two adjacent hierarchies in the balanced data storage structure. By adopting the method, the merging times executed in the data storage process can be reduced, so that the write amplification is reduced, and the data storage speed and efficiency are improved.

The invention discloses an in-disk fast copy mapping management method and device for NAND storage, equipment and a medium, and relates to the technical field of solid state hard disks, and the method comprises the steps that a mapping table from a logic address to a physical address is maintained, and each mapping table item stores first address information and an address type identifier, the address type identifier is used for indicating that the first address information is a physical address or a logical address. And in response to a data writing operation, allocating a physical address to the source logic address, setting a mapping table entry of the source logic address to be in a first state, and at the moment, indicating that the first address information is the physical address by the address type identifier. And in response to an in-disk copy command, setting the mapping table item of the destination logic address to be in a second state, at the moment, the address type identifier indicates that the first address information is a source logic address, and establishing indirect reference between the logic addresses. Only the mapping relation is updated during copying, data are prevented from being transmitted back and forth through a bus and being written repeatedly, and therefore the copying speed is remarkably increased, writing amplification is reduced, and flash memory abrasion is reduced.

The invention discloses a duplication data deletion metadata management method based on a persistent memory. The method comprises the following steps: managing a global fingerprint table and a logic address to a fingerprint mapping cache in a DRAM (Dynamic Random Access Memory), and managing an additional redo log and a global mapping table in the persistent memory. The write-in path firstly writes metadata update into the DRAM and forms a transaction record, the transaction record is sequentially added to a persistent memory log in a group submission mode, persistence is aligned according to the read-write granularity of the persistent memory, and random write and lock competition is reduced; in the checkpoint stage, an aggregation strategy is dynamically selected according to the redundancy rate and the coverage write proportion obtained through online load detection, and therefore metadata write amplification is remarkably reduced on the premise that collapse consistency is guaranteed. Meanwhile, three self-adaptive deduplication modes are provided, fingerprint management is bypassed in the low redundancy stage, redundancy is sampled and estimated in the medium redundancy stage, and full deduplication is started in the high redundancy stage, so that the metadata management cost is stably reduced under different loads, and the system throughput is improved.

The invention discloses a data writing method applied to a storage device and the storage device. The data writing method comprises the following steps: writing data to be written into a source physical block in a storage medium according to a single-bit programming mode; backing up the data to be written into a host memory buffer area; writing the to-be-written data into a target physical block in the storage medium according to a multi-bit programming mode to complete rough programming; re-acquiring the data to be written from the host memory buffer area; and writing the to-be-written data into the target physical block in the storage medium according to a multi-bit programming mode to complete fine programming. Compared with an existing scheme, the data writing method has the advantages that intermediate data carrying is reduced, performance is improved, writing amplification of flash memory particles is reduced, and the service life of the whole storage device is prolonged.

A storage controller for writing first data to a first memory cell by performing programming of the first memory cell N-times, where N is a positive integer greater than 1, includes a write amplification manager and a central processing unit. The write amplification manager checks whether the first data is invalid data before an Nth programming of the first memory cell is performed, and the central processing unit does not perform the N-th programming of the first memory cell when the first data is the invalid data.

The present disclosure relates to the technical field of database, and particularly relates to a multi-table association query method, device, storage medium and program product. The method determines the record position of the matching data of the connection field when performing a merge connection on the corresponding data table based on each group of equivalent connection relationships, instead of directly generating and transferring complete intermediate data; and subsequently, the target data is read and the final query result is generated according to the located record position, without landing, reading and writing disk temporary files throughout the process, so that the read-write amplification problem caused by multiple disk interactions is avoided from the root, the system I / O resource consumption is reduced, and the overall execution efficiency of the multi-table association query is effectively improved.

The application provides a flash / SCM hybrid array method based on data reference rate, and relates to the field of computer storage. A RAID4 architecture for storing data in flash memory and storing check information in a new storage medium SCM is constructed, comprising: (1) initialization: the host module first initializes the flash memory and the SCM, and then initializes the metadata; (2) read / write request processing: after the first step of initialization, after receiving a read / write request, entering a read / write request processing module, which processes the read / write request initiated by the user; (3) data recovery: after the first step of initialization, after receiving a data recovery request, entering a data recovery module, which completes the data recovery work; (4) strip-based garbage collection; using the characteristics that the SCM has better durability and read / write performance than the flash medium, the overall performance of the storage system is improved. Using a strip-based garbage collection strategy, data blocks with high reference times are stored in the same strip, reducing write amplification.

The invention relates to the technical field of data processing, provides an object storage method and device based on time aggregation index, equipment and a medium, is applied to financial and medical health care service scenes, and can perform hash fragmentation processing on a target object according to a bucket-key structure and disassemble mass data into a plurality of small task units capable of being processed in parallel; metadata in each object fragment is read batch by batch by adopting a streaming processing model, so that the memory occupation amount can be effectively controlled, and high reliability and high efficiency of data reading are ensured; an index is established for each metadata according to a time aggregation mechanism, and each metadata index is compressed by adopting a ZSTD lossless data compression algorithm, so that the storage pressure and the data write amplification problem can be remarkably reduced; and determining the data type of the object fragment corresponding to each aggregation index, and storing each object fragment according to the data type to realize hierarchical efficient storage of the object.

Embodiments of the present application provide a garbage collection method, an electronic device and a storage medium. The method comprises determining a plurality of source blocks to be recycled and a corresponding destination block of each source block; performing secondary alternating programming on the plurality of source blocks to the plurality of destination blocks, and marking the source blocks that have participated in the first programming; and in response to one or more source blocks in the plurality of marked source blocks satisfying a preset condition, releasing the source blocks that satisfy the preset condition. By using secondary alternating programming and marking the source blocks that have participated in the first programming, the plurality of source blocks can interleave and program their respective valid data into the destination blocks, thereby delaying the release of each source block, and thus enabling the source block release step to be executed uniformly. Compared to recycling each source block one by one in the original manner, secondary alternating programming can reduce the additional data migration writing step, effectively suppressing the write amplification phenomenon, and thus improving the performance of the QLC storage device.

The invention provides a data deletion method and device for a block storage system, electronic equipment and a storage medium, and the method comprises the steps: responding to a deletion instruction for a logic data block, and determining a physical data unit occupied by the logic data block in a physical data region of the storage equipment; constructing a deallocation range based on the physical address information of the physical data unit; sending a de-allocation command containing the de-allocation range to the storage device to instruct the storage device to execute a de-allocation operation on the physical data unit; and after the storage equipment finishes processing the deallocation command, resetting a metadata region corresponding to the logic data block. By means of the scheme, the source of invalid data migration is effectively cut off, write amplification is directly reduced, the overall performance of a storage system is improved, and the service life of the solid state disk is prolonged.

The invention discloses a method and a device for realizing FTP (File Transfer Protocol) file restoration, on one hand, continuous data streams are restored under the conditions of disorder, retransmission, short-time packet loss and the like by analyzing an FTP control channel and combining indexed backtracking and TCP (Transmission Control Protocol) stream recombination, so that a restoration gap is reduced to the maximum extent, and the integrity and the success rate of FTP file restoration are improved; and on the other hand, by pre-filtering the non-preset protocol and cooperating with establishment of the file index information relationship of the original message corresponding to the quintuple information, the original message retention amount and disk write amplification are significantly reduced, so that the storage and I / O overhead is reduced, the system throughput is improved, and the historical retention window is prolonged.

The invention discloses a storage array control method, a controller and a storage device, a storage array physical space is divided into a plurality of storage areas, sequential writing is executed in each storage area, and array random writing is converted into sequential writing in the storage areas. When a write request is processed, new data is additionally written into the current active storage area, the mapping relation is updated, old data is marked to be invalid, the old data and a check block are prevented from being read back, and write amplification and read amplification are eliminated. The random writing performance is remarkably improved, and the service life is prolonged.

The invention relates to a storage method and device, equipment, a storage medium and a computer program product, and the method comprises the steps: receiving a storage request, dividing to-be-stored data according to the life cycle of the data, generating a plurality of data blocks, caching each data block to a corresponding logic disk, and storing the data blocks in the logic disk; and transferring and storing each data block from the corresponding logic disk to the physical disk. Wherein the storage request comprises to-be-stored data. According to the method, the data of the same or similar life cycles in a certain range is divided into one data block, and then the data block is stored, so that the uniformity of the data life cycles can be prevented from being damaged, the disk layout of the storage equipment is optimized, the effect of reducing the garbage collection cost and the write amplification factor of the storage equipment is realized, and the storage efficiency is improved. And the write amplification phenomenon is avoided, so that the storage performance of the storage equipment is improved.

A controller is configured to execute a garbage collection operation on a nonvolatile memory including blocks, which includes calculating a write amplification; calculating a first threshold changing as a number of first blocks changes based on a number of first blocks storing valid data; increasing the first threshold when the calculated write amplification is less than a target value of the write amplification, decreasing the first threshold when the calculated write amplification is greater than the target value; specifying a valid data ratio indicating a ratio of a total amount of valid data to a total amount of data able to be stored in one block for each of the first blocks; selecting each block with the corresponding valid data ratio less than the first threshold from the first blocks; and setting each selected block to a free block by copying valid data stored therein to a copy destination block.

The invention discloses a parity check code updating method, a computer readable storage medium and a solid state disk. The parity check code updating method comprises the following steps: acquiring a new write instruction; writing the write-in data corresponding to the new write-in instruction into the data block of the corresponding strip, and changing the flag bit of the strip into a first state so as to mark the parity check bit of the corresponding strip to be in a failure state; after the data is written, detecting whether the current state of the strip meets a preset condition for updating the parity check bit or not; if yes, reading data in the strip, calculating a corresponding parity check code, updating the parity check code obtained through calculation to the parity check bit of the strip, and changing the flag bit of the strip into the second state so as to mark the parity check bit of the corresponding strip to be in an effective state. In the data writing process, calculation of parity check codes is not carried out, so that the burden of real-time writing is greatly reduced, and the problem of write amplification and energy consumption are effectively reduced.

The invention provides a memory data dropping optimization method, system and device, a computer readable medium and a computer program product, and the method comprises the steps: S101, setting a first threshold value of the size of memory data when the memory data is triggered to drop according to the total storage data amount of a disk, when the size of the memory data reaches a first threshold value, writing the memory data into a disk to form a new storage file; s102, when the number of the storage files in the disk reaches a preset second threshold value, merging the memory data with the minimum storage file in the disk and then writing the merged data into the disk; s103, when the size difference between the memory data and the file stored in the disk exceeds a preset third threshold value, increasing the upper limit of a second threshold value, and repeatedly executing the steps S101 and S102; and S104, when the number of the storage files in the disk reaches a preset fourth threshold value, combining two storage files with the minimum size difference in the storage files in the disk, and repeating the steps S101 to S104. The write amplification of the storage system is reduced by times, and the system performance is improved.

The application relates to the technical field of big data analysis, and discloses an OT / IT heterogeneous data unified access and standardized conversion method, which comprises the following steps: acquiring a data stream and extracting service load data and metadata therefrom; converting the metadata into fixed-length hash features by using hash operation, stripping the metadata, and encapsulating the service load data and the hash features; dividing a plurality of continuous memory buffer areas corresponding to a bit mask base number of a memory space, and extracting feature offsets of the hash features by using the bit mask; sequentially appending and writing data into the buffer areas according to receiving time; and sequentially writing the data into a storage medium in batches when the memory occupation reaches a preset threshold. The application converts the device logical owner relationship into the continuity of the medium physical block by using the hash routing, suppresses write amplification, avoids system lagging caused by fragment consolidation, and guarantees the continuous and smooth data collection of a data surge.

The invention discloses a data processing method of a distributed block storage system and electronic equipment, and relates to the technical field of distributed block storage. At least one volume in the distributed block storage system has corresponding difference log objects in a plurality of placement groups, and the difference log objects are used for storing difference log information of blocks belonging to the same placement group. In other words, the difference log objects are dispersed in the multiple placement groups, the difference logs are stored through the multiple difference log objects, and lock conflicts of the difference log objects are reduced; and the difference log objects are dispersed in a plurality of placement groups, so that the two operations of writing the difference log and writing the data inevitably fall into the same placement group. By combining IO paths of the two, the actual IO writing times in the system are also reduced, namely, the influence of write amplification is reduced, and the additional network transmission time delay introduced by the write difference log operation is reduced; in addition, resources and scheduling overhead are saved through transaction merging, and high consistency between data difference logs and data is guaranteed.

The application discloses a kind of object data storage method, device and medium, applicable to data storage technical field.AEP file is written into AEP file with the help of the characteristics of low latency, non-volatile, long life of object data to be aggregated, control aggregation thread traverses the object data in AEP file;When object data meets the aggregation requirement, the object data that meets the aggregation requirement is aggregated into rados object, and is flushed to HDD disk to complete the storage of object data.The existing offline aggregation is adjusted from NVME disk to HDD disk data writing to AEP file, which alleviates the write amplification, flushing problem of existing offline aggregation mode, improves the writing speed and aggregation efficiency of object, reduces the number of rados objects through object aggregation, improves the reconstruction speed and system performance, and improves the space utilization and fault recovery speed of HDD disk.