69 results about "Defragmentation" patented technology

The invention discloses a repeated data deletion method and system based on ZNS SSD, and belongs to the field of repeated data deletion.The method comprises the steps that a fingerprint index table and a fingerprint updating cache are maintained, three-time opportunity repeated data deletion is achieved, and repeated data search is conducted in the fingerprint index table, the fingerprint updating cache and an address mapping table respectively; an address mapping table, an address reverse mapping table and a reference counting table are maintained, physical space garbage collection / defragmentation and write request merging are achieved, the address mapping table, the address reverse mapping table and the reference counting table conduct garbage collection with partition as granularity when free space is insufficient, and a fingerprint index table is not updated after garbage collection data migration; and instead, the mapping of fingerprints to migration-in physical pages is stored in a fingerprint updating cache, the physical pages are sequentially distributed in partitions by the fingerprint updating cache, normal data write requests and garbage collection data migration write requests are respectively cached, and the write requests which are continuous in sequence are merged and then issued. According to the method, the high-performance and low-cost efficient storage system can be constructed on the basis of the ZNS SSD at relatively low overhead.

The invention provides a data defragmentation method and a storage device. The method comprises the steps that a plurality of candidate logic intervals are determined from a plurality of logic intervals according to access popularity information, and the access popularity information reflects host access popularity corresponding to the logic intervals respectively; after the plurality of candidate logic intervals are determined, a first logic interval in the plurality of candidate logic intervals is determined as a target logic interval according to fragment level information, and the fragment level information reflects data fragmentation degrees respectively corresponding to the plurality of candidate logic intervals; and performing defragmentation on the target data belonging to the target logic interval. Therefore, the operation efficiency of the storage device can be improved.

The invention provides a data defragmentation method and a data storage system. The method comprises the steps that a plurality of candidate logic intervals are determined from a plurality of logic intervals according to access popularity information, and the access popularity information reflects host access popularity corresponding to the logic intervals respectively; after the plurality of candidate logic intervals are determined, a first logic interval in the plurality of candidate logic intervals is determined as a target logic interval according to response time information, and the response time information reflects storage device response time corresponding to the plurality of candidate logic intervals respectively; and instructing the storage device to perform defragmentation on the target data belonging to the target logic interval. Therefore, the operation efficiency of the storage device can be improved.

The invention discloses a defragmentation method and device, electronic equipment, a storage medium and a program product. According to the implementation scheme, the method comprises the steps of determining a defragmentation transaction, and a to-be-defragmented table and a to-be-defragmented B + tree corresponding to the defragmentation transaction; blocking the to-be-sorted table, and creating a temporary table corresponding to the to-be-sorted table; executing degradation processing on blocking of the to-be-sorted table, and inserting the to-be-sorted data into the temporary table to generate an inserted temporary table; operation corresponding to the logic log is redone on the inserted temporary table, and a redone temporary table and a redone temporary B + tree are obtained; and exchanging the to-be-sorted B + tree corresponding to the to-be-sorted table into the remade temporary B + tree to obtain a fragment sorting table. When the defragmentation transaction is executed, exclusive blocking does not need to be executed on the to-be-sorted table in the whole process, the to-be-sorted table is made to be in the available state, defragmentation of the to-be-sorted table is completed, and the concurrency and availability of a database system are improved.

The invention discloses an embedded file system implementation method based on SPI Flash, which comprises the following steps: reading a super block during initialization, and establishing a state mapping table of file record items and data sectors; when the file is opened, quickly matching record items through file name metadata, and if the matching fails, allocating new items; during reading and writing, aggregating data through the write cache, asynchronously updating the data to the flash memory and refreshing the mapping table; if a change occurs when the file is closed, distributing a new space and marking the original space as failure to avoid real-time erasing; and finally, garbage collection and defragmentation are executed in the idle period of the system. According to the method, the direct access to the SPI Flash is reduced through the memory mapping table, the erasing frequency is reduced by utilizing the write cache and an asynchronous mechanism, and the time-consuming erasing operation is transferred to the background, so that the processing efficiency of file operation and the real-time performance of the system are remarkably improved, and meanwhile, the service life of a storage medium is prolonged.

Methods, systems, and devices for data defragmentation control are described. A memory system may include one or more regions of logical addresses and a plurality of memory cells arranged according to a plurality of physical addresses. In some instances, data may be stored to one or more discontinuous physical addresses and it may be desirable rearrange the data to be within continuous physical addresses (e.g., it may be desirable to defragment the data). Accordingly, the data stored to the one or more discontinuous physical addresses may be arranged (e.g., rearranged) to be within continuous physical addresses based at least in part on a value stored to one or more registers of the memory system.

A host Flash Translation Layer (FTL) synchronizes host FTL operations with the drive FTL operations to reduce write amplification and over-provisioning. Embodiments of FTL synchronization map, at the host FTL software (SW) stack level, logical bands in which data is managed, referred to as host bands, to the physical bands on a drive where data is stored. The host FTL tracks validity levels of data managed in host bands to determine validity levels of data stored in corresponding physical bands, and optimizes defragmentation operations (such as garbage collection processes and trim operations) applied by the host FTL SW stack to the physical bands based on the tracked validity levels.

The invention relates to the technical field of data processing, in particular to a multi-dimensional interactive control system based on digital human demonstration, and the system comprises a state collection module which is used for calculating the space mapping relation between skeleton node displacement and joint angle values, connecting the displacement data of each skeleton node with the corresponding joint angle value, and generating a multi-dimensional state vector; and carrying out operation on displacement data of each skeleton node in the multi-dimensional state vector to generate an action feature mapping graph. According to the method, by calculating the space mapping relation between the bone node displacement and the joint angle value, the multi-dimensional state vector is generated, the action feature mapping graph is constructed, and collection and expression of human body action data can be achieved; and block division and address allocation of the memory space are performed based on the displacement data of each skeleton node in the action feature mapping graph, and memory defragmentation is executed to generate a dynamic resource allocation table, so that the system operation efficiency and the resource utilization rate are improved.

Methods, systems, and devices for increased file loads using file based optimization (FBO) and prefetch within a memory system are described. A system may support using a prefetch command to prioritize prefetching for artificial intelligence (AI) related data (or other high priority data). A host system may transmit a prefetch command for the AI-related data before performing one or more AI operations that use the data. In response, the memory system may prefetch the AI-related data and store the data in a cache. The storage of the AI-related data in the cache may be prioritized over other data types associated with lower priorities. The host system may transmit an FBO command to defragment the AI-related data in the memory system prior to the prefetch, which may reduce latency of the prefetch.

A method for defragmentation management, the method includes (a) moving, by a defragmentation process related to a set of database entries, content of a current subset of the set to a next subset of the set; (b) marking the current subset as being moved and linking the current subset to a location of the next subset; (c) receiving a user transaction write request for writing user data related to the current subset, following the moving; (d) writing the user data to the current subset and to the next subset, before the current subset is deleted; and (e) deleting the current subset when a start time of an oldest transaction follows a time of commit of the defragmentation process.

This application provides a memory defragmentation method performed by a computer device. The method includes: obtaining n idle blocks before a tail idle block in a memory of a graphic processing unit, the tail idle block located at a head of remaining unallocated idle blocks in the memory, and n being a positive integer; obtaining tail non-idle blocks in the memory, the tail non-idle blocks comprising one allocated block or a plurality of continuous allocated blocks immediately before the tail idle block in the memory; determining one or more target idle blocks in the n idle blocks, a size of the target idle block being not less than a size of the allocated blocks among the tail non-idle blocks; migrating data in the tail non-idle blocks to the target idle blocks; and updating the head of remaining unallocated idle blocks in the memory by releasing the tail non-idle blocks.

The application provides a resource fragment consolidation method, a controller, a control node and a computer cluster, and relates to the technical field of data processing. The method simulates resource scheduling according to a first resource capacity and a first task currently being executed to obtain a plurality of candidate resource scheduling paths; determines a target resource scheduling path from the plurality of candidate resource scheduling paths; and schedules the first task based on the target resource scheduling path to perform resource fragment consolidation. After each candidate resource scheduling path is used to perform resource fragment consolidation, the residual resource capacity of at least one node can reach the first resource capacity. That is, the method can actively eliminate resource fragments in the computer cluster by scheduling the first task to perform resource fragment consolidation. Compared with the method of slowing down the generation of new resource fragments from the source, the scheduling reliability of a task with high resource demand can be effectively improved.

The application relates to the technical field of big data analysis, and discloses an OT / IT heterogeneous data unified access and standardized conversion method, which comprises the following steps: acquiring a data stream and extracting service load data and metadata therefrom; converting the metadata into fixed-length hash features by using hash operation, stripping the metadata, and encapsulating the service load data and the hash features; dividing a plurality of continuous memory buffer areas corresponding to a bit mask base number of a memory space, and extracting feature offsets of the hash features by using the bit mask; sequentially appending and writing data into the buffer areas according to receiving time; and sequentially writing the data into a storage medium in batches when the memory occupation reaches a preset threshold. The application converts the device logical owner relationship into the continuity of the medium physical block by using the hash routing, suppresses write amplification, avoids system lagging caused by fragment consolidation, and guarantees the continuous and smooth data collection of a data surge.

Aspects of the disclosure provide methods, apparatuses, and non-transitory computer-readable storage mediums for receiving media data. One apparatus includes processing circuitry that receives a timed metadata track of a plurality of segments including a plurality of metadata samples. A duration of each of the plurality of segments is equal to a duration of a metadata sample included in the respective segment. A correct timing of each of the plurality of metadata samples is maintained during fragmentation and defragmentation processes that are performed on the timed metadata track. The processing circuitry determines a presentation time of one of the plurality of metadata samples. The processing circuitry processes event information included in one or more event message boxes included in the one of the plurality of metadata samples based on the presentation time of the one of the plurality of metadata samples.

The invention discloses a data writing method, device and equipment and a computer readable storage medium, which are applied to the technical field of data storage, and the method comprises the following steps: loading a secondary queue list; the second-level queue list comprises a first-level candidate queue and a second-level high-frequency acknowledgement queue; selecting a first target block and a second target block from the blocks; the updating frequency of the data in the first target block is higher than the updating frequency of the data in the second target block; determining whether a current logical block address is in a secondary queue table when it is determined that the data is written in the block to the logical block address; and if the current logic block address is in the secondary queue list, writing the current logic block address into the first target block, otherwise, writing the current logic block address into the second target block. Compared with the mode of randomly storing the data into the data blocks at present, the method has the advantages that the data with low updating frequency and the data with high updating frequency are respectively written into the different target blocks as far as possible on the basis of the secondary queue list, so that the invalid data in the blocks with low updating frequency are received very little, and the frequency of garbage collection and fragment arrangement operation is remarkably reduced.

The present application relates to the field of cloud computing container platform, and discloses a container cluster resource fragmentation arrangement method, device, equipment and storage medium, the method comprises collecting node multidimensional resource utilization index data, and constructing a dynamic coloring model;The multidimensional resource utilization index data is input into the dynamic coloring model to obtain the node fragmentation index calculated by the model;Get the node fragmentation index threshold, compare the node fragmentation index with the node fragmentation index threshold, screen out the initial node to be arranged, calculate the arrangement investment return rate of each initial node to be arranged, select the node with a return rate greater than 1 as the target node to be arranged;Determine the arrangement window period;In the arrangement window period, the Pod migration problem of the target node to be arranged is modeled as a multidimensional packing problem, and the to-be-migrated Pod is obtained through an optimization algorithm, and the target node to be arranged is arranged based on the minimum migration cost principle.The present application solves the problem of resource fragmentation recovery lag and low cluster utilization in the prior art.

A data storage device having a flash translation layer configured to handle file-system defragmentation in a manner that substantially avoids physical data movement in a flash memory. In an example embodiment, a memory controller operates to update a logical-to-physical table thereof to change association of physical addresses of sections of user data from being associated with source logical addresses to being associated with destination logical addresses of the host defragmentation requests without moving the user data in the flash memory. Such updates can reduce the number of instances in which the host addresses a non-contiguous logical-address range, which results in a beneficial reduction of the number of input / output commands sent to the data storage device and of the associated processing overhead.

One example method includes identifying a cloud object as a potential candidate for defragmentation, evaluating the cloud object to determine what portion of segments of the cloud object are expired, when the portion of expired segments meets or exceeds a threshold, segregating the expired segments and unexpired segments of the cloud object, creating a first new cloud object that includes only unexpired segments, creating a second new cloud object that includes only expired segments, and deleting the cloud object from storage.

The application provides a memory fragment arrangement method and device, electronic equipment and storage medium, and relates to the technical field of storage. The memory fragmentation index is determined by the proportion of the free pages in each partition that do not meet the memory allocation request in the total free pages in the partition and the memory weight factor of each partition, so that the memory fragmentation index is more consistent with the actual use of the memory, so that the memory fragment arrangement is performed when the memory fragmentation index is greater than the preset arrangement threshold, which is more consistent with the arrangement expectation of the memory, so that the memory arrangement is actively performed in time, the case of excessive memory fragments in the memory is avoided, the probability of meeting the memory demand of the application program is improved, and the normal operation of the application program is ensured.

A defragmentation process is provided for contiguous persistent or volatile memory regions. The defragmentation process selects and moves extents, updates extent maps, and ensures all read / write operations are consistent and uninterrupted. The defragmentation process can be applied to online maintenance defragmentation, online on-demand defragmentation, or offline defragmentation. The illustrative embodiments provide a source-destination mapping algorithm that allows for optimal defragmentation outcome with least amount of space relocation. In some embodiments, a cost-based greedy algorithm is used for source-destination mapping. Quiesce and unquiesce mechanisms allow for fine-grained access control for the extent currently being relocated by defragmentation.

This application discloses a defragmentation method, device, and medium for GoldenDB databases, relating to the field of database data processing technology. The method includes: obtaining metadata information of all tables within the cluster from the management node of the GoldenDB distributed database; filtering tables that meet the defragmentation conditions from all tables based on preset dual thresholds to accurately determine the defragmentation targets; intelligently grouping the tables to be defragmented according to a preset defragmentation strategy and generating corresponding defragmentation tasks based on the grouping results; and distributing the defragmentation tasks to the computing nodes of the GoldenDB database for execution to complete the data defragmentation operation within the cluster. Through parallel execution by grouping, the total defragmentation time is reduced compared to existing random parallel schemes. The dual threshold filtering based on table size and fragmentation rate ensures that only tables requiring defragmentation are processed, saving system resources.

A method for transaction conflict management, the method includes writing, by a transaction controller and to a given logical address within a logical address space supported by a storage system, defragmentation data related to a defragmentation process, the defragmentation data being overwritable; identifying, by transaction controller, a transaction conflict associated with the defragmentation process, and (b) one or more user transaction requests related to the given logical address; and resolving the transaction conflict based on (i) the defragmentation data being overwritable, (ii) a timing of the writing of the defragmentation data, and (iii) one or more timings of one or more user transaction operations in relation to the one or more user transaction requests.

The invention provides a method for identifying and processing a read command, a fragmented data arrangement method and a storage device. The method for identifying whether the read command belongs to the sequence flow is applied to the storage device and comprises the steps that a control component obtains a first read command sent by a host; in response to the first read command, the control component judges whether the first read command belongs to a sequence flow read command or not according to a first address space indicated by the first read command; if yes, the control component obtains the random degree of the first address space from the address space attribute table; and the control component judges whether the first read command belongs to a sequence flow read command or not again according to the random degree of the first address space.

Methods, systems, and devices for memory system logical unit number (LUN) procedures are described. A memory system may receive an indication to convert a LUN for storing LBAs associated with an application from a first type to a second type, where the second type is associated with a higher performance defragmentation process than the first type. The memory system may perform defragmentation on data associated with the LUN based on converting the LUN to the second type. The memory system may determine whether the LBAs stored in the LUN are ordered based on the defragmentation, and the memory system may operate (e.g., execute) the application based on the LBAs being ordered.

The invention provides a defragmentation method and device, equipment and a computer program product. The method comprises the steps that under the condition that the state of the electronic equipment meets a first condition, a first command is sent to a storage device of the electronic equipment; wherein the first condition at least comprises that the state of the electronic equipment is an idle state; the first command is used for indicating one or more block addresses with invalid data; the fragmentation degree of the storage device is determined, and a second command is sent to the storage device under the condition that the fragmentation degree meets a second condition; wherein the second command is used for indicating to execute defragmentation.

This application relates to the field of computer technologies, and provides a memory defragmentation method and apparatus, a device, and a storage medium. The method includes: obtaining n idle blocks located before a tail idle block in a memory, the tail idle block comprising at least one idle block among idle blocks in the memory located at a tail, and n being a positive integer; obtaining a tail non-idle block in the memory, the tail non-idle block including one block or a plurality of continuous blocks located before the tail idle block in the memory, and including at least one allocated block; determining a target idle block in the n idle blocks, a size of the target idle block being equal to or greater than a size of the at least one allocated block among the tail non-idle block; and migrating data in the tail non-idle block to the target idle block. In this application, a size of available idle blocks in the memory can be expanded, and the idle blocks in the memory can be fully used.

The application discloses a kind of based on SPI Flash's embedded file system implementation method, comprising: initialization reads superblock, establishes the state mapping table of file record item and data sector;When opening file, record item is quickly matched by file name metadata, and new item is allocated if it fails;When reading and writing, aggregate data through write cache, asynchronously update to flash memory and refresh mapping table;When closing file, if change occurs, then allocate new space and mark original space as invalid, avoid real-time erasing;Finally, garbage collection and fragmentation are arranged in system idle period.This method reduces direct access to SPI Flash through memory mapping table, reduces the frequency of erasing and writing by using write cache and asynchronous mechanism, moves time-consuming erasing operation to the background, significantly improves the processing efficiency of file operation and system real-time, while prolonging the service life of storage medium.

The invention discloses a data defragmentation method and device and a computer readable storage medium, belongs to the field of storage systems, and aims to defragmentize data to be defragmented of a target business object at a proper opportunity, firstly, the defragmentation degree of the data to be defragmented of the target business object can be determined, and then the access probability of the data to be defragmented in the future is estimated; the method comprises the following steps of: firstly, determining a yield rate of defragmentation of to-be-defragmented data by integrating a fragmentation degree and an accessed probability of the to-be-defragmented data, and finally, defragmenting the to-be-defragmented data when the yield rate reaches the standard. Decision contribution is made to the'defragmentation work 'from two dimensions of fragmentization degree and accessed probability, so that accurate and appropriate defragmentation opportunity is determined, and the situation of'unnecessary or too late' defragmentation is avoided.

A host Flash Translation Layer (FTL) synchronizes host FTL operations with the drive FTL operations to reduce write amplification and over-provisioning. Embodiments of FTL synchronization map, at the host FTL software (SW) stack level, logical bands in which data is managed, referred to as host bands, to the physical bands on a drive where data is stored. The host FTL tracks validity levels of data managed in host bands to determine validity levels of data stored in corresponding physical bands, and optimizes defragmentation operations (such as garbage collection processes and trim operations) applied by the host FTL SW stack to the physical bands based on the tracked validity levels.

The invention relates to the field of computer technology storage, in particular to a method and system for managing a large-value data storage space, and the method comprises the following steps: managing the state of a data storage block in a memory; realizing persistence of a data storage block state change event in a hard disk; the memory state change of the data storage block and the persistence process of the change event in the hard disk are separately realized. The method has the beneficial effects that the distributor is introduced, and the process separation of memory management storage space and distribution / recovery event persistence is realized through the use of the bitmap and the reference count table, so that the optimal management of defragmentation and space release is realized. Through atomization distribution, writing and metadata persistence methods, the method is suitable for a high-reliability storage scene. In the write-in stage, when data write-in fails, it is ensured that stock metadata and a data storage block are not affected; in the deletion stage, it is ensured that deletion operation does not block the reading request; and in the data adding stage, the operation is more efficient.