48 results about "Defragmentation" patented technology

The invention discloses a defragmentation method and device, electronic equipment, a storage medium and a program product. According to the implementation scheme, the method comprises the steps of determining a defragmentation transaction, and a to-be-defragmented table and a to-be-defragmented B + tree corresponding to the defragmentation transaction; blocking the to-be-sorted table, and creating a temporary table corresponding to the to-be-sorted table; executing degradation processing on blocking of the to-be-sorted table, and inserting the to-be-sorted data into the temporary table to generate an inserted temporary table; operation corresponding to the logic log is redone on the inserted temporary table, and a redone temporary table and a redone temporary B + tree are obtained; and exchanging the to-be-sorted B + tree corresponding to the to-be-sorted table into the remade temporary B + tree to obtain a fragment sorting table. When the defragmentation transaction is executed, exclusive blocking does not need to be executed on the to-be-sorted table in the whole process, the to-be-sorted table is made to be in the available state, defragmentation of the to-be-sorted table is completed, and the concurrency and availability of a database system are improved.

The invention discloses an embedded file system implementation method based on SPI Flash, which comprises the following steps: reading a super block during initialization, and establishing a state mapping table of file record items and data sectors; when the file is opened, quickly matching record items through file name metadata, and if the matching fails, allocating new items; during reading and writing, aggregating data through the write cache, asynchronously updating the data to the flash memory and refreshing the mapping table; if a change occurs when the file is closed, distributing a new space and marking the original space as failure to avoid real-time erasing; and finally, garbage collection and defragmentation are executed in the idle period of the system. According to the method, the direct access to the SPI Flash is reduced through the memory mapping table, the erasing frequency is reduced by utilizing the write cache and an asynchronous mechanism, and the time-consuming erasing operation is transferred to the background, so that the processing efficiency of file operation and the real-time performance of the system are remarkably improved, and meanwhile, the service life of a storage medium is prolonged.

Methods, systems, and devices for increased file loads using file based optimization (FBO) and prefetch within a memory system are described. A system may support using a prefetch command to prioritize prefetching for artificial intelligence (AI) related data (or other high priority data). A host system may transmit a prefetch command for the AI-related data before performing one or more AI operations that use the data. In response, the memory system may prefetch the AI-related data and store the data in a cache. The storage of the AI-related data in the cache may be prioritized over other data types associated with lower priorities. The host system may transmit an FBO command to defragment the AI-related data in the memory system prior to the prefetch, which may reduce latency of the prefetch.

A method for defragmentation management, the method includes (a) moving, by a defragmentation process related to a set of database entries, content of a current subset of the set to a next subset of the set; (b) marking the current subset as being moved and linking the current subset to a location of the next subset; (c) receiving a user transaction write request for writing user data related to the current subset, following the moving; (d) writing the user data to the current subset and to the next subset, before the current subset is deleted; and (e) deleting the current subset when a start time of an oldest transaction follows a time of commit of the defragmentation process.

The application provides a resource fragment consolidation method, a controller, a control node and a computer cluster, and relates to the technical field of data processing. The method simulates resource scheduling according to a first resource capacity and a first task currently being executed to obtain a plurality of candidate resource scheduling paths; determines a target resource scheduling path from the plurality of candidate resource scheduling paths; and schedules the first task based on the target resource scheduling path to perform resource fragment consolidation. After each candidate resource scheduling path is used to perform resource fragment consolidation, the residual resource capacity of at least one node can reach the first resource capacity. That is, the method can actively eliminate resource fragments in the computer cluster by scheduling the first task to perform resource fragment consolidation. Compared with the method of slowing down the generation of new resource fragments from the source, the scheduling reliability of a task with high resource demand can be effectively improved.

The application relates to the technical field of big data analysis, and discloses an OT / IT heterogeneous data unified access and standardized conversion method, which comprises the following steps: acquiring a data stream and extracting service load data and metadata therefrom; converting the metadata into fixed-length hash features by using hash operation, stripping the metadata, and encapsulating the service load data and the hash features; dividing a plurality of continuous memory buffer areas corresponding to a bit mask base number of a memory space, and extracting feature offsets of the hash features by using the bit mask; sequentially appending and writing data into the buffer areas according to receiving time; and sequentially writing the data into a storage medium in batches when the memory occupation reaches a preset threshold. The application converts the device logical owner relationship into the continuity of the medium physical block by using the hash routing, suppresses write amplification, avoids system lagging caused by fragment consolidation, and guarantees the continuous and smooth data collection of a data surge.

The invention discloses a data writing method, device and equipment and a computer readable storage medium, which are applied to the technical field of data storage, and the method comprises the following steps: loading a secondary queue list; the second-level queue list comprises a first-level candidate queue and a second-level high-frequency acknowledgement queue; selecting a first target block and a second target block from the blocks; the updating frequency of the data in the first target block is higher than the updating frequency of the data in the second target block; determining whether a current logical block address is in a secondary queue table when it is determined that the data is written in the block to the logical block address; and if the current logic block address is in the secondary queue list, writing the current logic block address into the first target block, otherwise, writing the current logic block address into the second target block. Compared with the mode of randomly storing the data into the data blocks at present, the method has the advantages that the data with low updating frequency and the data with high updating frequency are respectively written into the different target blocks as far as possible on the basis of the secondary queue list, so that the invalid data in the blocks with low updating frequency are received very little, and the frequency of garbage collection and fragment arrangement operation is remarkably reduced.

The present application relates to the field of cloud computing container platform, and discloses a container cluster resource fragmentation arrangement method, device, equipment and storage medium, the method comprises collecting node multidimensional resource utilization index data, and constructing a dynamic coloring model;The multidimensional resource utilization index data is input into the dynamic coloring model to obtain the node fragmentation index calculated by the model;Get the node fragmentation index threshold, compare the node fragmentation index with the node fragmentation index threshold, screen out the initial node to be arranged, calculate the arrangement investment return rate of each initial node to be arranged, select the node with a return rate greater than 1 as the target node to be arranged;Determine the arrangement window period;In the arrangement window period, the Pod migration problem of the target node to be arranged is modeled as a multidimensional packing problem, and the to-be-migrated Pod is obtained through an optimization algorithm, and the target node to be arranged is arranged based on the minimum migration cost principle.The present application solves the problem of resource fragmentation recovery lag and low cluster utilization in the prior art.

One example method includes identifying a cloud object as a potential candidate for defragmentation, evaluating the cloud object to determine what portion of segments of the cloud object are expired, when the portion of expired segments meets or exceeds a threshold, segregating the expired segments and unexpired segments of the cloud object, creating a first new cloud object that includes only unexpired segments, creating a second new cloud object that includes only expired segments, and deleting the cloud object from storage.

The application provides a memory fragment arrangement method and device, electronic equipment and storage medium, and relates to the technical field of storage. The memory fragmentation index is determined by the proportion of the free pages in each partition that do not meet the memory allocation request in the total free pages in the partition and the memory weight factor of each partition, so that the memory fragmentation index is more consistent with the actual use of the memory, so that the memory fragment arrangement is performed when the memory fragmentation index is greater than the preset arrangement threshold, which is more consistent with the arrangement expectation of the memory, so that the memory arrangement is actively performed in time, the case of excessive memory fragments in the memory is avoided, the probability of meeting the memory demand of the application program is improved, and the normal operation of the application program is ensured.

A defragmentation process is provided for contiguous persistent or volatile memory regions. The defragmentation process selects and moves extents, updates extent maps, and ensures all read / write operations are consistent and uninterrupted. The defragmentation process can be applied to online maintenance defragmentation, online on-demand defragmentation, or offline defragmentation. The illustrative embodiments provide a source-destination mapping algorithm that allows for optimal defragmentation outcome with least amount of space relocation. In some embodiments, a cost-based greedy algorithm is used for source-destination mapping. Quiesce and unquiesce mechanisms allow for fine-grained access control for the extent currently being relocated by defragmentation.

This application discloses a defragmentation method, device, and medium for GoldenDB databases, relating to the field of database data processing technology. The method includes: obtaining metadata information of all tables within the cluster from the management node of the GoldenDB distributed database; filtering tables that meet the defragmentation conditions from all tables based on preset dual thresholds to accurately determine the defragmentation targets; intelligently grouping the tables to be defragmented according to a preset defragmentation strategy and generating corresponding defragmentation tasks based on the grouping results; and distributing the defragmentation tasks to the computing nodes of the GoldenDB database for execution to complete the data defragmentation operation within the cluster. Through parallel execution by grouping, the total defragmentation time is reduced compared to existing random parallel schemes. The dual threshold filtering based on table size and fragmentation rate ensures that only tables requiring defragmentation are processed, saving system resources.

A method for transaction conflict management, the method includes writing, by a transaction controller and to a given logical address within a logical address space supported by a storage system, defragmentation data related to a defragmentation process, the defragmentation data being overwritable; identifying, by transaction controller, a transaction conflict associated with the defragmentation process, and (b) one or more user transaction requests related to the given logical address; and resolving the transaction conflict based on (i) the defragmentation data being overwritable, (ii) a timing of the writing of the defragmentation data, and (iii) one or more timings of one or more user transaction operations in relation to the one or more user transaction requests.

The invention provides a method for identifying and processing a read command, a fragmented data arrangement method and a storage device. The method for identifying whether the read command belongs to the sequence flow is applied to the storage device and comprises the steps that a control component obtains a first read command sent by a host; in response to the first read command, the control component judges whether the first read command belongs to a sequence flow read command or not according to a first address space indicated by the first read command; if yes, the control component obtains the random degree of the first address space from the address space attribute table; and the control component judges whether the first read command belongs to a sequence flow read command or not again according to the random degree of the first address space.

Methods, systems, and devices for memory system logical unit number (LUN) procedures are described. A memory system may receive an indication to convert a LUN for storing LBAs associated with an application from a first type to a second type, where the second type is associated with a higher performance defragmentation process than the first type. The memory system may perform defragmentation on data associated with the LUN based on converting the LUN to the second type. The memory system may determine whether the LBAs stored in the LUN are ordered based on the defragmentation, and the memory system may operate (e.g., execute) the application based on the LBAs being ordered.

The invention provides a defragmentation method and device, equipment and a computer program product. The method comprises the steps that under the condition that the state of the electronic equipment meets a first condition, a first command is sent to a storage device of the electronic equipment; wherein the first condition at least comprises that the state of the electronic equipment is an idle state; the first command is used for indicating one or more block addresses with invalid data; the fragmentation degree of the storage device is determined, and a second command is sent to the storage device under the condition that the fragmentation degree meets a second condition; wherein the second command is used for indicating to execute defragmentation.

This application relates to the field of computer technologies, and provides a memory defragmentation method and apparatus, a device, and a storage medium. The method includes: obtaining n idle blocks located before a tail idle block in a memory, the tail idle block comprising at least one idle block among idle blocks in the memory located at a tail, and n being a positive integer; obtaining a tail non-idle block in the memory, the tail non-idle block including one block or a plurality of continuous blocks located before the tail idle block in the memory, and including at least one allocated block; determining a target idle block in the n idle blocks, a size of the target idle block being equal to or greater than a size of the at least one allocated block among the tail non-idle block; and migrating data in the tail non-idle block to the target idle block. In this application, a size of available idle blocks in the memory can be expanded, and the idle blocks in the memory can be fully used.

The application discloses a kind of based on SPI Flash's embedded file system implementation method, comprising: initialization reads superblock, establishes the state mapping table of file record item and data sector;When opening file, record item is quickly matched by file name metadata, and new item is allocated if it fails;When reading and writing, aggregate data through write cache, asynchronously update to flash memory and refresh mapping table;When closing file, if change occurs, then allocate new space and mark original space as invalid, avoid real-time erasing;Finally, garbage collection and fragmentation are arranged in system idle period.This method reduces direct access to SPI Flash through memory mapping table, reduces the frequency of erasing and writing by using write cache and asynchronous mechanism, moves time-consuming erasing operation to the background, significantly improves the processing efficiency of file operation and system real-time, while prolonging the service life of storage medium.

A host Flash Translation Layer (FTL) synchronizes host FTL operations with the drive FTL operations to reduce write amplification and over-provisioning. Embodiments of FTL synchronization map, at the host FTL software (SW) stack level, logical bands in which data is managed, referred to as host bands, to the physical bands on a drive where data is stored. The host FTL tracks validity levels of data managed in host bands to determine validity levels of data stored in corresponding physical bands, and optimizes defragmentation operations (such as garbage collection processes and trim operations) applied by the host FTL SW stack to the physical bands based on the tracked validity levels.

Techniques are provided for implementing a defragmentation process during a merge operation performed by a re-compaction process upon a log structured merge tree. The log structured merge tree is used to store keys of key-value pairs within a key-value store. As the log structured merge tree fills with keys over time, the re-compaction process is performed to merge keys down to lower levels of the log structured merge tree to re-compact the keys. Re-compaction can result in fragmentation because there is a lack of spatial locality of where the re-compaction operations re-writes the keys within storage. Fragmentation increases read and write amplification when accessing the keys stored in different locations within the storage. Accordingly, the defragmentation process is performed during a last merge operation of the re-compaction process in order to store keys together within the storage, thus reducing read and write amplification when accessing the keys.

The invention provides a resource defragmentation method, a controller, a control node and a computer cluster, and relates to the technical field of data processing. The method comprises the following steps: simulating resource scheduling according to a first resource capacity and a first task which is being executed at present to obtain a plurality of candidate resource scheduling paths; determining a target resource scheduling path from the plurality of candidate resource scheduling paths; and based on the target resource scheduling path, scheduling the first task to perform resource defragmentation. After each candidate resource scheduling path is adopted for resource defragmentation, the residual resource capacity of at least one node can reach the first resource capacity. According to the method, the resource fragments in the computer cluster can be actively eliminated by scheduling the first task to arrange the resource fragments. Compared with a mode of slowing down generation of new resource fragments from the source, the scheduling reliability of tasks with relatively high resource requirements can be effectively improved.

Methods, systems, and devices for boot operations and logical block addresses are described. A memory system may identify logical block addresses (LBAs) that are more frequently accessed during the boot phase relative to other LBAs. Such LBAs may be referred to herein as boot LBAs. For example, the memory system may maintain a counter for each LBA read during the boot phase, which may enable the memory system to track and record the frequency of each LBA accessed during the boot phase. Using the information gathered about the boot LBAs (e.g., the quantity of accesses of each boot LBA), the memory system may perform a defragmentation process, which may be triggered by a host system, to consolidate the boot LBAs within a threshold quantity of memory blocks. In some examples, defragmentation of the boot LBAs may also be based on a sequence of the boot LBAs.

Techniques are provided for implementing a defragmentation process during a merge operation performed by a re-compaction process upon a log structured merge tree. The log structured merge tree is used to store keys of key-value pairs within a key-value store. As the log structured merge tree fills with keys over time, the re-compaction process is performed to merge keys down to lower levels of the log structured merge tree to re-compact the keys. Re-compaction can result in fragmentation because there is a lack of spatial locality of where the re-compaction operations re-writes the keys within storage. Fragmentation increases read and write amplification when accessing the keys stored in different locations within the storage. Accordingly, the defragmentation process is performed during a last merge operation of the re-compaction process in order to store keys together within the storage, thus reducing read and write amplification when accessing the keys.

This document describes methods, systems, and devices for handling increased file loads using file-based optimization (FBO) and prefetching within a storage system. A system can support the use of a prefetch command to prioritize the prefetching of artificial intelligence (AI)-related data (or other high-priority data). A host system can transmit a prefetch command for the AI-related data before performing one or more AI operations that utilize the data. In response, the storage system can prefetch the AI-related data and store it in a cache. Storing the AI-related data in the cache can be prioritized over other, lower-priority data types. The host system can also transmit an FBO command to defragment the AI-related data in the storage system prior to prefetching, which can reduce prefetching latency.

The application relates to a high-performance storage device operation optimization method and system, and belongs to the technical field of data reading and writing. The method comprises the following steps: acquiring the response time of an I / O port of a high-performance storage device within a preset historical time period and calculating a defragmentation rate through historical defragmentation data of the high-performance storage device; based on the relationship between the response time of the I / O port and the defragmentation rate, a comparison table is constructed, and a target defragmentation rate range with a response time less than a preset time threshold is screened out; if the current disk fragmentation rate is greater than a preset fragmentation rate threshold, the current I / O port queue depth is acquired; based on the current I / O port queue depth, the current disk fragmentation rate and the target defragmentation rate range, the target defragmentation rate is determined, and defragmentation is performed. Through the above mode, the adaptation capability and operation stability of the high-performance storage device in a high-load and high-real-time scene can be significantly enhanced, and the core performance advantage of the high-performance storage device can be fully exerted.

The embodiment of the present application provides a memory fragmentation arrangement method and device, the method comprises the following steps: predicting an application program which can be started by a user in a preset time period after a current time, and if memory resources required by the application program are greater than preset memory resources, performing memory fragmentation arrangement. The memory fragmentation problem can be predicted in advance and the memory fragmentation arrangement is performed, so that the normal operation of the application program is ensured and the user experience is improved.

The full-disk defragmentation triggering method and related device provided by the embodiments of the present application relate to the technical field of terminals. The method comprises the following steps: determining whether to trigger the execution of FDD according to the IO state of the UFS device, the remaining life of the UFS device, and the time interval from the last execution of FDD, and dynamically calculating the interval time required for the next execution of FDD. In this way, the execution of FDD can be triggered on demand according to the state of the UFS device, such as the IO state and the remaining life, so that the freezing of the electronic device can be reduced, and the wear and tear on the UFS device can be reduced.

The invention relates to the technical field of data processing, and provides an object storage defragmentation method, device and equipment and a medium, which are applied to the medical field and the financial field, and the method comprises the following steps: carrying out real-time acquisition by using a first monitoring strategy to obtain a first data set; performing prediction processing on the first data set to obtain a predicted fragment rate; performing real-time acquisition by using a second monitoring strategy to obtain a second data set; performing decision processing on the predicted fragment rate and the second data set to obtain an arrangement priority; generating by utilizing the arrangement priority to obtain an arrangement instruction; and sorting the fragment sorting task according to the sorting instruction to obtain a sorting result. By implementing the embodiment of the invention, a real-time monitoring strategy, a prediction strategy and a decision strategy are introduced, a storage state and a service load can be dynamically perceived, and a fragment rate trend can be accurately predicted, so that an intelligent and self-adaptive fragment arrangement decision is adopted, resource scrambling and performance jitter are avoided, the system stability and response efficiency are improved, and the arrangement efficiency is improved.

This invention relates to the field of data security protection technology, and in particular to a method for big data security protection. The method includes the following steps: acquiring system operation data; mapping the storage locations of the system operation data to obtain a data location mapping table; performing real-time data encryption based on the data location mapping table to obtain an encrypted data storage table; acquiring system memory data; performing reference counting statistics on the system memory data based on the encrypted data storage table to obtain a data reference status table; performing residual detection analysis on the data reference status table to obtain a data residual marker set; performing multi-level cache clearing of residual data based on the data residual marker set to obtain a memory cleanup verification table; and performing defragmentation optimization on the memory cleanup verification table to obtain an optimized memory layout diagram. This invention effectively solves the weaknesses in dynamic data protection in existing technologies, significantly improving the security and reliability of big data systems.

The invention discloses a cross-disk block-level recombination migration method supporting capacity self-adaption, and relates to the technical field of computer storage management, and the method specifically comprises the following steps: collecting and preprocessing source disk information; planning a target environment and constructing a mapping table; transmitting a core structure and initializing a target partition; block-level data migration and log synchronization are carried out; according to the method, a partition table structure, a file system structure and an initial bitmap are constructed by scanning a source disk partition table and file system metadata to lock effective data, and a target new bitmap and a source-target data block mapping table are generated based on a target disk capacity planning mapping rule; according to the method, capacity adaptation and defragmentation are achieved, a core structure is transmitted, a target partition table and a file system are initialized, effective data blocks and log blocks are migrated according to a mapping table, finally, the migration effect is ensured through multi-layer verification, partition intelligent expansion and contraction, cross-disk multi-target mapping and fragment optimization are achieved, and migration flexibility and efficiency are improved.