85 results about "Reference counting" patented technology

The invention relates to the field of data processing, and discloses a cloud computing-based data processing method, which comprises the following steps of: constructing a reference count management micro-service cluster at a cloud end, receiving a reference count request carrying a count type through an API (Application Program Interface) gateway, and analyzing the reference count request into a structured operation instruction; reading a first state identifier of the referenced object and a second state identifier of the reference object from the distributed cache system according to the metadata snapshot; according to the method, through dynamic scheduling of a reference counting management micro-service cluster and a container arrangement engine, efficient allocation and scheduling of computing resources can be realized, maximization of resource utilization is ensured, resource waste is avoided, and high-speed query capability of a distributed cache system and a memory database is utilized; the state identification of the object can be retrieved at a millisecond level, the system response speed is improved, and the performance requirement under a high-concurrency scene is met.

The invention relates to a distributed storage access method initiated by a GPU (Graphics Processing Unit) in an AI (Artificial Intelligence) scene. The method comprises the following steps: S100: constructing a software cache module and an I / O (Input / Output) queue module in a GPU memory; s200, the GPU thread queries the target data through the software cache module, reads the target data if the target data is hit, and enters the next step if the target data is not hit; s300, when the I / O request is not hit, the repeated I / O requests initiated by multiple threads are automatically combined to generate a single effective I / O request, and the request is written into an I / O queue module; s400, the user mode distributed storage client monitors the written submission queue, reads the target data and then transmits the target data to the software cache module, and a completion queue of the I / O queue module is updated; and S500, the GPU thread polls the updated completion queue, and updates the reference count of the software cache module after reading the data. The main purpose of the invention is to realize high-throughput and low-delay data transmission without the participation of a CPU (Central Processing Unit).

The invention discloses a repeated data deletion method and system based on ZNS SSD, and belongs to the field of repeated data deletion.The method comprises the steps that a fingerprint index table and a fingerprint updating cache are maintained, three-time opportunity repeated data deletion is achieved, and repeated data search is conducted in the fingerprint index table, the fingerprint updating cache and an address mapping table respectively; an address mapping table, an address reverse mapping table and a reference counting table are maintained, physical space garbage collection / defragmentation and write request merging are achieved, the address mapping table, the address reverse mapping table and the reference counting table conduct garbage collection with partition as granularity when free space is insufficient, and a fingerprint index table is not updated after garbage collection data migration; and instead, the mapping of fingerprints to migration-in physical pages is stored in a fingerprint updating cache, the physical pages are sequentially distributed in partitions by the fingerprint updating cache, normal data write requests and garbage collection data migration write requests are respectively cached, and the write requests which are continuous in sequence are merged and then issued. According to the method, the high-performance and low-cost efficient storage system can be constructed on the basis of the ZNS SSD at relatively low overhead.

In some implementations, a computer system includes one or more processors and a non-transitory computer-readable medium storing programming for execution by the one or more processors. The programming includes instructions to receive a request to create a second air-gapped backup corresponding to a second transient backup. The second transient backup is subsequent to a first transient backup corresponding to a first air-gapped backup. The programming includes instructions to determine, in response to the request, whether the second transient backup matches the first transient backup, and to create, in response to determining that the second transient backup matches the first transient backup, a database entry for the second air-gapped backup that references the first air-gapped backup. The programming includes instructions to increment, in response to determining that the second transient backup matches the first transient backup, a reference counter for the first air-gapped backup.

The invention discloses a method and a system for realizing compatible file hard link and S3 shallow copy based on a distributed key-value pair, a file metadata key-value pair is used as a metadata storage main body of a file or a directory, a key consists of a file system ID (Identity), a parent directory ID and a file or directory name, and a value contains metadata information; a reverse index file handle key value pair is introduced to serve as a unique invariant identifier, a key is composed of a file system ID, an initial parent directory ID, an initial file or first N characters of a directory name and a file ID, and a value contains a current file metadata key and a reference count. When files or directories are created and deleted, the files or the directories are located at the same one-key value pair node in a large probability, and one-stage submission is supported to improve performance; hard linking and S3 shallow copying are achieved at the same time, independent file metadata key-value pairs and reverse index file handle key-value pairs are created for the hard linking and the S3 shallow copying, reverse index file handle keys of the target file are recorded by file metadata values of the file metadata key-value pairs, reference counts of the reverse index file handle values of the target file are updated, and performance and compatibility are both considered.

Caching a memory descriptor across input / output (I / O) requests. Based at least on receiving a request from a buffer consumer to register a buffer, a memory descriptor describing a layout of a set of physical memory pages assigned to the buffer is created. Creating the memory descriptor includes assigning and locking the set of physical memory pages to the memory descriptor. For each I / O request of a plurality of I / O requests on the buffer, at least a subset of the memory descriptor for use by the I / O request is returned, and a reference count is used to track the I / O request as pending on the memory descriptor. Later, the memory descriptor is destroyed, including unlocking the set of physical memory pages, based at least on the reference count indicating that there is no pending I / O request on the memory descriptor.

Techniques can include: recording, in a log, log entries for write operations writing data pages to logical addresses; and flushing the log entries from the log, wherein flushing includes performing deduplication of the data pages. Flushing can include: partitioning the data pages into i) a first portion of the data pages that are determined to be duplicates of existing data pages, and ii) a second portion of the data pages determined to be unique data pages; atomically incrementing reference counts each corresponding to one of the existing data pages determined as identical to a data page of the first portion; and atomically binding the logical addresses to the data pages, including: binding the first portion of data pages to a corresponding third portion of the logical addresses; and binding the second portion of data pages to a corresponding fourth portion of the logical addresses.

In some implementations, a memory apparatus may receive a command to write data to volatile memory of the memory system. The memory apparatus may compare the data to one or more data patterns to identify whether the data matches a data pattern of the one or more data patterns. The memory apparatus may increment a reference counter associated with the data pattern based on identifying that the data matches the data pattern. The memory apparatus may write, based on identifying that a measure flag associated with the data pattern is set, the data to the volatile memory.

This describes the unloading of shared resources. A shared library correlation table (SLCT) is generated in a simulated resource. During a close function, selecting an entry in the SLCT decrements the reference counter of the selected entry, checks the state of the selected entry based on the reference counter, and removes its associated container from memory if the state of the selected entry indicates that the container is not a shared resource, thus preventing segmentation errors.

The invention discloses a GPU dynamic memory resource management method and system based on time sequence historical analysis. According to the method, a CUDAAPI function during running of a deep learning model is intercepted, and an execution history table containing a kernel execution sequence, a memory block size and an address is generated. By comparing two latest execution history tables, calculating a consistency ratio to judge whether a memory access mode is stable or not, so that a memory prefetch strategy is started; and when the GPU memory is insufficient, the memory blocks are exchanged to the CPU memory based on an expelling strategy of reference counting, so that sufficient GPU memory can be available during kernel execution. The system comprises a memory extraction module, a prefetching module and a dynamic decision module, supports dynamic asynchronous memory exchange, and optimizes GPU memory utilization efficiency. The method is suitable for deep learning model training in a GPU memory resource limited environment, a larger-scale deep learning model and training batches can be supported on the GPU with the same configuration, memory waste is reduced, and operation performance is improved.

The embodiment of the invention provides a data transfer system and an image processing device. According to the data circulation system, the public cache module is arranged, so that the upstream module can call the cache blocks which are divided in advance from the public cache module to cache the target data. And the downstream module can search the cache block through the preset identifier issued by the upstream module to obtain the target data. By introducing a reference counting mechanism, write protection can be set for the called cache block, and the public cache module can release the cache block only when the reference times of the cache block return to zero from a non-zero number. Therefore, one-to-many transmission between the upstream module and the downstream module can be realized, the control logic of the target data acquisition process is reduced, and the data circulation efficiency is improved.

The present disclosure relates to a request-level consistency control method and system for dynamic configuration hot update. The method comprises: organizing configuration content into multiple versioned configuration snapshots, establishing a corresponding reference count for each versioned configuration snapshot, and setting a candidate version; enhancing a configuration reading path in a service node by plugging so as to be able to perform a configuration reading operation according to a bound version; in response to receiving a service request, creating an activation token for the service request, obtaining a current candidate version as the bound version, writing the bound version into the activation token, and increasing the reference count of the bound version; during processing of the service request, when configuration needs to be read, reading a configuration value from a corresponding versioned configuration snapshot according to the bound version in the activation token; and in response to the processing of the service request being completed, decreasing the reference count of the bound version by one.

A method, computer program product, and computing system for expanding the lifetime of virtual entries in a data storage system. A deduplication function of a virtual layer of a data storage system is monitored, a reference count of a virtual entry is incremented when a data page is written to a physical layer of the data storage system and the reference count of the virtual entry is decremented when a data page is deleted from the virtual layer. The virtual entry in the virtual layer is maintained when the reference count reaches a predetermined value and the virtual entry is reclaimed when a predetermined action of the data storage system is to be performed.

The invention discloses a value log memory mapping management method and system supporting concurrent index construction, and the method comprises the steps: carrying out the partitioning of a value log file, and obtaining a plurality of blocks with a fixed size; defining a block mapping array, wherein each element in the block mapping array records the mapping state of each block of the value log file; introducing a lock array corresponding to the block mapping array, wherein each lock in the lock array corresponds to the entry indexes of the plurality of blocks; when a task needs to read data of a certain offset of the value log file, calculating a value log block id according to the offset; searching whether a slot position corresponding to the block mapping array is empty or not according to the value log block id, and locking a corresponding position of the lock array only when the slot position is empty; after locking succeeds, file mapping operation is executed; when the block is not subjected to the offset, the data of the offset is not accessed within a period of time, that is, the state that the reference count of the block is 0 exceeds the preset duration, the timed cleaning mechanism can actively remove memory mapping at the corresponding position of the block mapping array, and related system resources are released.

The present application is applicable to the field of artificial intelligence technology, and provides an optimization method for large model systems, including: in the process of target model reasoning, reference counting is performed on the data involved in the calculation in the target model to obtain a reference count value; model data with a reference count value less than a first preset value is determined as low-frequency reference data; low-quality corpus is determined based on the low-frequency reference data; low-quality corpus is eliminated from the pre-training corpus to obtain training corpus, and the training corpus is used to train the optimized model. This method determines the actual application of the model data by reference counting the model data involved in the FFN calculation in the model, and then determines the low-quality corpus based on the application of the model data, and eliminates the low-quality corpus, thereby reducing the model parameters obtained by training, reducing the model data storage and calculation costs, and optimizing the model while ensuring the reasoning service quality of the model.

The invention discloses a transaction closed-loop processing method and system based on double-end collaborative hot update and containerization, and the method comprises the steps: carrying out the encryption storage of a script based on a management end, and pushing the encrypted script to a server; after receiving the script based on the server side, executing decryption and real-time compiling, generating an executable instance and storing the executable instance into a pre-compilation instance pool; the instance is automatically released by the JVM when the reference count returns to zero; the server side is deployed in a Kubernetes cluster in a containerization manner, and the capacity is dynamically expanded and shrunk; the method comprises the following steps of: performing closed-loop processing on a service request by the same server instance: completing multi-step operation through a local database transaction; realizing cross-method state transfer by using JVM memory sharing; the call link depth is constantly a preset value and is directly referenced and executed through a memory. According to the method, the management end pushes the script to the server end, the server end compiles in real time, and Kubernetes containerization capacity expansion and contraction are combined, so that service request closed-loop processing is realized, distributed transaction coordination is eliminated through local transaction and memory sharing, and transaction time consumption is reduced.

The embodiment of the invention provides a dynamic shared memory control method and device, a medium and a program product, and relates to the technical field of databases. The method comprises the following steps: setting an initial reference count and an initial fixed mark of a dynamic shared memory block; updating the reference count and / or the fixed mark according to the association operation or the disassociation operation of the process on the dynamic shared memory block; under the condition that the reference count is a first preset value and the fixed mark is in a first state, the operation of releasing the dynamic shared memory block is triggered, the first preset value is a preset reference count threshold value used for triggering the memory release operation, and the first state is a preset fixed mark state used for triggering the memory release operation. Based on the method, high flexibility and safety can be taken into account, so that the dynamic shared memory block is flexibly and safely released, and the requirements in an actual application scene are met.

A system can maintain a copy-on-write b+ tree for a data storage system that is configured to store data as chunks, and a mapping between tree page identifiers of the b+ tree and first in-use chunk identifiers. The system can perform reference count validation, comprising iterating over the mapping, wherein the iterating over the mapping comprises, based on a first tree page identifier of the tree page identifiers being present in the mapping, loading second in-use chunk identifiers that correspond to the first tree page identifier directly from the mapping; and based on the first tree page identifier of the tree page identifiers being absent, loading corresponding content, extracting third in-use chunk identifiers from object metadata of the content, and updating the mapping based on the third in-use chunk identifiers, to produce an updated mapping. The system can perform garbage collection of the data storage system based on the validation.

A method in a data storage system includes foreground processes that produce incorrect-high reference counts greater than corresponding numbers of actual references to associated data instances (e.g., virtual large blocks), representing loss of storage capacity. A background process corrects a reference count for a data instance by (1) copying the data instance to a new data instance with an initial reference count of zero and making the new data instance unavailable for reclaiming, (2) scanning a set of referencing structures for all references to the data instance, and for each reference (i) replacing the reference with a new reference to the new data instance, and (ii) incrementing the reference count of the new data instance, and (3) upon completing the scanning, making the new data instance available for eventual reclaiming based on its reference count reaching zero. The process restores reference count accuracy and reduces loss / leakage of storage capacity.

In at least one embodiment, destaging a write of a user data (UD) log may include: performing deduplication that determines content written by the write to a logical address LA is a duplicate of existing content; and establishing mapping information of a chain of metadata (MD) pages mapping LA to a physical address PA storing the existing content, wherein the chain includes a MD leaf page and a VLB (virtual layer block) page. An update of a MD log can update an indirect pointer (IDP) field of a MD leaf entry of the MD leaf page to reference a VLB entry of the VLB page where the VLB entry further includes PA. A second update of the MD log can increment a reference count of the VLB entry. The second update can be recorded blindly in the MD log without reading or accessing reference count during destaging the write.

A deduplication service can be provided to a storage domain from a services framework that expands and contracts to both meet service demand and to conform to resource management of a compute domain. The deduplication service maintains a fingerprint database and reference count data in compute domain resources, but persists these into the storage domain for use in the case of a failure or interruption of the deduplication service in the compute domain. The deduplication service responds to service requests from the storage domain with indications of paths in a user namespace and whether or not a piece of data had a fingerprint match in the fingerprint database. The indication of a match guides the storage domain to either store the piece of data into the storage backend or to reference another piece of data. The deduplication service uses the fingerprints to define paths for corresponding pieces of data.

The invention discloses an aggregation file dynamic recovery method and device, electronic equipment and a storage medium, and relates to a distributed storage system.According to the technical scheme, a reference counter is set for each aggregation file, the numerical value of the counter is obtained in real time, and the change frequency of the numerical value is calculated; therefore, the problems of high I / O pressure and response delay caused by disk persistence when a counter is frequently updated and resource waste caused by memory residence of a cold data counter can be solved, and the technical effects of reducing performance overhead and optimizing the resource utilization rate are achieved. When the target file is migrated from the source aggregation file to the target aggregation file, the metadata direction of the target file is updated through the atomization operation, and the source aggregation file counter is adjusted to decrease progressively and the target aggregation file counter is adjusted to increase progressively. The problems that the verification overhead is large and the consistency is difficult to guarantee due to the fact that a verification mechanism is complex during small file migration can be solved.

Unloading shared resources is described. A shared library correlation table (SLCT) is generated in a mock resource. During a close function, selecting an entry in the SLCT is selected for reducing a reference count of the selected entry in the SLCT, a status of the selected entry based on the reference count is verified, and an associated container of the selected entry is removed from memory when the status of the selected entry indicates the associated container is not a shared resource, thereby avoiding segmentation faults.

Techniques for concurrent lazy reference tracking in an old garbage collection generation are disclosed, including: encountering, by a mutator thread during a garbage collection epoch, a first instruction to write a first value to a field; responsive to encountering the first instruction to write the first value to the field: entering a slow-path write barrier; performing, by the slow-path write barrier, a first one or more reference counting operations with respect to the field; encountering, by the mutator thread during the first garbage collection epoch and subsequent to encountering the first instruction to write the first value to the field, a second instruction to write a second value to the field; responsive to encountering the second instruction to write the second value to the field: entering a fast-path write barrier; wherein the fast-path write barrier does not perform any reference counting operations with respect to the field.

Embodiments of the present application disclose a reference counting method and a related apparatus. The method comprises: acquiring an idle target data block from a storage space; establishing a plurality of counters in the target data block, the number of the plurality of counters corresponding to the grouping situation of a plurality of cores of a central processing unit (CPU); in response to a Pin / Unpin operation performed on the target data block, determining a target core corresponding to the Pin / Unpin operation, the target core belonging to the plurality of cores; on the basis of the target core, determining a corresponding target counter, the target counter belonging to the plurality of counters; and performing, by means of the target counter, reference counting corresponding to the Pin / Unpin operation. The method can be used for increasing the speed of Pin / Unpin on a multi-core CPU, thereby improving the overall performance of a system.

Offloading shared resources is described. A shared library correlation table (SLCT) is generated in a simulated resource. During the shutdown of the function, an entry in the SLCT is selected to reduce a reference count for the selected entry in the SLCT, a state of the selected entry is verified based on the reference count, and when the state of the selected entry indicates that an associated container of the selected entry is not a shared resource, the associated container is removed from the memory, thereby avoiding a segmentation error.

This application discloses a reference counting method and related apparatus. The method includes: acquiring a free target data block in storage space; establishing multiple counters in the target data block, the number of counters corresponding to the grouping of multiple cores of a central processing unit (CPU); determining the target core corresponding to the pin / unpin operation in response to a pin / unpin operation on the target data block, wherein the target core belongs to multiple cores; determining the corresponding target counter based on the target core, wherein the target counter belongs to multiple counters; and performing reference counting corresponding to the pin / unpin operation using the target counter. This method can be used to improve the speed of pin / unpin on multi-core CPUs, thereby improving the overall system performance.

The invention relates to the field of data storage, medical health and financial science and technology, and discloses a junk data deletion method and device, electronic equipment and a storage medium. The method comprises the following steps: in response to a junk data deletion instruction sent by a user, screening all data blocks based on final reference time and reference degree data of the data blocks to obtain junk data blocks; wherein the reference degree data comprises at least one of the following data: a reference frequency and a reference trend; and deleting each junk data block. According to the method, the final reference time and the reference degree data of the data blocks are introduced to serve as dual judgment bases, junk data and effective data are accurately distinguished through cross validation of the time dimension and the activeness, the problem that a traditional reference counting method excessively depends on a single count value is solved to a certain extent, and the accuracy of the data block is improved. And the accuracy and reliability of junk data identification in a complex data environment are improved.

The present disclosure provides a distributed data storage system including a fast partition splitting method. The distributed data storage system includes a parent partition, multiple intermediate partitions and multiple child partitions. The parent partition is divided into multiple intermediate partitions, including a first intermediate partition. When data replication occurs from the parent partition to the first intermediate partition, the first intermediate partition is divided into multiple child partitions. Each intermediate partition includes a reference file pointing to a data file in the parent partition. Each child partition includes a reference file pointing to a reference file of the first intermediate partition, and another reference file pointing to a data file in the first intermediate partition. The distributed data storage system of the present disclosure also includes a split reference counter to track when a partition can be deleted from the system. In addition, the present disclosure provides a method for fast partition splitting, and a non-temporary computer-readable medium storing an instruction set, which can be executed by one or more processors to perform the method for fast partition splitting. The present disclosure can effectively reduce the access adjustments of traditional systems caused by partition splitting.

This application provides a method and apparatus for memory garbage collection. The method includes: determining whether a marked node is located in a garbage circular reference (RC ring) based on the marked node of the application and the reference count value of the marked node, wherein nodes in the garbage RC ring have no external references; and reclaiming the nodes in the garbage RC ring when the marked node is located in the garbage RC ring. This application can realize the reclamation of the garbage RC ring.