53 results about "Reference counting" patented technology

The invention relates to the field of data processing, and discloses a cloud computing-based data processing method, which comprises the following steps of: constructing a reference count management micro-service cluster at a cloud end, receiving a reference count request carrying a count type through an API (Application Program Interface) gateway, and analyzing the reference count request into a structured operation instruction; reading a first state identifier of the referenced object and a second state identifier of the reference object from the distributed cache system according to the metadata snapshot; according to the method, through dynamic scheduling of a reference counting management micro-service cluster and a container arrangement engine, efficient allocation and scheduling of computing resources can be realized, maximization of resource utilization is ensured, resource waste is avoided, and high-speed query capability of a distributed cache system and a memory database is utilized; the state identification of the object can be retrieved at a millisecond level, the system response speed is improved, and the performance requirement under a high-concurrency scene is met.

The invention relates to a distributed storage access method initiated by a GPU (Graphics Processing Unit) in an AI (Artificial Intelligence) scene. The method comprises the following steps: S100: constructing a software cache module and an I / O (Input / Output) queue module in a GPU memory; s200, the GPU thread queries the target data through the software cache module, reads the target data if the target data is hit, and enters the next step if the target data is not hit; s300, when the I / O request is not hit, the repeated I / O requests initiated by multiple threads are automatically combined to generate a single effective I / O request, and the request is written into an I / O queue module; s400, the user mode distributed storage client monitors the written submission queue, reads the target data and then transmits the target data to the software cache module, and a completion queue of the I / O queue module is updated; and S500, the GPU thread polls the updated completion queue, and updates the reference count of the software cache module after reading the data. The main purpose of the invention is to realize high-throughput and low-delay data transmission without the participation of a CPU (Central Processing Unit).

In some implementations, a computer system includes one or more processors and a non-transitory computer-readable medium storing programming for execution by the one or more processors. The programming includes instructions to receive a request to create a second air-gapped backup corresponding to a second transient backup. The second transient backup is subsequent to a first transient backup corresponding to a first air-gapped backup. The programming includes instructions to determine, in response to the request, whether the second transient backup matches the first transient backup, and to create, in response to determining that the second transient backup matches the first transient backup, a database entry for the second air-gapped backup that references the first air-gapped backup. The programming includes instructions to increment, in response to determining that the second transient backup matches the first transient backup, a reference counter for the first air-gapped backup.

The invention discloses a method and a system for realizing compatible file hard link and S3 shallow copy based on a distributed key-value pair, a file metadata key-value pair is used as a metadata storage main body of a file or a directory, a key consists of a file system ID (Identity), a parent directory ID and a file or directory name, and a value contains metadata information; a reverse index file handle key value pair is introduced to serve as a unique invariant identifier, a key is composed of a file system ID, an initial parent directory ID, an initial file or first N characters of a directory name and a file ID, and a value contains a current file metadata key and a reference count. When files or directories are created and deleted, the files or the directories are located at the same one-key value pair node in a large probability, and one-stage submission is supported to improve performance; hard linking and S3 shallow copying are achieved at the same time, independent file metadata key-value pairs and reverse index file handle key-value pairs are created for the hard linking and the S3 shallow copying, reverse index file handle keys of the target file are recorded by file metadata values of the file metadata key-value pairs, reference counts of the reverse index file handle values of the target file are updated, and performance and compatibility are both considered.

Techniques can include: recording, in a log, log entries for write operations writing data pages to logical addresses; and flushing the log entries from the log, wherein flushing includes performing deduplication of the data pages. Flushing can include: partitioning the data pages into i) a first portion of the data pages that are determined to be duplicates of existing data pages, and ii) a second portion of the data pages determined to be unique data pages; atomically incrementing reference counts each corresponding to one of the existing data pages determined as identical to a data page of the first portion; and atomically binding the logical addresses to the data pages, including: binding the first portion of data pages to a corresponding third portion of the logical addresses; and binding the second portion of data pages to a corresponding fourth portion of the logical addresses.

In some implementations, a memory apparatus may receive a command to write data to volatile memory of the memory system. The memory apparatus may compare the data to one or more data patterns to identify whether the data matches a data pattern of the one or more data patterns. The memory apparatus may increment a reference counter associated with the data pattern based on identifying that the data matches the data pattern. The memory apparatus may write, based on identifying that a measure flag associated with the data pattern is set, the data to the volatile memory.

This describes the unloading of shared resources. A shared library correlation table (SLCT) is generated in a simulated resource. During a close function, selecting an entry in the SLCT decrements the reference counter of the selected entry, checks the state of the selected entry based on the reference counter, and removes its associated container from memory if the state of the selected entry indicates that the container is not a shared resource, thus preventing segmentation errors.

The invention discloses a GPU dynamic memory resource management method and system based on time sequence historical analysis. According to the method, a CUDAAPI function during running of a deep learning model is intercepted, and an execution history table containing a kernel execution sequence, a memory block size and an address is generated. By comparing two latest execution history tables, calculating a consistency ratio to judge whether a memory access mode is stable or not, so that a memory prefetch strategy is started; and when the GPU memory is insufficient, the memory blocks are exchanged to the CPU memory based on an expelling strategy of reference counting, so that sufficient GPU memory can be available during kernel execution. The system comprises a memory extraction module, a prefetching module and a dynamic decision module, supports dynamic asynchronous memory exchange, and optimizes GPU memory utilization efficiency. The method is suitable for deep learning model training in a GPU memory resource limited environment, a larger-scale deep learning model and training batches can be supported on the GPU with the same configuration, memory waste is reduced, and operation performance is improved.

The present disclosure relates to a request-level consistency control method and system for dynamic configuration hot update. The method comprises: organizing configuration content into multiple versioned configuration snapshots, establishing a corresponding reference count for each versioned configuration snapshot, and setting a candidate version; enhancing a configuration reading path in a service node by plugging so as to be able to perform a configuration reading operation according to a bound version; in response to receiving a service request, creating an activation token for the service request, obtaining a current candidate version as the bound version, writing the bound version into the activation token, and increasing the reference count of the bound version; during processing of the service request, when configuration needs to be read, reading a configuration value from a corresponding versioned configuration snapshot according to the bound version in the activation token; and in response to the processing of the service request being completed, decreasing the reference count of the bound version by one.

The invention discloses a value log memory mapping management method and system supporting concurrent index construction, and the method comprises the steps: carrying out the partitioning of a value log file, and obtaining a plurality of blocks with a fixed size; defining a block mapping array, wherein each element in the block mapping array records the mapping state of each block of the value log file; introducing a lock array corresponding to the block mapping array, wherein each lock in the lock array corresponds to the entry indexes of the plurality of blocks; when a task needs to read data of a certain offset of the value log file, calculating a value log block id according to the offset; searching whether a slot position corresponding to the block mapping array is empty or not according to the value log block id, and locking a corresponding position of the lock array only when the slot position is empty; after locking succeeds, file mapping operation is executed; when the block is not subjected to the offset, the data of the offset is not accessed within a period of time, that is, the state that the reference count of the block is 0 exceeds the preset duration, the timed cleaning mechanism can actively remove memory mapping at the corresponding position of the block mapping array, and related system resources are released.

The invention discloses a transaction closed-loop processing method and system based on double-end collaborative hot update and containerization, and the method comprises the steps: carrying out the encryption storage of a script based on a management end, and pushing the encrypted script to a server; after receiving the script based on the server side, executing decryption and real-time compiling, generating an executable instance and storing the executable instance into a pre-compilation instance pool; the instance is automatically released by the JVM when the reference count returns to zero; the server side is deployed in a Kubernetes cluster in a containerization manner, and the capacity is dynamically expanded and shrunk; the method comprises the following steps of: performing closed-loop processing on a service request by the same server instance: completing multi-step operation through a local database transaction; realizing cross-method state transfer by using JVM memory sharing; the call link depth is constantly a preset value and is directly referenced and executed through a memory. According to the method, the management end pushes the script to the server end, the server end compiles in real time, and Kubernetes containerization capacity expansion and contraction are combined, so that service request closed-loop processing is realized, distributed transaction coordination is eliminated through local transaction and memory sharing, and transaction time consumption is reduced.

The embodiment of the invention provides a dynamic shared memory control method and device, a medium and a program product, and relates to the technical field of databases. The method comprises the following steps: setting an initial reference count and an initial fixed mark of a dynamic shared memory block; updating the reference count and / or the fixed mark according to the association operation or the disassociation operation of the process on the dynamic shared memory block; under the condition that the reference count is a first preset value and the fixed mark is in a first state, the operation of releasing the dynamic shared memory block is triggered, the first preset value is a preset reference count threshold value used for triggering the memory release operation, and the first state is a preset fixed mark state used for triggering the memory release operation. Based on the method, high flexibility and safety can be taken into account, so that the dynamic shared memory block is flexibly and safely released, and the requirements in an actual application scene are met.

A system can maintain a copy-on-write b+ tree for a data storage system that is configured to store data as chunks, and a mapping between tree page identifiers of the b+ tree and first in-use chunk identifiers. The system can perform reference count validation, comprising iterating over the mapping, wherein the iterating over the mapping comprises, based on a first tree page identifier of the tree page identifiers being present in the mapping, loading second in-use chunk identifiers that correspond to the first tree page identifier directly from the mapping; and based on the first tree page identifier of the tree page identifiers being absent, loading corresponding content, extracting third in-use chunk identifiers from object metadata of the content, and updating the mapping based on the third in-use chunk identifiers, to produce an updated mapping. The system can perform garbage collection of the data storage system based on the validation.

The invention discloses an aggregation file dynamic recovery method and device, electronic equipment and a storage medium, and relates to a distributed storage system.According to the technical scheme, a reference counter is set for each aggregation file, the numerical value of the counter is obtained in real time, and the change frequency of the numerical value is calculated; therefore, the problems of high I / O pressure and response delay caused by disk persistence when a counter is frequently updated and resource waste caused by memory residence of a cold data counter can be solved, and the technical effects of reducing performance overhead and optimizing the resource utilization rate are achieved. When the target file is migrated from the source aggregation file to the target aggregation file, the metadata direction of the target file is updated through the atomization operation, and the source aggregation file counter is adjusted to decrease progressively and the target aggregation file counter is adjusted to increase progressively. The problems that the verification overhead is large and the consistency is difficult to guarantee due to the fact that a verification mechanism is complex during small file migration can be solved.

Unloading shared resources is described. A shared library correlation table (SLCT) is generated in a mock resource. During a close function, selecting an entry in the SLCT is selected for reducing a reference count of the selected entry in the SLCT, a status of the selected entry based on the reference count is verified, and an associated container of the selected entry is removed from memory when the status of the selected entry indicates the associated container is not a shared resource, thereby avoiding segmentation faults.

Techniques for concurrent lazy reference tracking in an old garbage collection generation are disclosed, including: encountering, by a mutator thread during a garbage collection epoch, a first instruction to write a first value to a field; responsive to encountering the first instruction to write the first value to the field: entering a slow-path write barrier; performing, by the slow-path write barrier, a first one or more reference counting operations with respect to the field; encountering, by the mutator thread during the first garbage collection epoch and subsequent to encountering the first instruction to write the first value to the field, a second instruction to write a second value to the field; responsive to encountering the second instruction to write the second value to the field: entering a fast-path write barrier; wherein the fast-path write barrier does not perform any reference counting operations with respect to the field.

Embodiments of the present application disclose a reference counting method and a related apparatus. The method comprises: acquiring an idle target data block from a storage space; establishing a plurality of counters in the target data block, the number of the plurality of counters corresponding to the grouping situation of a plurality of cores of a central processing unit (CPU); in response to a Pin / Unpin operation performed on the target data block, determining a target core corresponding to the Pin / Unpin operation, the target core belonging to the plurality of cores; on the basis of the target core, determining a corresponding target counter, the target counter belonging to the plurality of counters; and performing, by means of the target counter, reference counting corresponding to the Pin / Unpin operation. The method can be used for increasing the speed of Pin / Unpin on a multi-core CPU, thereby improving the overall performance of a system.

This application discloses a reference counting method and related apparatus. The method includes: acquiring a free target data block in storage space; establishing multiple counters in the target data block, the number of counters corresponding to the grouping of multiple cores of a central processing unit (CPU); determining the target core corresponding to the pin / unpin operation in response to a pin / unpin operation on the target data block, wherein the target core belongs to multiple cores; determining the corresponding target counter based on the target core, wherein the target counter belongs to multiple counters; and performing reference counting corresponding to the pin / unpin operation using the target counter. This method can be used to improve the speed of pin / unpin on multi-core CPUs, thereby improving the overall system performance.

This application provides a method and apparatus for memory garbage collection. The method includes: determining whether a marked node is located in a garbage circular reference (RC ring) based on the marked node of the application and the reference count value of the marked node, wherein nodes in the garbage RC ring have no external references; and reclaiming the nodes in the garbage RC ring when the marked node is located in the garbage RC ring. This application can realize the reclamation of the garbage RC ring.

The present application belongs to the technical field of machine learning, and particularly relates to an inference service system and method supporting multi-model hybrid deployment. A plurality of inference instances and a scheduler are communicatively connected, and each inference instance is configured with a local storage space for caching model weight files. The scheduler analyzes a model set required by an inference request, and the local cache state and load condition of each inference instance, intelligently assigns the request to an optimal target inference instance; the target inference instance uses a dynamic downloading mechanism for a model not stored locally, and directly obtains a local model path through the cache for a locally stored model; and fine cache life cycle management is realized through reference counting and LRU strategy, so as to combine cache-aware scheduling and dynamic model loading, solve the problem that a traditional single-instance single-model deployment mode cannot well support multi-model hybrid deployment, and realize efficient use of resources and rapid response of requests.

An apparatus includes a processing means configured to maintain a deduplicated data structure including sub-sections of data block identifiers associated with different compression ratio ranges and having different numbers of data block identifiers. The processing means is further configured to identify a given data block identifier and a given compression ratio for a given data block to be stored, and to determine whether the given data block identifier is in a given sub-section having a given compression ratio range including the given compression ratio. The processing means is further configured to write the given data block to a physical space block of a storage system in response to determining that the given data block identifier is not in the given sub-section, and to increment a deduplication reference count of the given data block identifier in response to determining that the given data block identifier is in the given sub-section.

The present disclosure relates to an object generation method and device, a storage medium and an electronic device, and particularly relates to the technical field of computers, which can comprise: obtaining a target class, wherein the target class comprises common attributes corresponding to the target class and a target algorithm for creating a reference counter; generating a target object according to the common attributes in the target class and generating a reference counter of the target object according to the target algorithm in the target class. The present disclosure has the beneficial effect that the target class comprises the common attributes for creating an object and the target algorithm for creating a reference counter, and when the target object and the reference counter are created based on the target class, the memory spaces of the target object and the reference counter can be allocated at the same time, so that the target object can be generated based on the common attributes of the target class and the reference counter of the target object can be generated according to the target algorithm through one-time memory allocation, thereby providing the generation efficiency of an instance object.

In some implementations, a memory apparatus may receive a command to write data to volatile memory of the memory system. The memory apparatus may compare the data to one or more data patterns to identify whether the data matches a data pattern of the one or more data patterns. The memory apparatus may increment a reference counter associated with the data pattern based on identifying that the data matches the data pattern. The memory apparatus may write, based on identifying that a measure flag associated with the data pattern is set, the data to the volatile memory.

The application discloses a method for volume priority reconstruction, comprising: creating at least two priority containers and associating storage volumes to direct data storage; scanning gc objects and associating storage objects by GC to record garbage amount and garbage bitmap of the storage objects for triggering garbage collection of the storage objects and identifying valid data; calculating target disks of the storage objects and their states by a layout algorithm and marking objects with target disk failure as to-be-reconstructed objects; periodically iterating the gc objects by a GC producer to add the to-be-reconstructed objects to a failure hash bucket and increase reference counts; extracting the to-be-reconstructed objects by a GC consumer to move valid data of the to-be-reconstructed objects to new storage objects with healthy target disks through an aggregation module; reducing counts and removing the objects after the moving is completed; and reporting reconstruction progress based on hash bucket count states of each node. The application realizes specified data priority reconstruction and guarantees important business data priority recovery safety through priority container association and GC directed moving.

The invention discloses an industrial unstructured data processing method and system, and relates to the technical field of data storage. The method comprises the following steps: firstly, calculating an overall fingerprint of a to-be-stored industrial unstructured data file so as to carry out global de-duplication; if the file is not hit, slicing the file and calculating the fingerprint of each fragment so as to perform deduplication comparison on finer granularity; and finally, through two-stage Hash fingerprint comparison and reference counting management, only writing the unique fragment into a storage system according to a protection strategy configured by a file, and binding and recording the physical position of the unique fragment and the fragment fingerprint in a metadata warehouse. Therefore, the core problems of metadata performance bottleneck, data consistency risk under network faults, low fixed block duplicate removal efficiency and the like of traditional distributed storage in an industrial scene can be fundamentally solved, industrial data management with high throughput, strong consistency and optimized storage efficiency is realized, and practical application and popularization are facilitated.

The invention belongs to the technical field of machine learning, and particularly relates to an inference service system and method supporting multi-model hybrid deployment. A plurality of reasoning instances are in communication connection with a scheduler, each reasoning instance is configured with a local storage space for caching a model weight file, a model set required by a reasoning request and the local cache state and load condition of each reasoning instance are analyzed through the scheduler, and the request is intelligently allocated to an optimal target reasoning instance; the target reasoning instance adopts a dynamic downloading mechanism for a model which is not locally stored, and a local model path is directly acquired for the model which is locally stored through a cache; refined cache life cycle management is achieved through reference counting and an LRU strategy, so that cache sensing scheduling and dynamic model loading are combined, the problem that a traditional single-instance single-model deployment mode cannot well support multi-model mixed deployment is solved, and efficient utilization of resources and quick response of requests are achieved.

Unloading a shared resource is described. A shared library correlation table (SLCT) is generated with mock resources. During a close function, an entry in the SLCT is selected to decrease the reference count of the selected entry in the SLCT, and the status of the selected entry based on the reference count is verified. If the status of the selected entry indicates that the associated container is not a shared resource, the associated container of the selected entry is deleted from memory, thereby avoiding a segmentation fault.

The application discloses a backup deduplication system, comprising a fingerprint deduplication subsystem, wherein the fingerprint deduplication subsystem comprises a deduplication accelerator module and a fingerprint library; the deduplication accelerator module is used for linearly adding fingerprints and related information of non-repeated backup data in time sequence to form fingerprint record information of the non-repeated backup data, wherein time sequence linear storage of the fingerprint record information facilitates disk sequential access during backup deduplication operation; the fingerprint record information comprises reference count of the non-repeated backup data; the reference count of the non-repeated backup data is only stored in the deduplication accelerator module, the reference count is not stored in the fingerprint library, and the reference count is not recorded in data index relied on by fingerprint deduplication based on the fingerprint library; the fingerprint library is used for storing the fingerprints and related information of the non-repeated backup data through a relied-on key-value pair engine to form fingerprint storage information of the non-repeated backup data. The backup deduplication system can reduce memory occupation and improve the query rate of data fingerprints.

A system can maintain a copy-on-write b+ tree for a data storage system that is configured to store data as chunks, and a mapping between tree page identifiers of the b+ tree and first in-use chunk identifiers. The system can perform reference count validation, comprising iterating over the mapping, wherein the iterating over the mapping comprises, based on a first tree page identifier of the tree page identifiers being present in the mapping, loading second in-use chunk identifiers that correspond to the first tree page identifier directly from the mapping; and based on the first tree page identifier of the tree page identifiers being absent, loading corresponding content, extracting third in-use chunk identifiers from object metadata of the content, and updating the mapping based on the third in-use chunk identifiers, to produce an updated mapping. The system can perform garbage collection of the data storage system based on the validation.

The application discloses a kind of application data reference management and garbage collection.The application data reference management method includes: when detecting that target application data is referenced by target variable, and target variable belongs to target variable type, the logical address of the target application data is written into the physical address of the target variable;The logical address corresponding to the physical address of target variable is written into the reference address table as a reference address.The technical scheme of the embodiment of the application uses the reference address table to manage the application data reference not recorded in the reference array, avoids mistaking such application data as garbage data and being deleted, thereby avoiding the empty exception caused by referencing non-existent application data, the reference address table records reference address instead of the logical address of application data, avoids the need to modify application data reference counter or the content in reference address table when reference relationship is cleared, and ensures application execution efficiency.