84 results about "Key-value database" patented technology

The invention relates to the technical field of data storage, and discloses a CSD-based key value database storage method and system, and the method comprises the steps: responding to a user inserting a key value pair into a key value database, storing the key value pair into a memory table, and converting the memory table into an immutable memory table after a preset triggering condition is satisfied; aligning and coding the key value pairs in the immutable memory table according to a uniform length to generate key value pairs with fixed step lengths; calling a computing storage device to automatically compress the key value pair with the fixed step length and flash the key value pair into an SSTable file; receiving a merging operation triggering instruction, and controlling the computing storage device to obtain the SSTable file and decompress the SSTable file; and calling the graphic processing unit to merge the decompressed files to generate a new SSTable file, and calling the computing storage device to automatically compress the new file and flash the new file to a disk. According to the method, the read-write throughput and merging efficiency of the key value database are remarkably improved, and the problems of write blockage and write pause are effectively relieved.

The embodiment of the invention relates to a data processing method and device based on a key value database, equipment, a storage medium and a key value database system.The method comprises the steps that after it is determined that to-be-cached key value data exists in a video memory of a graphic processing unit, a data storage request for the to-be-cached key value data is sent to a key value database server side, the data storage request carries storage address information and storage length information of value data of to-be-cached key value data in a video memory; receiving a data reading request which is sent by a key value database server and carries storage address information, and controlling the first network card supporting the remote direct memory access protocol to read value data of the key value data to be cached from the video memory according to the storage address information, sending the value data to the key value database server side, and writing the value data through the second network card supporting the remote direct memory access protocol. Visibly, according to the embodiment of the invention, the key value data in the GPU video memory can be stored without a memory copying process, and the processing efficiency of the key value data is improved.

The invention discloses a data processing method, which comprises the following steps of: performing parallel data processing on key value pair data in a key value database according to at least one data processing instruction to obtain data processing results respectively corresponding to the at least one data processing instruction; according to the method, at least one data processing instruction is received, and the data processing result corresponding to the at least one data processing instruction is fed back to the client, so that the client can be efficiently connected to the key value database by using the preset data processing instruction, and parallel data processing can be performed on the key value data in the key value database; according to the method and the device, efficient parallel data processing is realized, so that the performance of the application program is improved, the performance of the client is improved, the operation delay is reduced, the data processing efficiency can be improved, the data processing cost is reduced, and the user experience of data processing is improved.

The invention provides a graph data processing method, device and equipment and a computer readable storage medium. The method comprises the following steps: acquiring to-be-processed graph data, and acquiring a plurality of attribute indexes corresponding to the to-be-processed graph data; obtaining index data of each attribute index, and obtaining a vertex identifier and an attribute value corresponding to each attribute index; encoding the index data of each attribute index and the vertex identifier and the attribute value corresponding to each attribute index to obtain key value pair information corresponding to each attribute index; and storing the key value pair information corresponding to each attribute index into a key value database. Through the method and the device, high attribute query efficiency can be realized.

The present application relates to the technical field of storage, and discloses a data attribute-aware storage system and data management method for a key-value database. The method comprises two parts, i.e., data hotness attribute-based writing and data density attribute-based merging. The data hotness attribute-based writing comprises classifying key-value pairs into cold key-value pairs and hot key-value pairs, to realize separate storage of cold data and hot data by means of a data hotness attribute-based writing strategy. The data density attribute-based merging comprises: when key-value pairs in an SSTable are extracted during a merging operation and form a sorted key-value pair sequence, extracting the key-value pairs one by one, and calculating a binary difference between every two adjacent key-value pairs; and if the difference is greater than a set threshold, stopping filling the current SSTable, and creating a new SSTable. In the present invention, by analyzing data hotness and data density, data having different attributes are classified to different data partitions, thereby reducing I / O amplification caused by repeated reading and writing in a compression process.

The invention discloses a metadata persistence method, an electronic device, a storage medium and a program product, and relates to the technical field of computers, when a request is written, a newly added forward relation of a small amount of data is generated from written data, the newly added forward relation is added to the tail of an existing forward relation of a distributed key value database, and the newly added forward relation is added to the tail of the existing forward relation of the distributed key value database. The method does not need to acquire the existing forward relation and the newly added forward relation, update the forward relation and then write the updated forward relation into the distributed key value database again, reduces the number of times of writing into the distributed key value database, reduces the time delay of updating the forward relation, does not need to read out the existing forward relation and deserialize the existing forward relation, reduces the time delay of updating the forward relation, and reduces the time delay of updating the forward relation. And then serialized writing is performed, so that the processor consumption is reduced. According to the method, the forward relation is spliced to form the spliced forward relation when the request is read, the data is read from the data pool according to the spliced forward relation, the read request is associated with the written data, data reading is not affected, and the consumption of a processor can be comprehensively reduced.

The invention relates to the technical field of key value databases, and discloses a key value database near-storage computing system based on computing storage equipment, the system comprises a host, a memory and a computing memory, a decoder in the computing memory is used for reading files needing to be merged from the memory, analyzing all key value pairs from the files, and storing the key value pairs in the computing memory; placing the key value pairs in a buffer area connected with a sequencer; the sequencer is used for comparing the size relationship of the key value pairs, taking out the key value pair from the corresponding decoder buffer area according to the comparison result of each time, selecting the key value pair with the minimum key value according to the size of the key in the key value pair, and transmitting the corresponding key value pair to the buffer area connected with the encoder; and the encoder is used for extracting the key value pair from the buffer area connected with the sequencer, generating an SS-Table file according to a file encoding rule and storing the SS-Table file into the memory. Deep fusion of calculation and storage can be realized, calculation resources in a host are fully utilized, and the overall system performance is improved.

The invention provides a data migration method and device, electronic equipment, a computer readable medium and a computer program product. The method comprises the steps that in response to a data migration instruction, a first data processing strategy is executed to process a data write-in request of a client side, and the first data processing strategy executes corresponding data write-in operation in a source database and records the data write-in request with a target key name in a key value database; synchronizing target data in the source data to a key value database through asynchronous data processing, wherein the target data corresponds to the data segment selected by the user; in response to an instruction of completing data migration, switching the executed first data processing strategy to a second data processing strategy, the second data processing strategy enabling data to be operated with an original key name in a key value database; and cleaning the target key names in the key value database by scanning the target key names in the key value database.

The present disclosure relates to a data processing method and device, electronic equipment and storage medium. The method comprises: receiving a data processing request sent by a client, the data processing request being used to indicate the Nth query and carrying a target cursor parameter; the target cursor parameter is an encoding parameter generated in the (N-1)th query, N being a positive integer greater than 1; the target key corresponding to the target cursor parameter is obtained from the mapping relationship between the cursor parameter and the key, the target key indicating the end position of the (N-1)th query in the distributed key-value database; and the (N)th query is performed on the distributed key-value database according to the target key. According to the technical scheme of the present disclosure, the cursor parameter is configured according to the demand of the original command, the corresponding relationship between the cursor parameter and the key is stored, so that the starting position of the query can be accurately analyzed based on the target cursor parameter, and the distributed key-value database has good compatibility.

The present disclosure provides a file processing method and device for a key-value database, equipment, a storage medium and a program product, relates to the technical field of computers, and can be applied to the technical field of finance. The method comprises the following steps: in response to receiving a merging operation instruction, determining a first target layer of a storage component of the key-value database; determining a candidate first sorting string table file stored in the first target layer; determining a candidate second sorting string table file stored in a second target layer according to the candidate first sorting string table file; wherein the second target layer is a storage layer one level lower than the first target layer; performing the merging operation according to the candidate first sorting string table file and the candidate second sorting string table file to obtain a target sorting string table file after merging; and storing the target sorting string table file to the storage component.

Various techniques are provided for enabling an iterator (called persistent key-value (KV) iterator) that uses a special key structure embedded with secondary index information for efficient search (or query) of a key-value database (KVDB). In some embodiments, the key structure comprises a plurality of fields categorized into two groups, infrastructure hierarchy fields and object characteristic fields. Keys for the KVDB may be partitioned into one or more levels of hierarchy to generate many smaller key lists for easy search / query of the KVDB. These smaller key lists can be used for concurrent searches / queries.

The present disclosure relates to controlling the quality of service of input / output flows associated with a key-value database. One or more input / output flows associated with one or more key-value databases may be received. Corresponding tags of the one or more input / output flows may be examined. Based on identification data obtained by examining the corresponding tags, a corresponding amount of bandwidth to be allocated to the one or more input / output flows may be determined to meet threshold criteria related to predetermined quality of service (QoS) parameters associated with the one or more input / output flows. The one or more input / output flows may be dynamically throttled across the one or more key-value databases using the corresponding allocated bandwidth amount to adjust processing time of input / output operations in the one or more input / output flows according to the QoS parameters.

The application provides a video recommendation method and system based on a large language model, an electronic device, and a storage medium. At least one target user intention label data is obtained from a key-value database according to a recommendation request initiated by a target user. A first video item set is screened from a streaming media platform according to a first content label set. A second video item set is screened from each first video item according to the first content label set and a first video label set of each first video item. A third video item set is screened from each video item according to target user text feedback data. A fourth video item set is screened from each video item according to each preset recall algorithm, and the fourth video item set is fused with the second video item set and the third video item set to obtain a fifth video item set. A video recommendation list generated according to a sixth video item set obtained by sorting the fifth video item set based on a second content label set is used to recommend videos to the target user.

Metadata generated during the lifetime of machine learning and artificial intelligence systems is valuable. However, such metadata can also be generated after data generation and thus written to the storage device later. Further support for KV databases at the storage device level can improve performance in terms of transfers per second. This is due to the removal of the translation layer on the host, which was previously required for data storage. Removing the translation layer eliminates two layers of mapping and transaction information. This increases the number of transactions per second, write gains, and read gains, while reducing latency.Additionally, future metadata additions are accommodated by reserving excess memory when storing the initial key value. The future additions are then stored in this reserved memory.

A method for performing a backup in relation to a separated database, the method includes creating, by a processing circuit, a new backup log structured merge (LSM) tree of the separated database, wherein the new backup LSM tree is associated with a new point in time (PIT) and belongs to a group of LSM trees that comprise a primary LSM tree and one or more backup LSM trees that are associated with different PITs, and share a key value (KV) database that is mutable and is separated from the group; wherein the creating comprises storing the new backup LSM tree to a destination.

The present disclosure provides an information processing method, apparatus, device and storage medium, which relate to the field of computer technology. The method includes: generating a resource modification request in response to a resource modification operation; processing the resource modification request and updating a first database, wherein the first database is a key-value database; sending a resource modification notification to a message queue through the first database, wherein the resource modification notification is obtained according to the update result of the first database; monitoring the message queue to update a second database according to the resource modification notification, wherein the second database is a relational database. The method realizes high-performance processing through a key-value database, and switches to a relational database for processing when the key-value database is down, thereby improving the stability and availability of information processing in the number source system.

The application relates to the technical field of data storage, and discloses a CSD-based key-value database storage method and system, which comprises the following steps: in response to a user inserting a key-value pair into a key-value database, storing the key-value pair into an in-memory table, and converting the in-memory table into an immutable in-memory table after a preset trigger condition is met; aligning and encoding the key-value pairs in the immutable in-memory table according to a uniform length to generate fixed-step key-value pairs; calling a computing storage device to automatically compress and flush the fixed-step key-value pairs into an SSTable file; receiving a trigger merge operation instruction, controlling the computing storage device to acquire the SSTable file and decompress the SSTable file; calling a graphics processing unit to perform merge processing on the decompressed file to generate a new SSTable file, and calling the computing storage device to automatically compress and flush the new file to a disk. The application significantly improves the read-write throughput and merge efficiency of the key-value database, and effectively alleviates the write blocking and write pausing problems.

This invention discloses a method, device, equipment, and medium for determining the grounding status of a grounding stake. When the contact of the code reading area in the grounding socket comes into contact with the contact of the coding area in the grounding head, the contact coding data corresponding to each conductive contact in the grounding socket is acquired. A preset feature key value database is retrieved according to the contact coding data to determine the corresponding target grounding wire. The target voltage level of the target grounding wire is compared with the standard voltage level corresponding to the associated grounding equipment. If the target voltage level matches the standard voltage level, the grounding status of the grounding stake to be judged is determined to be normal. This invention solves the technical problem that traditional grounding stake installation methods cannot monitor the grounding status of the grounding wire in real time. It enables the determination of the installation or removal status of the grounding wire, identifies the grounding wire number, and determines the grounding status of the grounding stake.

The invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of computers, in particular to the technical field of databases, distributed storage, data backup and the like. According to the specific implementation scheme, under the condition that a data backup instruction is received, a to-be-backed-up data identifier is determined; determining a data file where the to-be-backed-up data is located and a key value database associated with the data file; obtaining a consistency snapshot of the key value database at the first moment and path information of the data file; and loading the to-be-backed-up data to a specified position based on the consistent snapshot and the path information under the condition that a recovery instruction for the to-be-backed-up data is received. Therefore, the data is backed up and recovered by storing the file path, the snapshot speed can be improved, and resource consumption is reduced.

The application discloses a metadata storage method and device based on a distributed key-value database, and relates to the field of data processing.The method comprises the following steps: determining the metadata of an object stored in a bucket, and determining the preset type and key-value pair of the metadata; based on the determined preset type and key-value pair of the metadata, saving the metadata into a corresponding hash table and an ordered list; each bucket in the distributed storage system stores metadata through at least one hash table and at least one ordered list, the hash table is used for storing metadata, and the ordered list is used for storing retrieval information of the metadata.The application reduces the complexity of I / O operation, improves the single-bucket storage scale while ensuring efficiency, and realizes efficient, low-space-usage metadata retrieval applicable to multiple ordered lists.

A method for processing analytical-style queries on a transactional database is provided. A key-value database configured for point lookups and singular transactions is maintained. An analytical-style query request against the key-value database is received and analyzed to determine if it requires accessing more than a threshold amount of data. In response to a threshold, key-value data is separated into range granules covering specified data ranges. For each range granule, a snapshot file comprising key-value pairs at a specific database version and delta files specifying changes during time ranges are generated. The delta files are streamed via change feeds and buffered until accumulating a predetermined amount of mutations. The snapshot and delta files are provided to client execution nodes for parallel processing of the analytical-style query request.

The invention relates to an index calculation system and method, a computer readable storage medium and electronic equipment, and relates to the technical field of big data processing, the system comprises a distributed storage system and an index calculation client; wherein the index calculation client is used for determining original index groups in response to an input operation for an index configuration page, and performing merging processing on the original index groups to generate a first entity data acquisition request; the distributed storage system is used for determining an entity data acquisition rule based on the first entity data acquisition request, and extracting target entity data from a preset distributed key value database based on the entity data acquisition rule; and the index calculation client is also used for extracting original entity data corresponding to the original index group from the target entity data, and performing aggregation calculation on the original entity data to obtain an index calculation result corresponding to the original index group. The index calculation efficiency is improved.

A global lock is used to access a first set of data structures. An active transaction having a transaction start identifier is identified as a globally oldest active transaction associated with the first set of data structures. A first marker value of a first data structure of a second set of data structures is compared to the transaction start identifier to determine satisfaction of a first condition. In response to satisfying the first condition, the first data structure is accessed to identify a first set of data locks associated with one or more transactions each having a transaction completion identifier that satisfies a second condition when compared to the transaction start identifier. In response to satisfying the second condition, the first set of data locks is released.

The embodiment of the application discloses a batch anonymous query method, system and device for multiple data sources, which comprises the following steps: a client synchronizes an original polynomial and a polynomial quotient ring of the original polynomial to each of k servers; an n-order polynomial is calculated according to n query key values to obtain a first polynomial; each of the k servers calculates k second polynomials and k interpolation polynomials according to respective key value databases; the client and each client perform operations to obtain a first calculation result and k second calculation results; the client decomposes the first calculation result according to a preset constraint relationship to obtain a first decomposition result; the k servers determine k third calculation results; the client determines a fourth calculation result according to the k third calculation results and the original polynomial, and obtains a query result by traversing the n query key values. The embodiment of the application can improve the data query efficiency.

Embodiments of the present disclosure relate to control plane configuration. Examples relate to a system including a cluster of worker nodes communicatively connected to the control plane. The control plane includes a first control switch and a second control switch. The control switches are configured with virtual IP addresses. The IP addresses are associated with the control switches operating in active mode. The control switches communicate with the cluster of worker nodes via the virtual IP addresses. A single instance of a key-value database is deployed on one of the first and second control switches operating in active mode, and the first and second control switches are functionally coupled via a distributed block system for data synchronization.

Systems and methods for a serverless tiebreaker for a shared-nothing architecture are provided. In some examples, a cloud-native service that supports serialization of writes (or write fencing), for example, via atomic operations with persistent locking and / or reservations, is used to support HA mediation instead of a separate server operating as a tiebreaker, thereby reducing costs and complexity as well as increasing availability and durability of the HA mediation functionality. For example, a fast, fully managed, serverless, key-value noSQL database service (e.g., the Amazon DynamoDB) may be used to perform one or more of maintaining the authoritative source of information regarding which node of an HA pair currently represents the primary node for serving data from a particular dataset, persisting HA metadata, and / or assisting in the failover and failback processes.

This invention discloses a near-memory computing method and system for key-value databases based on computing storage devices, belonging to the field of computer information processing technology. The method includes the following steps: collecting access records formed by multiple storage nodes in a distributed computing storage device under a unified time scale, and labeling each access record with a corresponding time offset value to construct an access record set; performing time position callback processing based on the access record set, relocating the access records in time using the time offset value, so that the access records are merged into the same logical time period, forming a unified time period access record set. This invention achieves consistent expression of access data in logical time through unified time scale and time offset processing, improving the accuracy of hotspot identification; simultaneously, through access change splitting and progressive migration control, the migration process is advanced in stages and synchronized with access changes, reducing the concentration of resource consumption and improving system operational stability.

The invention relates to an object storage method and device, computer equipment, a storage medium and a program product. Receiving an object uploading request sent by a client, storing to-be-uploaded objects and object names in a cache pool under the condition that the data volume is smaller than a preset data volume, aggregating the object names of the preset number of to-be-uploaded objects in the cache pool in a memory to obtain aggregated object names, and sending the aggregated object names to the client; aggregating a preset number of to-be-uploaded objects in the cache pool in a memory through an asynchronous thread to obtain an aggregated object, and storing metadata of the aggregated object in the memory into a primary key value database, and storing the aggregation objects in the memory and the starting positions and the ending positions of the preset number of to-be-uploaded objects in the storage cluster to the storage cluster. In the embodiment of the invention, the small objects are firstly written into the cache pool and then aggregated in the memory to obtain the aggregated objects, and the aggregated objects are stored in the storage cluster, so that the I / O operation frequency of the storage cluster is greatly reduced, and the space utilization rate of the storage cluster is improved.

The invention relates to a ZNS SSD operation partition resetting and redistributing method based on LSM tree heat perception, which comprises the following steps: aiming at a key value database based on an LSM tree, proposing a hierarchy-based data heat classification method, dividing data heat into hot data, warm data and cold data, writing different heat data into different partitions of a ZNS SSD, and distributing heat labels for the partitions; according to the heat labels of the partitions, a heat-perceived runtime partition resetting method is provided to dynamically set a runtime partition resetting threshold value, so that the partitions for storing cold data are reset as soon as possible, the cold data are prevented from occupying unused space for a long time, and meanwhile, the partitions for storing hot data are written with data as much as possible. The space utilization rate of the partition for storing the hot data is improved; for an invalid data partition with a low space utilization rate for storing cold data, a partition redistribution method during operation is provided, the unused space of the partition is utilized again, the space utilization rate of the partition is further improved, and the erasing frequency is reduced.

The invention provides a micro-service local debugging method and device, electronic equipment and a storage medium, and relates to the technical field of micro-services. The method comprises the following steps: starting a local service and calling an intranet penetration tool to obtain a public network address which can be accessed by a micro-service cluster; the local service registers the public network address and the interface information of the local configuration corresponding to the public network address into a key value database; a target micro-service in the micro-service cluster receives a business request, and sends request data to a locally deployed interceptor for preprocessing; the interceptor obtains the public network address and the interface information from the key value database, and performs matching according to a preset matching rule based on the request data; if matching succeeds, the interceptor forwards the service request to the public network address corresponding to the matched local service, so that the matched local service processes the service request and completes local debugging of the service request, intermediate nodes do not need to be newly added or extra containers do not need to be deployed, resource consumption is reduced, and the micro-service debugging efficiency is greatly improved.