111 results about "Key-value database" patented technology

The invention relates to the technical field of data storage, and discloses a CSD-based key value database storage method and system, and the method comprises the steps: responding to a user inserting a key value pair into a key value database, storing the key value pair into a memory table, and converting the memory table into an immutable memory table after a preset triggering condition is satisfied; aligning and coding the key value pairs in the immutable memory table according to a uniform length to generate key value pairs with fixed step lengths; calling a computing storage device to automatically compress the key value pair with the fixed step length and flash the key value pair into an SSTable file; receiving a merging operation triggering instruction, and controlling the computing storage device to obtain the SSTable file and decompress the SSTable file; and calling the graphic processing unit to merge the decompressed files to generate a new SSTable file, and calling the computing storage device to automatically compress the new file and flash the new file to a disk. According to the method, the read-write throughput and merging efficiency of the key value database are remarkably improved, and the problems of write blockage and write pause are effectively relieved.

The embodiment of the invention relates to a data processing method and device based on a key value database, equipment, a storage medium and a key value database system.The method comprises the steps that after it is determined that to-be-cached key value data exists in a video memory of a graphic processing unit, a data storage request for the to-be-cached key value data is sent to a key value database server side, the data storage request carries storage address information and storage length information of value data of to-be-cached key value data in a video memory; receiving a data reading request which is sent by a key value database server and carries storage address information, and controlling the first network card supporting the remote direct memory access protocol to read value data of the key value data to be cached from the video memory according to the storage address information, sending the value data to the key value database server side, and writing the value data through the second network card supporting the remote direct memory access protocol. Visibly, according to the embodiment of the invention, the key value data in the GPU video memory can be stored without a memory copying process, and the processing efficiency of the key value data is improved.

The invention discloses a data processing method, which comprises the following steps of: performing parallel data processing on key value pair data in a key value database according to at least one data processing instruction to obtain data processing results respectively corresponding to the at least one data processing instruction; according to the method, at least one data processing instruction is received, and the data processing result corresponding to the at least one data processing instruction is fed back to the client, so that the client can be efficiently connected to the key value database by using the preset data processing instruction, and parallel data processing can be performed on the key value data in the key value database; according to the method and the device, efficient parallel data processing is realized, so that the performance of the application program is improved, the performance of the client is improved, the operation delay is reduced, the data processing efficiency can be improved, the data processing cost is reduced, and the user experience of data processing is improved.

The invention provides a graph data processing method, device and equipment and a computer readable storage medium. The method comprises the following steps: acquiring to-be-processed graph data, and acquiring a plurality of attribute indexes corresponding to the to-be-processed graph data; obtaining index data of each attribute index, and obtaining a vertex identifier and an attribute value corresponding to each attribute index; encoding the index data of each attribute index and the vertex identifier and the attribute value corresponding to each attribute index to obtain key value pair information corresponding to each attribute index; and storing the key value pair information corresponding to each attribute index into a key value database. Through the method and the device, high attribute query efficiency can be realized.

The invention provides a video recommendation method and system based on a large language model, electronic equipment and a storage medium. The method comprises the following steps: acquiring at least one target user intention tag data from a key value database according to a recommendation request initiated by a target user; screening out a first video item set from the streaming media platform according to the first content label set; screening out a second video item set from each first video item according to the first content label set and the first video label set of each first video item; screening out a third video entry set from the video entries according to the target user text feedback data; screening out a fourth video entry set from the video entries according to each preset recall algorithm, and fusing the fourth video entry set with the second video entry set and the third video entry set to obtain a fifth video entry set; and recommending videos to the target user according to a video recommendation list generated by a sixth video item set obtained by sorting the fifth video item set based on the second content label set.

The invention belongs to the field of big data, and relates to a directory path analysis method, which comprises the following steps: acquiring directory information of a file system, and creating a metadata cache in a metadata client for storing the directory information; receiving a directory path analysis request, and judging whether a directory accessed by the directory path analysis request exists in the metadata cache or not based on the directory information; if the directory accessed by the directory path analysis request exists in the metadata cache, acquiring an entity identifier corresponding to the directory from the metadata cache; sending the entity identifier to a metadata server, so as to query result data corresponding to the directory from a key value database through the metadata server; and returning result data corresponding to the directory path analysis request. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and directory information and result data can be stored in a block chain. According to the method, the directory path analysis time delay can be reduced, and the problem of near root hotspots is solved.

The present application relates to the technical field of storage, and discloses a data attribute-aware storage system and data management method for a key-value database. The method comprises two parts, i.e., data hotness attribute-based writing and data density attribute-based merging. The data hotness attribute-based writing comprises classifying key-value pairs into cold key-value pairs and hot key-value pairs, to realize separate storage of cold data and hot data by means of a data hotness attribute-based writing strategy. The data density attribute-based merging comprises: when key-value pairs in an SSTable are extracted during a merging operation and form a sorted key-value pair sequence, extracting the key-value pairs one by one, and calculating a binary difference between every two adjacent key-value pairs; and if the difference is greater than a set threshold, stopping filling the current SSTable, and creating a new SSTable. In the present invention, by analyzing data hotness and data density, data having different attributes are classified to different data partitions, thereby reducing I / O amplification caused by repeated reading and writing in a compression process.

The invention discloses a metadata persistence method, an electronic device, a storage medium and a program product, and relates to the technical field of computers, when a request is written, a newly added forward relation of a small amount of data is generated from written data, the newly added forward relation is added to the tail of an existing forward relation of a distributed key value database, and the newly added forward relation is added to the tail of the existing forward relation of the distributed key value database. The method does not need to acquire the existing forward relation and the newly added forward relation, update the forward relation and then write the updated forward relation into the distributed key value database again, reduces the number of times of writing into the distributed key value database, reduces the time delay of updating the forward relation, does not need to read out the existing forward relation and deserialize the existing forward relation, reduces the time delay of updating the forward relation, and reduces the time delay of updating the forward relation. And then serialized writing is performed, so that the processor consumption is reduced. According to the method, the forward relation is spliced to form the spliced forward relation when the request is read, the data is read from the data pool according to the spliced forward relation, the read request is associated with the written data, data reading is not affected, and the consumption of a processor can be comprehensively reduced.

The invention relates to the technical field of key value databases, and discloses a key value database near-storage computing system based on computing storage equipment, the system comprises a host, a memory and a computing memory, a decoder in the computing memory is used for reading files needing to be merged from the memory, analyzing all key value pairs from the files, and storing the key value pairs in the computing memory; placing the key value pairs in a buffer area connected with a sequencer; the sequencer is used for comparing the size relationship of the key value pairs, taking out the key value pair from the corresponding decoder buffer area according to the comparison result of each time, selecting the key value pair with the minimum key value according to the size of the key in the key value pair, and transmitting the corresponding key value pair to the buffer area connected with the encoder; and the encoder is used for extracting the key value pair from the buffer area connected with the sequencer, generating an SS-Table file according to a file encoding rule and storing the SS-Table file into the memory. Deep fusion of calculation and storage can be realized, calculation resources in a host are fully utilized, and the overall system performance is improved.

The invention provides a data migration method and device, electronic equipment, a computer readable medium and a computer program product. The method comprises the steps that in response to a data migration instruction, a first data processing strategy is executed to process a data write-in request of a client side, and the first data processing strategy executes corresponding data write-in operation in a source database and records the data write-in request with a target key name in a key value database; synchronizing target data in the source data to a key value database through asynchronous data processing, wherein the target data corresponds to the data segment selected by the user; in response to an instruction of completing data migration, switching the executed first data processing strategy to a second data processing strategy, the second data processing strategy enabling data to be operated with an original key name in a key value database; and cleaning the target key names in the key value database by scanning the target key names in the key value database.

The invention belongs to the field of artificial intelligence and financial science and technology, and relates to a data backup recovery method which comprises the following steps: monitoring through a monitoring mechanism of a key value database, packaging a backup request into a backup request event, and writing the backup request event into a backup key value pair of the key value database; when a backup request event is monitored, calling a data writing instruction to write a data file into a preset storage position, and meanwhile, pausing a message of the consumption message middleware; after writing of the data file is completed, an index file is constructed according to metadata of the data file, and when construction of the index file is completed, snapshot backup operation is executed for backup; and after backup is completed, normal data processing is recovered. The invention further provides a data backup recovery device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the data file can be stored in a block chain. According to the method, the data backup and recovery time can be greatly shortened, and the data backup and recovery timeliness is improved.

The invention discloses an index operation method and device, a computing device cluster and a storage medium, and the method comprises the following steps: firstly, a key value database receives a first index operation command, the first index operation command comprises an index name of a target index and a first index operation type, and then the index operation type of the target index is determined according to the first index operation command; and executing the operation of the first index operation type on the target index. Wherein the target index comprises a target primary key index and / or a target secondary index, the target primary key index comprises a target primary key and target data corresponding to the target primary key, the target secondary index comprises a target secondary index key and the target primary key, and the target secondary index is used for the key value database to obtain the target primary key; and obtaining target data corresponding to the target primary key based on the target primary key. According to the scheme, the user is supported to specify the index to operate, and the operation efficiency and the user experience of the key value database are improved.

The present disclosure relates to a data processing method and device, electronic equipment and storage medium. The method comprises: receiving a data processing request sent by a client, the data processing request being used to indicate the Nth query and carrying a target cursor parameter; the target cursor parameter is an encoding parameter generated in the (N-1)th query, N being a positive integer greater than 1; the target key corresponding to the target cursor parameter is obtained from the mapping relationship between the cursor parameter and the key, the target key indicating the end position of the (N-1)th query in the distributed key-value database; and the (N)th query is performed on the distributed key-value database according to the target key. According to the technical scheme of the present disclosure, the cursor parameter is configured according to the demand of the original command, the corresponding relationship between the cursor parameter and the key is stored, so that the starting position of the query can be accurately analyzed based on the target cursor parameter, and the distributed key-value database has good compatibility.

The present disclosure provides a file processing method and device for a key-value database, equipment, a storage medium and a program product, relates to the technical field of computers, and can be applied to the technical field of finance. The method comprises the following steps: in response to receiving a merging operation instruction, determining a first target layer of a storage component of the key-value database; determining a candidate first sorting string table file stored in the first target layer; determining a candidate second sorting string table file stored in a second target layer according to the candidate first sorting string table file; wherein the second target layer is a storage layer one level lower than the first target layer; performing the merging operation according to the candidate first sorting string table file and the candidate second sorting string table file to obtain a target sorting string table file after merging; and storing the target sorting string table file to the storage component.

The invention relates to a database splitting method and device. The method comprises the following steps: acquiring database information of an original database; wherein the original database is a key value database, and the database information comprises file attribute information of each disk file in the original database; the file attribute information of each magnetic disk file at least comprises a maximum key and a middle key in a key value pair stored in the magnetic disk file, layer information of a storage layer to which the magnetic disk file belongs in the original database, and the size of the magnetic disk file; according to the database information, selecting split keys used for splitting the original database from the key value pairs stored in the disk files; and splitting the original database according to the splitting key, the database information and the system characteristics of the file system corresponding to the original database. By adopting the method, the split key can be quickly determined, and the database can be quickly and effectively split.

Various techniques are provided for enabling an iterator (called persistent key-value (KV) iterator) that uses a special key structure embedded with secondary index information for efficient search (or query) of a key-value database (KVDB). In some embodiments, the key structure comprises a plurality of fields categorized into two groups, infrastructure hierarchy fields and object characteristic fields. Keys for the KVDB may be partitioned into one or more levels of hierarchy to generate many smaller key lists for easy search / query of the KVDB. These smaller key lists can be used for concurrent searches / queries.

The present disclosure relates to controlling the quality of service of input / output flows associated with a key-value database. One or more input / output flows associated with one or more key-value databases may be received. Corresponding tags of the one or more input / output flows may be examined. Based on identification data obtained by examining the corresponding tags, a corresponding amount of bandwidth to be allocated to the one or more input / output flows may be determined to meet threshold criteria related to predetermined quality of service (QoS) parameters associated with the one or more input / output flows. The one or more input / output flows may be dynamically throttled across the one or more key-value databases using the corresponding allocated bandwidth amount to adjust processing time of input / output operations in the one or more input / output flows according to the QoS parameters.

The application provides a video recommendation method and system based on a large language model, an electronic device, and a storage medium. At least one target user intention label data is obtained from a key-value database according to a recommendation request initiated by a target user. A first video item set is screened from a streaming media platform according to a first content label set. A second video item set is screened from each first video item according to the first content label set and a first video label set of each first video item. A third video item set is screened from each video item according to target user text feedback data. A fourth video item set is screened from each video item according to each preset recall algorithm, and the fourth video item set is fused with the second video item set and the third video item set to obtain a fifth video item set. A video recommendation list generated according to a sixth video item set obtained by sorting the fifth video item set based on a second content label set is used to recommend videos to the target user.

Metadata generated during the lifetime of machine learning and artificial intelligence systems is valuable. However, such metadata can also be generated after data generation and thus written to the storage device later. Further support for KV databases at the storage device level can improve performance in terms of transfers per second. This is due to the removal of the translation layer on the host, which was previously required for data storage. Removing the translation layer eliminates two layers of mapping and transaction information. This increases the number of transactions per second, write gains, and read gains, while reducing latency.Additionally, future metadata additions are accommodated by reserving excess memory when storing the initial key value. The future additions are then stored in this reserved memory.

A method for performing a backup in relation to a separated database, the method includes creating, by a processing circuit, a new backup log structured merge (LSM) tree of the separated database, wherein the new backup LSM tree is associated with a new point in time (PIT) and belongs to a group of LSM trees that comprise a primary LSM tree and one or more backup LSM trees that are associated with different PITs, and share a key value (KV) database that is mutable and is separated from the group; wherein the creating comprises storing the new backup LSM tree to a destination.

The present disclosure provides an information processing method, apparatus, device and storage medium, which relate to the field of computer technology. The method includes: generating a resource modification request in response to a resource modification operation; processing the resource modification request and updating a first database, wherein the first database is a key-value database; sending a resource modification notification to a message queue through the first database, wherein the resource modification notification is obtained according to the update result of the first database; monitoring the message queue to update a second database according to the resource modification notification, wherein the second database is a relational database. The method realizes high-performance processing through a key-value database, and switches to a relational database for processing when the key-value database is down, thereby improving the stability and availability of information processing in the number source system.

The application relates to the technical field of data storage, and discloses a CSD-based key-value database storage method and system, which comprises the following steps: in response to a user inserting a key-value pair into a key-value database, storing the key-value pair into an in-memory table, and converting the in-memory table into an immutable in-memory table after a preset trigger condition is met; aligning and encoding the key-value pairs in the immutable in-memory table according to a uniform length to generate fixed-step key-value pairs; calling a computing storage device to automatically compress and flush the fixed-step key-value pairs into an SSTable file; receiving a trigger merge operation instruction, controlling the computing storage device to acquire the SSTable file and decompress the SSTable file; calling a graphics processing unit to perform merge processing on the decompressed file to generate a new SSTable file, and calling the computing storage device to automatically compress and flush the new file to a disk. The application significantly improves the read-write throughput and merge efficiency of the key-value database, and effectively alleviates the write blocking and write pausing problems.

This invention discloses a method, device, equipment, and medium for determining the grounding status of a grounding stake. When the contact of the code reading area in the grounding socket comes into contact with the contact of the coding area in the grounding head, the contact coding data corresponding to each conductive contact in the grounding socket is acquired. A preset feature key value database is retrieved according to the contact coding data to determine the corresponding target grounding wire. The target voltage level of the target grounding wire is compared with the standard voltage level corresponding to the associated grounding equipment. If the target voltage level matches the standard voltage level, the grounding status of the grounding stake to be judged is determined to be normal. This invention solves the technical problem that traditional grounding stake installation methods cannot monitor the grounding status of the grounding wire in real time. It enables the determination of the installation or removal status of the grounding wire, identifies the grounding wire number, and determines the grounding status of the grounding stake.

A distributed key-value database column data and statistics system and method includes more than two data storage servers and statistics servers; the data storage server includes a data shard basic statistical value management module; the data shard basic statistical value management module is used to manage basic statistical values, which correspond to data shards; the data shard statistical value change management module is used to manage statistical change values, which correspond to data shards, and are used to record the change statistics of data columns; the application decomposes the calculation operation of "the statistical sum of a column of data in an ultra-large-scale data list" into local calculations of each data storage data shard, summarizes the statistical sum of the entire data, and changes it into the sum of the basic statistical value and the statistical change value in each data storage data shard, greatly improving the operating efficiency of artificial intelligence or large-scale databases and greatly reducing power consumption.

The invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of computers, in particular to the technical field of databases, distributed storage, data backup and the like. According to the specific implementation scheme, under the condition that a data backup instruction is received, a to-be-backed-up data identifier is determined; determining a data file where the to-be-backed-up data is located and a key value database associated with the data file; obtaining a consistency snapshot of the key value database at the first moment and path information of the data file; and loading the to-be-backed-up data to a specified position based on the consistent snapshot and the path information under the condition that a recovery instruction for the to-be-backed-up data is received. Therefore, the data is backed up and recovered by storing the file path, the snapshot speed can be improved, and resource consumption is reduced.

An event trigger to flush key-value data from a volatile memory device is detected. In response to detection of the event trigger, a determination is made whether the key-value data comprises uncommitted data for one or more active database transactions. If the key-value data comprises uncommitted data for one or more active database transactions, the uncommitted data is moved to a hold aside buffer in the volatile memory device, and committed data in the key-value data is stored on a non-volatile memory device.

The application discloses a metadata storage method and device based on a distributed key-value database, and relates to the field of data processing.The method comprises the following steps: determining the metadata of an object stored in a bucket, and determining the preset type and key-value pair of the metadata; based on the determined preset type and key-value pair of the metadata, saving the metadata into a corresponding hash table and an ordered list; each bucket in the distributed storage system stores metadata through at least one hash table and at least one ordered list, the hash table is used for storing metadata, and the ordered list is used for storing retrieval information of the metadata.The application reduces the complexity of I / O operation, improves the single-bucket storage scale while ensuring efficiency, and realizes efficient, low-space-usage metadata retrieval applicable to multiple ordered lists.

The present invention relates to the field of cloud storage technology, and discloses a distributed database supporting a massive version of metadata storage method and a retrieval method, including: obtaining initial metadata and a timestamp corresponding to the initial metadata; adding the timestamp as version information to the initial key and modifying the initial key; sorting the modified metadata from large to small by the timestamp to construct a large-capacity table. The present invention has the advantage of wide applicability and is applicable to most distributed key-value databases, as long as the database has the ability to sort according to the lexicographic order of the keys. Without affecting the function and performance of the distributed key-value database itself, it is achieved that there is no upper limit on the number of versions of metadata records supported. The retrieval performance and service quality are not affected by the number of versions of the metadata record, which solves the problems of load imbalance, performance degradation, service unavailability, etc. faced by large-capacity tables in distributed key-value databases when storing massive versions of metadata records.

A method for processing analytical-style queries on a transactional database is provided. A key-value database configured for point lookups and singular transactions is maintained. An analytical-style query request against the key-value database is received and analyzed to determine if it requires accessing more than a threshold amount of data. In response to a threshold, key-value data is separated into range granules covering specified data ranges. For each range granule, a snapshot file comprising key-value pairs at a specific database version and delta files specifying changes during time ranges are generated. The delta files are streamed via change feeds and buffered until accumulating a predetermined amount of mutations. The snapshot and delta files are provided to client execution nodes for parallel processing of the analytical-style query request.

The invention relates to an index calculation system and method, a computer readable storage medium and electronic equipment, and relates to the technical field of big data processing, the system comprises a distributed storage system and an index calculation client; wherein the index calculation client is used for determining original index groups in response to an input operation for an index configuration page, and performing merging processing on the original index groups to generate a first entity data acquisition request; the distributed storage system is used for determining an entity data acquisition rule based on the first entity data acquisition request, and extracting target entity data from a preset distributed key value database based on the entity data acquisition rule; and the index calculation client is also used for extracting original entity data corresponding to the original index group from the target entity data, and performing aggregation calculation on the original entity data to obtain an index calculation result corresponding to the original index group. The index calculation efficiency is improved.