67 results about "Key-value database" patented technology

The invention relates to the technical field of data storage, and discloses a CSD-based key value database storage method and system, and the method comprises the steps: responding to a user inserting a key value pair into a key value database, storing the key value pair into a memory table, and converting the memory table into an immutable memory table after a preset triggering condition is satisfied; aligning and coding the key value pairs in the immutable memory table according to a uniform length to generate key value pairs with fixed step lengths; calling a computing storage device to automatically compress the key value pair with the fixed step length and flash the key value pair into an SSTable file; receiving a merging operation triggering instruction, and controlling the computing storage device to obtain the SSTable file and decompress the SSTable file; and calling the graphic processing unit to merge the decompressed files to generate a new SSTable file, and calling the computing storage device to automatically compress the new file and flash the new file to a disk. According to the method, the read-write throughput and merging efficiency of the key value database are remarkably improved, and the problems of write blockage and write pause are effectively relieved.

The invention discloses a data processing method, which comprises the following steps of: performing parallel data processing on key value pair data in a key value database according to at least one data processing instruction to obtain data processing results respectively corresponding to the at least one data processing instruction; according to the method, at least one data processing instruction is received, and the data processing result corresponding to the at least one data processing instruction is fed back to the client, so that the client can be efficiently connected to the key value database by using the preset data processing instruction, and parallel data processing can be performed on the key value data in the key value database; according to the method and the device, efficient parallel data processing is realized, so that the performance of the application program is improved, the performance of the client is improved, the operation delay is reduced, the data processing efficiency can be improved, the data processing cost is reduced, and the user experience of data processing is improved.

The present application relates to the technical field of storage, and discloses a data attribute-aware storage system and data management method for a key-value database. The method comprises two parts, i.e., data hotness attribute-based writing and data density attribute-based merging. The data hotness attribute-based writing comprises classifying key-value pairs into cold key-value pairs and hot key-value pairs, to realize separate storage of cold data and hot data by means of a data hotness attribute-based writing strategy. The data density attribute-based merging comprises: when key-value pairs in an SSTable are extracted during a merging operation and form a sorted key-value pair sequence, extracting the key-value pairs one by one, and calculating a binary difference between every two adjacent key-value pairs; and if the difference is greater than a set threshold, stopping filling the current SSTable, and creating a new SSTable. In the present invention, by analyzing data hotness and data density, data having different attributes are classified to different data partitions, thereby reducing I / O amplification caused by repeated reading and writing in a compression process.

The invention provides a data migration method and device, electronic equipment, a computer readable medium and a computer program product. The method comprises the steps that in response to a data migration instruction, a first data processing strategy is executed to process a data write-in request of a client side, and the first data processing strategy executes corresponding data write-in operation in a source database and records the data write-in request with a target key name in a key value database; synchronizing target data in the source data to a key value database through asynchronous data processing, wherein the target data corresponds to the data segment selected by the user; in response to an instruction of completing data migration, switching the executed first data processing strategy to a second data processing strategy, the second data processing strategy enabling data to be operated with an original key name in a key value database; and cleaning the target key names in the key value database by scanning the target key names in the key value database.

The present disclosure relates to a data processing method and device, electronic equipment and storage medium. The method comprises: receiving a data processing request sent by a client, the data processing request being used to indicate the Nth query and carrying a target cursor parameter; the target cursor parameter is an encoding parameter generated in the (N-1)th query, N being a positive integer greater than 1; the target key corresponding to the target cursor parameter is obtained from the mapping relationship between the cursor parameter and the key, the target key indicating the end position of the (N-1)th query in the distributed key-value database; and the (N)th query is performed on the distributed key-value database according to the target key. According to the technical scheme of the present disclosure, the cursor parameter is configured according to the demand of the original command, the corresponding relationship between the cursor parameter and the key is stored, so that the starting position of the query can be accurately analyzed based on the target cursor parameter, and the distributed key-value database has good compatibility.

The present disclosure provides a file processing method and device for a key-value database, equipment, a storage medium and a program product, relates to the technical field of computers, and can be applied to the technical field of finance. The method comprises the following steps: in response to receiving a merging operation instruction, determining a first target layer of a storage component of the key-value database; determining a candidate first sorting string table file stored in the first target layer; determining a candidate second sorting string table file stored in a second target layer according to the candidate first sorting string table file; wherein the second target layer is a storage layer one level lower than the first target layer; performing the merging operation according to the candidate first sorting string table file and the candidate second sorting string table file to obtain a target sorting string table file after merging; and storing the target sorting string table file to the storage component.

Various techniques are provided for enabling an iterator (called persistent key-value (KV) iterator) that uses a special key structure embedded with secondary index information for efficient search (or query) of a key-value database (KVDB). In some embodiments, the key structure comprises a plurality of fields categorized into two groups, infrastructure hierarchy fields and object characteristic fields. Keys for the KVDB may be partitioned into one or more levels of hierarchy to generate many smaller key lists for easy search / query of the KVDB. These smaller key lists can be used for concurrent searches / queries.

The application provides a video recommendation method and system based on a large language model, an electronic device, and a storage medium. At least one target user intention label data is obtained from a key-value database according to a recommendation request initiated by a target user. A first video item set is screened from a streaming media platform according to a first content label set. A second video item set is screened from each first video item according to the first content label set and a first video label set of each first video item. A third video item set is screened from each video item according to target user text feedback data. A fourth video item set is screened from each video item according to each preset recall algorithm, and the fourth video item set is fused with the second video item set and the third video item set to obtain a fifth video item set. A video recommendation list generated according to a sixth video item set obtained by sorting the fifth video item set based on a second content label set is used to recommend videos to the target user.

Metadata generated during the lifetime of machine learning and artificial intelligence systems is valuable. However, such metadata can also be generated after data generation and thus written to the storage device later. Further support for KV databases at the storage device level can improve performance in terms of transfers per second. This is due to the removal of the translation layer on the host, which was previously required for data storage. Removing the translation layer eliminates two layers of mapping and transaction information. This increases the number of transactions per second, write gains, and read gains, while reducing latency.Additionally, future metadata additions are accommodated by reserving excess memory when storing the initial key value. The future additions are then stored in this reserved memory.

The application relates to the technical field of data storage, and discloses a CSD-based key-value database storage method and system, which comprises the following steps: in response to a user inserting a key-value pair into a key-value database, storing the key-value pair into an in-memory table, and converting the in-memory table into an immutable in-memory table after a preset trigger condition is met; aligning and encoding the key-value pairs in the immutable in-memory table according to a uniform length to generate fixed-step key-value pairs; calling a computing storage device to automatically compress and flush the fixed-step key-value pairs into an SSTable file; receiving a trigger merge operation instruction, controlling the computing storage device to acquire the SSTable file and decompress the SSTable file; calling a graphics processing unit to perform merge processing on the decompressed file to generate a new SSTable file, and calling the computing storage device to automatically compress and flush the new file to a disk. The application significantly improves the read-write throughput and merge efficiency of the key-value database, and effectively alleviates the write blocking and write pausing problems.

This invention discloses a method, device, equipment, and medium for determining the grounding status of a grounding stake. When the contact of the code reading area in the grounding socket comes into contact with the contact of the coding area in the grounding head, the contact coding data corresponding to each conductive contact in the grounding socket is acquired. A preset feature key value database is retrieved according to the contact coding data to determine the corresponding target grounding wire. The target voltage level of the target grounding wire is compared with the standard voltage level corresponding to the associated grounding equipment. If the target voltage level matches the standard voltage level, the grounding status of the grounding stake to be judged is determined to be normal. This invention solves the technical problem that traditional grounding stake installation methods cannot monitor the grounding status of the grounding wire in real time. It enables the determination of the installation or removal status of the grounding wire, identifies the grounding wire number, and determines the grounding status of the grounding stake.

The invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of computers, in particular to the technical field of databases, distributed storage, data backup and the like. According to the specific implementation scheme, under the condition that a data backup instruction is received, a to-be-backed-up data identifier is determined; determining a data file where the to-be-backed-up data is located and a key value database associated with the data file; obtaining a consistency snapshot of the key value database at the first moment and path information of the data file; and loading the to-be-backed-up data to a specified position based on the consistent snapshot and the path information under the condition that a recovery instruction for the to-be-backed-up data is received. Therefore, the data is backed up and recovered by storing the file path, the snapshot speed can be improved, and resource consumption is reduced.

The application discloses a metadata storage method and device based on a distributed key-value database, and relates to the field of data processing.The method comprises the following steps: determining the metadata of an object stored in a bucket, and determining the preset type and key-value pair of the metadata; based on the determined preset type and key-value pair of the metadata, saving the metadata into a corresponding hash table and an ordered list; each bucket in the distributed storage system stores metadata through at least one hash table and at least one ordered list, the hash table is used for storing metadata, and the ordered list is used for storing retrieval information of the metadata.The application reduces the complexity of I / O operation, improves the single-bucket storage scale while ensuring efficiency, and realizes efficient, low-space-usage metadata retrieval applicable to multiple ordered lists.

A method for processing analytical-style queries on a transactional database is provided. A key-value database configured for point lookups and singular transactions is maintained. An analytical-style query request against the key-value database is received and analyzed to determine if it requires accessing more than a threshold amount of data. In response to a threshold, key-value data is separated into range granules covering specified data ranges. For each range granule, a snapshot file comprising key-value pairs at a specific database version and delta files specifying changes during time ranges are generated. The delta files are streamed via change feeds and buffered until accumulating a predetermined amount of mutations. The snapshot and delta files are provided to client execution nodes for parallel processing of the analytical-style query request.

The invention relates to an index calculation system and method, a computer readable storage medium and electronic equipment, and relates to the technical field of big data processing, the system comprises a distributed storage system and an index calculation client; wherein the index calculation client is used for determining original index groups in response to an input operation for an index configuration page, and performing merging processing on the original index groups to generate a first entity data acquisition request; the distributed storage system is used for determining an entity data acquisition rule based on the first entity data acquisition request, and extracting target entity data from a preset distributed key value database based on the entity data acquisition rule; and the index calculation client is also used for extracting original entity data corresponding to the original index group from the target entity data, and performing aggregation calculation on the original entity data to obtain an index calculation result corresponding to the original index group. The index calculation efficiency is improved.

The embodiment of the application discloses a batch anonymous query method, system and device for multiple data sources, which comprises the following steps: a client synchronizes an original polynomial and a polynomial quotient ring of the original polynomial to each of k servers; an n-order polynomial is calculated according to n query key values to obtain a first polynomial; each of the k servers calculates k second polynomials and k interpolation polynomials according to respective key value databases; the client and each client perform operations to obtain a first calculation result and k second calculation results; the client decomposes the first calculation result according to a preset constraint relationship to obtain a first decomposition result; the k servers determine k third calculation results; the client determines a fourth calculation result according to the k third calculation results and the original polynomial, and obtains a query result by traversing the n query key values. The embodiment of the application can improve the data query efficiency.

Embodiments of the present disclosure relate to control plane configuration. Examples relate to a system including a cluster of worker nodes communicatively connected to the control plane. The control plane includes a first control switch and a second control switch. The control switches are configured with virtual IP addresses. The IP addresses are associated with the control switches operating in active mode. The control switches communicate with the cluster of worker nodes via the virtual IP addresses. A single instance of a key-value database is deployed on one of the first and second control switches operating in active mode, and the first and second control switches are functionally coupled via a distributed block system for data synchronization.

Systems and methods for a serverless tiebreaker for a shared-nothing architecture are provided. In some examples, a cloud-native service that supports serialization of writes (or write fencing), for example, via atomic operations with persistent locking and / or reservations, is used to support HA mediation instead of a separate server operating as a tiebreaker, thereby reducing costs and complexity as well as increasing availability and durability of the HA mediation functionality. For example, a fast, fully managed, serverless, key-value noSQL database service (e.g., the Amazon DynamoDB) may be used to perform one or more of maintaining the authoritative source of information regarding which node of an HA pair currently represents the primary node for serving data from a particular dataset, persisting HA metadata, and / or assisting in the failover and failback processes.

This invention discloses a near-memory computing method and system for key-value databases based on computing storage devices, belonging to the field of computer information processing technology. The method includes the following steps: collecting access records formed by multiple storage nodes in a distributed computing storage device under a unified time scale, and labeling each access record with a corresponding time offset value to construct an access record set; performing time position callback processing based on the access record set, relocating the access records in time using the time offset value, so that the access records are merged into the same logical time period, forming a unified time period access record set. This invention achieves consistent expression of access data in logical time through unified time scale and time offset processing, improving the accuracy of hotspot identification; simultaneously, through access change splitting and progressive migration control, the migration process is advanced in stages and synchronized with access changes, reducing the concentration of resource consumption and improving system operational stability.

The invention relates to an object storage method and device, computer equipment, a storage medium and a program product. Receiving an object uploading request sent by a client, storing to-be-uploaded objects and object names in a cache pool under the condition that the data volume is smaller than a preset data volume, aggregating the object names of the preset number of to-be-uploaded objects in the cache pool in a memory to obtain aggregated object names, and sending the aggregated object names to the client; aggregating a preset number of to-be-uploaded objects in the cache pool in a memory through an asynchronous thread to obtain an aggregated object, and storing metadata of the aggregated object in the memory into a primary key value database, and storing the aggregation objects in the memory and the starting positions and the ending positions of the preset number of to-be-uploaded objects in the storage cluster to the storage cluster. In the embodiment of the invention, the small objects are firstly written into the cache pool and then aggregated in the memory to obtain the aggregated objects, and the aggregated objects are stored in the storage cluster, so that the I / O operation frequency of the storage cluster is greatly reduced, and the space utilization rate of the storage cluster is improved.

The invention relates to a ZNS SSD operation partition resetting and redistributing method based on LSM tree heat perception, which comprises the following steps: aiming at a key value database based on an LSM tree, proposing a hierarchy-based data heat classification method, dividing data heat into hot data, warm data and cold data, writing different heat data into different partitions of a ZNS SSD, and distributing heat labels for the partitions; according to the heat labels of the partitions, a heat-perceived runtime partition resetting method is provided to dynamically set a runtime partition resetting threshold value, so that the partitions for storing cold data are reset as soon as possible, the cold data are prevented from occupying unused space for a long time, and meanwhile, the partitions for storing hot data are written with data as much as possible. The space utilization rate of the partition for storing the hot data is improved; for an invalid data partition with a low space utilization rate for storing cold data, a partition redistribution method during operation is provided, the unused space of the partition is utilized again, the space utilization rate of the partition is further improved, and the erasing frequency is reduced.

The invention provides a micro-service local debugging method and device, electronic equipment and a storage medium, and relates to the technical field of micro-services. The method comprises the following steps: starting a local service and calling an intranet penetration tool to obtain a public network address which can be accessed by a micro-service cluster; the local service registers the public network address and the interface information of the local configuration corresponding to the public network address into a key value database; a target micro-service in the micro-service cluster receives a business request, and sends request data to a locally deployed interceptor for preprocessing; the interceptor obtains the public network address and the interface information from the key value database, and performs matching according to a preset matching rule based on the request data; if matching succeeds, the interceptor forwards the service request to the public network address corresponding to the matched local service, so that the matched local service processes the service request and completes local debugging of the service request, intermediate nodes do not need to be newly added or extra containers do not need to be deployed, resource consumption is reduced, and the micro-service debugging efficiency is greatly improved.

According to the distributed key value database garbage collection method, a database and a server system, each client obtains a timestamp ts by using a local clock, carries ts in a write transaction request and sends the write transaction request to a server; the server is used for maintaining a time stamp (min < write >); the server refreshes the minwrites at regular intervals, and it is guaranteed that the minwrites are continuously and progressively increased; if ts of the write transaction request is smaller than minwrite, the server refuses the write transaction request; the server reports the minwritets and the latstgcts to the bmd server side at regular intervals, and the minwritets and the latstgcts are reported to the bmd server side at regular intervals; the garbage collection management table records the ID of each server and the corresponding parameter A and parameter B; the minimum parameter in the garbage collection management table is calculated regularly; based on the latestgcts, an unlocking request is initiated to each server; after the bmd server side successfully completes unlocking of each server, a timestamp latestgcts corresponding to successful unlocking is recorded; the kv-server receives the latestgcts and the safegcts, the latestgcts and the safegcts are reported next time to be subjected to two-stage confirmation, the latestgcts and the latest minwrites are reported together, the server executes garbage collection regularly, and expired data smaller than the safegcts are cleared up.

The present application relates to the technical field of key-value databases, and discloses a key-value database near-storage computing system based on a computing storage device. The system includes: a host, a memory, and a computing memory. A decoder in the computing memory is used to read files to be merged from the memory, parse all key-value pairs from the files, and place the key-value pairs in a buffer connected to a sorter; the sorter is used to compare the size relationship of key-value pairs, and according to the result of each comparison, take out the key-value pairs from the corresponding decoder buffer, select the key-value pair with the smallest key according to the size of the key in the key-value pair, and transfer the corresponding key-value pair to a buffer connected to an encoder; the encoder is used to take out the key-value pairs from the buffer connected to the sorter, and generate an SS-Table file according to the file encoding rules and store it in the memory. This can achieve deep integration of computing and storage, fully utilize the computing resources in the host, and improve the overall system performance.

The application discloses a metadata persistence method, an electronic device, a storage medium and a program product, and relates to the technical field of computers. When a write request is received, only a small amount of data is generated to add a new forward relationship of the write-in data, and the new forward relationship is added to the tail of an existing forward relationship of a distributed key-value database. The existing forward relationship and the new forward relationship do not need to be acquired and updated, and then the updated forward relationship is written into the distributed key-value database again, so that the number of write-in times to the distributed key-value database is reduced, the time delay of forward relationship updating is reduced, and the processor consumption is reduced. When a read request is received, a spliced forward relationship is formed by splicing the forward relationship, and data is read from the data pool according to the spliced forward relationship. The read request and the write-in data are associated with each other, and the read data is not affected, and the processor consumption can be reduced.

Metadata that is generated during the life of machine learning and artificial intelligence systems are valuable. However, such metadata may be generated subsequent to the time of the data generation, and thus written to the storage device later. By further supporting KV databases on the storage device level, performance may be increased in terms of transfers per second. This is due to the removal of the translation layer in the host, which was previously required for data storage. The removal of the translation layer provides for the removal of two layers of mapping and transaction information. As a result, the number of transactions per second, write amplification, and read amplification increase, while latency decreases. Additionally, future additions of metadata are considered by reserving excess memory at the time of storing the start key value. The future additions are then saved to the reserved memory later.

Various techniques are provided for enabling an iterator (called persistent key-value (KV) iterator) that uses a special key structure embedded with secondary index information for efficient search (or query) of a key-value database (KVDB). In some embodiments, the key structure comprises a plurality of fields categorized into two groups, infrastructure hierarchy fields and object characteristic fields. Keys for the KVDB may be partitioned into one or more levels of hierarchy to generate many smaller key lists for easy search / query of the KVDB. These smaller key lists can be used for concurrent searches / queries.

The embodiment of the invention provides a data processing method and device, equipment, a computer readable storage medium and a product, and the method comprises the steps that search information input by a user is acquired, and the search information comprises identification information and a time range of the user; determining at least one data fragment matched with the search information in a key value database, storing index data of a plurality of users in the key value database, and performing fragment storage on the index data by taking a preset time interval as fragment granularity; and in response to the condition that the at least one data fragment meets a preset adjustment condition, adjusting the fragment granularity of the at least one data fragment, and obtaining the adjusted at least one data fragment, so that the associated parameters of the adjusted data fragments meet the preset condition. Therefore, the data fragments with small capacity can be automatically merged, the storage cost is reduced, the data fragments with large capacity are automatically split, and the problems that the data fragments with large capacity occupy large service bandwidth and are high in delay in the data reading and writing process are solved.

The application provides a lock execution method and device based on a transactional key-value database, equipment and medium, which can be applied to the field of electric digital data processing. The method comprises the following steps: in response to a first transaction operation request for a target resource in a transactional key-value database, reading a first lock key and a first lock value of a distributed lock corresponding to the target resource, wherein the first lock value comprises an instance identifier holding the distributed lock and valid timestamp information of the distributed lock, and the first transaction operation request comprises a target instance identifier and a lease duration of the distributed lock; determining an occupation result of the distributed lock according to the first lock key and the first lock value; in the case that the occupation result represents that the distributed lock is not occupied, updating the instance identifier and the valid timestamp information in the first lock value according to the target instance identifier and the lease duration, so that the target instance executes a transaction operation indicated by the first transaction operation request.

The invention discloses a data query method and device, electronic equipment and a storage medium, and relates to the technical field of computers.The data query method comprises the steps that a query key of to-be-queried data is constructed based on obtained storage features and data identifiers of the to-be-queried data; under the condition that the query key length of the query key is greater than a preset length threshold value, extracting a first prefix key from the query key according to the storage feature, and extracting a second prefix key from a target key according to the storage feature of the target key in the plurality of keys; a first prefix value of a first prefix key and a second prefix value of a second prefix key are respectively determined, a preset key value database is queried through the first prefix value and the second prefix value, and a key value matched with a query key is determined as a query result of data to be queried. And compared with a bitwise query mode, the comparison frequency can be remarkably reduced, and the data query efficiency is greatly improved.