41 results about "Key-value database" patented technology

The present application relates to the technical field of storage, and discloses a data attribute-aware storage system and data management method for a key-value database. The method comprises two parts, i.e., data hotness attribute-based writing and data density attribute-based merging. The data hotness attribute-based writing comprises classifying key-value pairs into cold key-value pairs and hot key-value pairs, to realize separate storage of cold data and hot data by means of a data hotness attribute-based writing strategy. The data density attribute-based merging comprises: when key-value pairs in an SSTable are extracted during a merging operation and form a sorted key-value pair sequence, extracting the key-value pairs one by one, and calculating a binary difference between every two adjacent key-value pairs; and if the difference is greater than a set threshold, stopping filling the current SSTable, and creating a new SSTable. In the present invention, by analyzing data hotness and data density, data having different attributes are classified to different data partitions, thereby reducing I / O amplification caused by repeated reading and writing in a compression process.

The present disclosure relates to a data processing method and device, electronic equipment and storage medium. The method comprises: receiving a data processing request sent by a client, the data processing request being used to indicate the Nth query and carrying a target cursor parameter; the target cursor parameter is an encoding parameter generated in the (N-1)th query, N being a positive integer greater than 1; the target key corresponding to the target cursor parameter is obtained from the mapping relationship between the cursor parameter and the key, the target key indicating the end position of the (N-1)th query in the distributed key-value database; and the (N)th query is performed on the distributed key-value database according to the target key. According to the technical scheme of the present disclosure, the cursor parameter is configured according to the demand of the original command, the corresponding relationship between the cursor parameter and the key is stored, so that the starting position of the query can be accurately analyzed based on the target cursor parameter, and the distributed key-value database has good compatibility.

The present disclosure provides a file processing method and device for a key-value database, equipment, a storage medium and a program product, relates to the technical field of computers, and can be applied to the technical field of finance. The method comprises the following steps: in response to receiving a merging operation instruction, determining a first target layer of a storage component of the key-value database; determining a candidate first sorting string table file stored in the first target layer; determining a candidate second sorting string table file stored in a second target layer according to the candidate first sorting string table file; wherein the second target layer is a storage layer one level lower than the first target layer; performing the merging operation according to the candidate first sorting string table file and the candidate second sorting string table file to obtain a target sorting string table file after merging; and storing the target sorting string table file to the storage component.

Various techniques are provided for enabling an iterator (called persistent key-value (KV) iterator) that uses a special key structure embedded with secondary index information for efficient search (or query) of a key-value database (KVDB). In some embodiments, the key structure comprises a plurality of fields categorized into two groups, infrastructure hierarchy fields and object characteristic fields. Keys for the KVDB may be partitioned into one or more levels of hierarchy to generate many smaller key lists for easy search / query of the KVDB. These smaller key lists can be used for concurrent searches / queries.

The application provides a video recommendation method and system based on a large language model, an electronic device, and a storage medium. At least one target user intention label data is obtained from a key-value database according to a recommendation request initiated by a target user. A first video item set is screened from a streaming media platform according to a first content label set. A second video item set is screened from each first video item according to the first content label set and a first video label set of each first video item. A third video item set is screened from each video item according to target user text feedback data. A fourth video item set is screened from each video item according to each preset recall algorithm, and the fourth video item set is fused with the second video item set and the third video item set to obtain a fifth video item set. A video recommendation list generated according to a sixth video item set obtained by sorting the fifth video item set based on a second content label set is used to recommend videos to the target user.

Metadata generated during the lifetime of machine learning and artificial intelligence systems is valuable. However, such metadata can also be generated after data generation and thus written to the storage device later. Further support for KV databases at the storage device level can improve performance in terms of transfers per second. This is due to the removal of the translation layer on the host, which was previously required for data storage. Removing the translation layer eliminates two layers of mapping and transaction information. This increases the number of transactions per second, write gains, and read gains, while reducing latency.Additionally, future metadata additions are accommodated by reserving excess memory when storing the initial key value. The future additions are then stored in this reserved memory.

This invention discloses a method, device, equipment, and medium for determining the grounding status of a grounding stake. When the contact of the code reading area in the grounding socket comes into contact with the contact of the coding area in the grounding head, the contact coding data corresponding to each conductive contact in the grounding socket is acquired. A preset feature key value database is retrieved according to the contact coding data to determine the corresponding target grounding wire. The target voltage level of the target grounding wire is compared with the standard voltage level corresponding to the associated grounding equipment. If the target voltage level matches the standard voltage level, the grounding status of the grounding stake to be judged is determined to be normal. This invention solves the technical problem that traditional grounding stake installation methods cannot monitor the grounding status of the grounding wire in real time. It enables the determination of the installation or removal status of the grounding wire, identifies the grounding wire number, and determines the grounding status of the grounding stake.

The application discloses a metadata storage method and device based on a distributed key-value database, and relates to the field of data processing.The method comprises the following steps: determining the metadata of an object stored in a bucket, and determining the preset type and key-value pair of the metadata; based on the determined preset type and key-value pair of the metadata, saving the metadata into a corresponding hash table and an ordered list; each bucket in the distributed storage system stores metadata through at least one hash table and at least one ordered list, the hash table is used for storing metadata, and the ordered list is used for storing retrieval information of the metadata.The application reduces the complexity of I / O operation, improves the single-bucket storage scale while ensuring efficiency, and realizes efficient, low-space-usage metadata retrieval applicable to multiple ordered lists.

A method for processing analytical-style queries on a transactional database is provided. A key-value database configured for point lookups and singular transactions is maintained. An analytical-style query request against the key-value database is received and analyzed to determine if it requires accessing more than a threshold amount of data. In response to a threshold, key-value data is separated into range granules covering specified data ranges. For each range granule, a snapshot file comprising key-value pairs at a specific database version and delta files specifying changes during time ranges are generated. The delta files are streamed via change feeds and buffered until accumulating a predetermined amount of mutations. The snapshot and delta files are provided to client execution nodes for parallel processing of the analytical-style query request.

The embodiment of the application discloses a batch anonymous query method, system and device for multiple data sources, which comprises the following steps: a client synchronizes an original polynomial and a polynomial quotient ring of the original polynomial to each of k servers; an n-order polynomial is calculated according to n query key values to obtain a first polynomial; each of the k servers calculates k second polynomials and k interpolation polynomials according to respective key value databases; the client and each client perform operations to obtain a first calculation result and k second calculation results; the client decomposes the first calculation result according to a preset constraint relationship to obtain a first decomposition result; the k servers determine k third calculation results; the client determines a fourth calculation result according to the k third calculation results and the original polynomial, and obtains a query result by traversing the n query key values. The embodiment of the application can improve the data query efficiency.

Systems and methods for a serverless tiebreaker for a shared-nothing architecture are provided. In some examples, a cloud-native service that supports serialization of writes (or write fencing), for example, via atomic operations with persistent locking and / or reservations, is used to support HA mediation instead of a separate server operating as a tiebreaker, thereby reducing costs and complexity as well as increasing availability and durability of the HA mediation functionality. For example, a fast, fully managed, serverless, key-value noSQL database service (e.g., the Amazon DynamoDB) may be used to perform one or more of maintaining the authoritative source of information regarding which node of an HA pair currently represents the primary node for serving data from a particular dataset, persisting HA metadata, and / or assisting in the failover and failback processes.

This invention discloses a near-memory computing method and system for key-value databases based on computing storage devices, belonging to the field of computer information processing technology. The method includes the following steps: collecting access records formed by multiple storage nodes in a distributed computing storage device under a unified time scale, and labeling each access record with a corresponding time offset value to construct an access record set; performing time position callback processing based on the access record set, relocating the access records in time using the time offset value, so that the access records are merged into the same logical time period, forming a unified time period access record set. This invention achieves consistent expression of access data in logical time through unified time scale and time offset processing, improving the accuracy of hotspot identification; simultaneously, through access change splitting and progressive migration control, the migration process is advanced in stages and synchronized with access changes, reducing the concentration of resource consumption and improving system operational stability.

The invention relates to a ZNS SSD operation partition resetting and redistributing method based on LSM tree heat perception, which comprises the following steps: aiming at a key value database based on an LSM tree, proposing a hierarchy-based data heat classification method, dividing data heat into hot data, warm data and cold data, writing different heat data into different partitions of a ZNS SSD, and distributing heat labels for the partitions; according to the heat labels of the partitions, a heat-perceived runtime partition resetting method is provided to dynamically set a runtime partition resetting threshold value, so that the partitions for storing cold data are reset as soon as possible, the cold data are prevented from occupying unused space for a long time, and meanwhile, the partitions for storing hot data are written with data as much as possible. The space utilization rate of the partition for storing the hot data is improved; for an invalid data partition with a low space utilization rate for storing cold data, a partition redistribution method during operation is provided, the unused space of the partition is utilized again, the space utilization rate of the partition is further improved, and the erasing frequency is reduced.

Metadata that is generated during the life of machine learning and artificial intelligence systems are valuable. However, such metadata may be generated subsequent to the time of the data generation, and thus written to the storage device later. By further supporting KV databases on the storage device level, performance may be increased in terms of transfers per second. This is due to the removal of the translation layer in the host, which was previously required for data storage. The removal of the translation layer provides for the removal of two layers of mapping and transaction information. As a result, the number of transactions per second, write amplification, and read amplification increase, while latency decreases. Additionally, future additions of metadata are considered by reserving excess memory at the time of storing the start key value. The future additions are then saved to the reserved memory later.

Various techniques are provided for enabling an iterator (called persistent key-value (KV) iterator) that uses a special key structure embedded with secondary index information for efficient search (or query) of a key-value database (KVDB). In some embodiments, the key structure comprises a plurality of fields categorized into two groups, infrastructure hierarchy fields and object characteristic fields. Keys for the KVDB may be partitioned into one or more levels of hierarchy to generate many smaller key lists for easy search / query of the KVDB. These smaller key lists can be used for concurrent searches / queries.

The embodiment of the invention provides a data processing method and device, equipment, a computer readable storage medium and a product, and the method comprises the steps that search information input by a user is acquired, and the search information comprises identification information and a time range of the user; determining at least one data fragment matched with the search information in a key value database, storing index data of a plurality of users in the key value database, and performing fragment storage on the index data by taking a preset time interval as fragment granularity; and in response to the condition that the at least one data fragment meets a preset adjustment condition, adjusting the fragment granularity of the at least one data fragment, and obtaining the adjusted at least one data fragment, so that the associated parameters of the adjusted data fragments meet the preset condition. Therefore, the data fragments with small capacity can be automatically merged, the storage cost is reduced, the data fragments with large capacity are automatically split, and the problems that the data fragments with large capacity occupy large service bandwidth and are high in delay in the data reading and writing process are solved.

The application provides a lock execution method and device based on a transactional key-value database, equipment and medium, which can be applied to the field of electric digital data processing. The method comprises the following steps: in response to a first transaction operation request for a target resource in a transactional key-value database, reading a first lock key and a first lock value of a distributed lock corresponding to the target resource, wherein the first lock value comprises an instance identifier holding the distributed lock and valid timestamp information of the distributed lock, and the first transaction operation request comprises a target instance identifier and a lease duration of the distributed lock; determining an occupation result of the distributed lock according to the first lock key and the first lock value; in the case that the occupation result represents that the distributed lock is not occupied, updating the instance identifier and the valid timestamp information in the first lock value according to the target instance identifier and the lease duration, so that the target instance executes a transaction operation indicated by the first transaction operation request.

This application relates to the field of key-value database technology, and discloses a method and system for optimizing key-value database read and write operations based on a CSD (Computer-Simplified Data Structure) learning index. The method includes: collecting key-value pairs to be persisted in memory; sorting them by key to form an ordered key sequence; and evaluating the access frequency of key intervals corresponding to the key sequence based on real-time access statistics. For key intervals with access frequency exceeding a preset threshold, training and utilizing the corresponding learning index model to output the predicted offset position of each key-value pair in the key sequence. Based on the predicted offset position, the key-value pairs are written to a storage device to form an ordered string table file, and the physical storage gaps in the predictive active layout are automatically compressed using a computational storage device. The method for constructing the newly generated ordered string table file is dynamically determined based on changes in access frequency, real-time system load, and the merging process of the LSM tree. This application significantly improves the efficiency of hot data reading and maintains a balance between read performance and system resource utilization.

The application provides a NUMA-aware optimization system and method for an NVM key-value database, comprising a proxy module: creating a proxy thread pool for each node of the NUMA; the thread pool converts data operation requests sent from each node to the local node into local access through a proxy mode; a monitoring module: monitoring the running state of the system and collecting running data; the running data comprises running data at the proxy thread level, running data at the proxy node level and running data at the system level; a scheduling module: scheduling the proxy threads in the proxy module. The application proposes a node multi-proxy mode, which can convert any node access request to the local NVM into local access through a certain number of proxy threads, thereby solving the performance decline problem of NVM database cross-end access and high concurrency access under the NUMA architecture.

A system and method for measuring latency is disclosed. In some implementations, the processor may include providing a switch API to process a plurality of microservices involved in a workflow and connected to a distributed in-memory keyvalue database, where the distributed in-memory keyvalue database further may include a plurality of key spaces. In addition, the processor may include managing configuration tools to track the latency of each microservice in the plurality of microservices. The processor may include programing data plane information of the microservices to create programmed data, where programming further may include collecting and analyzing data on the latency of each microservice of the plurality of microservices. Moreover, the processor may include feeding the programmed data into an ASIC.

The invention relates to the technical field of network security, in particular to a key value type database management system vulnerability detection method and system, and the method comprises the steps: collecting business related flow data of a key value type database management system; abstracting a key value database protocol into a metadata graph based on the business related flow data, wherein the metadata graph is a graph structure used for representing a key value dependency relationship; and performing variation on the metadata graph by utilizing a variation strategy to obtain a test case, and testing the key value type database management system by utilizing the test case. According to the method, the active detection and security reinforcement of the key value database kernel depth code path are realized by fusing the coverage guidance and the metadata graph variation to carry out the fuzzy test.

The application discloses a cold and hot data automatic conversion storage method, and applies to the technical field of databases. The method provided by the application comprises the following steps: obtaining a table name of a data table corresponding to a to-be-analyzed SQL script and a first primary key value from a slave database of a target database; if the table name and the first primary key value are successfully obtained, storing the table name and the first primary key value into a preset key-value database; obtaining a primary key value set, wherein a second primary key value of the primary key value set is obtained from the slave database of the target database; judging whether each second primary key value in the primary key value set exists in the key-value database, if not, obtaining first row data corresponding to the second primary key value from a master database of the target database; converting and storing the first row data into a preset non-relational database, and deleting the first row data from the master database of the target database.

Systems and methods for on-demand tokenization of data. Sensitive data elements are tokenized according to a standardized approach, with the resulting token being, or incorporating, a universally unique identifier (UUID). Each payload-token mapping is stored in a distributed key-value database, with the input payload serving as the basis for the key and the generated token stored as the value. The distributed key-value database is accessible to systems within a trusted environment, and also to authorized external applications via a tokenization API service, allowing a variety of applications to request tokenization in real-time, and their data sets to be cross-correlatable.

The application discloses a distributed transaction data consistency solution method and device, electronic equipment and medium. The distributed transaction data consistency solution method comprises the following steps: remotely calling execution data of each database corresponding to different services into a corresponding previous database, and storing each execution data into a first key-value database, wherein each database is arranged according to the order of executing a distributed transaction, and the execution data comprises pre-transaction execution data and post-transaction execution data; when each execution data is stored successfully, submitting a transaction based on the corresponding execution data through each database, and submitting the corresponding transaction data to a second key-value database; judging whether the data in the first key-value database and the second key-value database is consistent, and if yes, the distributed transaction is successfully submitted. The application solves the technical problem of high cost of the distributed transaction data consistency solution.

The invention relates to the technical field of key value databases, and discloses a key value database read-write optimization method and system based on CSD learning type index.The method comprises the steps that to-be-persisted key value pairs in a memory are collected, keys are sequenced to form an ordered key sequence, and based on real-time access statistical information, the access popularity of a key interval corresponding to the key sequence is evaluated; for the key interval of which the access popularity is higher than a preset threshold value, training and utilizing a corresponding learning type index model, and outputting a predicted offset position of each key value pair in the key sequence; writing the key value pair into a storage device based on the predicted offset position to form an ordered character string table file, and automatically compressing a physical storage gap in the predictive active layout by using a computing storage device; and dynamically deciding the construction mode of the newly generated ordered character string table file according to the change of access heat, the real-time load of the system and the merging process of the LSM tree. The hot data reading efficiency is remarkably improved, and the balance between the reading performance and the system resource utilization rate is maintained.

The application relates to the field of databases and discloses a network measurement data management method based on dynamic multi-copy indexes, which comprises the following steps: network measurement data collection, dynamic multi-copy index construction, dynamic multi-copy index query and update and the like; the application creates dynamic multi-copies with hot spot data and stores the dynamic multi-copies in an index structure based on a log-structured merge tree, realizes the function of multi-key query of network measurement data which cannot be completed by traditional key-value storage, improves the storage and query efficiency of network measurement data, reduces the index space overhead caused by query of different fields when a key-value database based on a log-structured merge tree stores network measurement data, and solves the storage bottleneck of existing network measurement data.

The invention relates to the technical field of big data, provides an object storage method, device and equipment based on storage buckets and a medium, is applied to financial and medical health care service scenes, and can allocate independent distributed transaction key value database instances for each storage bucket according to service attribute information of each storage bucket. Not only is the performance bottleneck of a single database instance avoided, but also complete resource isolation among the storage buckets is realized; establishing a corresponding relation mapping table between the storage buckets and the corresponding distributed transaction key value database instances so as to ensure that the meta-information of each storage bucket can only be stored in the corresponding distributed transaction key value database instance; according to the method, the simple storage service meta-information is divided according to the data attributes, the target storage bucket corresponding to each piece of sub-information is determined, the target instance corresponding to each target storage bucket is inquired in the corresponding relation mapping table, each piece of sub-information is written into each corresponding target instance, flexible allocation of resources is achieved, and the resource utilization rate is increased.

This application provides a data processing method and related equipment based on content-addressable storage nodes. The method includes: acquiring target data to be stored and dividing the target data into multiple data blocks; generating corresponding broadcast information for each data block, and sending multiple broadcast information blocks to a distributed key-value database for storage, wherein the distributed key-value database is independent of the content-addressable storage system; acquiring multiple target broadcast information blocks from the distributed key-value database; broadcasting the multiple target broadcast information blocks among multiple other content-addressable storage nodes in the content-addressable storage system, and generating a node identifier routing record corresponding to each target broadcast information block; and sending the node identifier routing record corresponding to each target broadcast information block to the distributed key-value database for storage. This fundamentally improves the performance of a single IPFS node while controlling the node scale.

The application relates to the technical field of key value database, and discloses a key value database task offloading method and system based on a computing storage device, which comprises the following steps: dynamically generating an offloading judgment standard according to metadata information of a host terminal historical merging task; identifying a merging task suitable for offloading to the computing storage device based on the offloading judgment standard; in response to a merging task triggering event, scheduling selection of the computing storage device or the host as an execution device of the merging task suitable for offloading; and screening a sorting string table file participating in the merging task and checking execution resources for the selected execution device, so as to complete task offloading. The application effectively solves the problem of system performance decline caused by simple offloading, realizes efficient utilization of computing storage resources, and improves the overall throughput rate of the system.

The present application belongs to the technical field of block chain, and specifically relates to a chain transaction data rapid analysis method for a UTXO model block chain. The present application mainly comprises the following steps: according to the data structure of the block chain, directly analyzing corresponding original binary data files to obtain each piece of structured chain transaction data of the currency; obtaining the input address and amount of each transaction from the chain transaction data and supplementing the input address and amount to the input field of the original transaction; and finally, supplementing the spending output transaction hash and spending time field to the output field of the original transaction. The present application comprehensively uses a key-value database and a file reading and writing method, can rapidly and synchronously analyze the data of UTXO block chains such as Bitcoin, Bitcoin Cash, Litecoin, Dogecoin and D0GE from the first transaction to the latest transaction, and supplement the key fields in the transaction input and output. The present application can be extended to the rapid analysis process of the chain transaction data of more UTXO block chains, and provides support for data analysis.