593 results about "Column (database)" patented technology

A database system generates a prompt for an LLM or other machine learning (ML) model to narrow the search space to highly relevant information about a database. A distinct instance of a classifier, a clustering algorithm, or a topic modeling model can be trained based on information from ML automation within the database system, respectively for each column or table in the database. Model instances can then be used during generative LLM inferencing to identify relevant sources of data to answer the user's question. Thus, the prompt generation combines ML automation and other ML models or an LLM for topic modeling and schema description.

The invention provides a Text2SQL generation method and system driven by a large language model, and the method comprises the steps: recognizing key entities and relationships in a natural language question through the large language model, and carrying out the construction to obtain a query graph; constructing a hierarchical graph representation model based on the database mode of the target database; searching candidate sub-image sets which have the same structure as the query graph and are in semantic association with the query graph from the constructed graph representation model; and constructing a structured Prompt template based on the candidate sub-image set and the user question, generating an initial SQL statement, and outputting a final query result after grammar and semantic detection correction. According to the method, a traditional database is converted into the graph knowledge base, so that the SQL generation accuracy and performance are improved; guiding the large model to generate an SQL (Structured Query Language) meeting business requirements through knowledge graph storage and retrieval table and column information; and the accuracy of the SQL is continuously optimized by utilizing a self-adaptive feedback mechanism, so that the method is particularly suitable for a large-scale complex query scene, and self-improvement and customization of the model are realized.

The invention provides an SQL statement analysis method and device, equipment and a storage medium, and relates to the technical field of databases. According to the specific scheme, the method comprises the steps that SQL statements and metadata thereof are obtained, wherein the metadata comprises table names, column names and predicates of the SQL statements; target statistical information corresponding to the metadata is obtained from a preset statistical information base, and the target statistical information comprises table statistical information, column statistical information and index statistical information. And loading the statistical information to a test environment database to obtain a simulation database corresponding to the SQL statement. And based on the statistical information, determining an execution plan of the SQL statement, and executing the SQL statement in the simulation database according to the execution plan to obtain a statement analysis result. The method can effectively solve the problem of analysis distortion caused by the distribution difference of test and production environment data, and can obtain a more accurate SQL analysis result without complex data preparation.

The invention provides a knowledge graph construction method based on a low-code development platform, and belongs to the technical field of knowledge graph construction. The method comprises the following steps: counting and carding metadata and business entity structures in a low-code development platform, and formulating a mapping specification for mapping from a relational database to semantic data; obtaining a target relational database in the configuration file, extracting table data and column data in the target relational database, and converting the table data and the column data into entity class parameters and attribute class parameters in the knowledge graph based on a mapping specification; extracting an unstructured data source related to the target business from the low-code development platform, and converting the unstructured data source into structured data; and associating the structured data with the entity class parameters and the attribute class parameters, and storing the associated data in the knowledge graph. Automatic mapping and storage from the relational database to the knowledge graph are realized, so that the data utilization rate and the business intelligence level are improved.

The invention relates to a Text-to-SQL (Structured Query Language) complex query statement generation method and system based on a pre-trained large model. The method comprises the following steps: acquiring an input sequence comprising a user question and dictionary interpretation of table names and column names in a database; respectively inputting the user question and the dictionary explanation into a Sension-BERT model, obtaining Top-K explanations most related to the user question, and obtaining an enhanced input sequence according to the Top-K explanations most related to the user question; encoding the enhanced input sequence to generate an enhanced semantic vector, and constructing a hierarchical database mode pattern according to the enhanced semantic vector; inputting the hierarchical database mode graph into a hypergraph neural network model to update node features, and obtaining optimized node features; and obtaining a structured query statement according to the optimized node features. According to the method, complex association and multi-level query logic among multiple tables in a complex query scene can be better processed, and the performance in the complex query scene is improved.

Data structures and methods are described to provide a unique extendable group identifier for efficient aggregation query processing using multiple grouping keys. A method comprises retrieving a database query comprising an aggregate function of a selected column from a database table, grouped by a plurality of columns. The method further comprises maintaining a plurality of hash tables that map column values to grouping keys and identify a maximum grouping key. The method further comprises updating a plurality of bitmasks that define per-column grouping key bit positions, capable of storing the maximum grouping key, within a combined index value. The method further comprises allocating a result array sized according to the plurality of bitmasks. The method further comprises using the plurality of bitmasks and the plurality of hash tables to determine the combined index value to apply the aggregate function on the selected column for each row in the database table.

The invention relates to the technical field of natural language processing, and discloses an intelligent slot filling and mode linking method based on RAG and M-Schema, and the method comprises the steps: rapidly matching a preset FAQ knowledge base through an RAG-FAQ module, and extracting a predefined medical intention and a slot structure; if the matching fails, starting an NLU module to perform intention classification and slot position extraction; performing entity linking and normalization on the slot value by utilizing an RAG-medical field knowledge base context enhancement module, performing multi-dimensional verification in combination with M-Schema metadata of a target database, and outputting a matching column and a confidence score; aiming at uncertainty types such as medical term ambiguity and business rule conflict, a clarification problem is dynamically generated, user feedback is received, SQL query conforming to business rules is generated after slot position information is corrected, and finally the SQL query is transmitted to a medical database execution module to output an analysis result. According to the invention, the availability and accuracy of the general framework in the scene facing the hospital management field can be improved.

Techniques for priority-based masking policy selection in a database environment are described. Masking policies are defined and attached to columns of relational data for particular users or roles. The attachment of a masking policy to a column includes a user-specified priority value. When multiple policies could apply to a particular query, the conflict can be easily resolved and understood by use of the priority values, for example, by selecting a candidate policy having a highest priority value.

The invention discloses a method and a system for generating an SQL (Structured Query Language) query based on a natural language question. The method comprises the following steps: converting mode information of a target database and the natural language question into semantic vector representation; based on the semantic vector representation, a simplified database mode set related to SQL query is screened out through an attention mechanism; based on the natural language problem, the simplified database mode set and preset database constraint information, generating an SQL structural skeleton, and filling specific elements of the SQL structural skeleton to form a preliminary SQL query; and performing dynamic correction and verification on the initial SQL query by utilizing a large language model, and outputting a final SQL query. According to the method, association mining between user query and a database mode is effectively enhanced by utilizing a context-aware cross-encoder mechanism, and an implicit corresponding relation between a natural language problem and a database table / column can be more accurately identified and utilized.

The invention discloses a time sequence database architecture, a data writing method and electronic equipment, and the time sequence database architecture comprises a distributed storage module which enables data of the same equipment to be continuously distributed in a physical storage layer based on an equipment dimension fragmentation strategy; the independent index layer is decoupled from the data storage module and maintains a mapping relationship between the equipment identifier and the corresponding physical fragment through a metadata structure; the migration module is used for performing rolling updating on the hot data layer; and the query engine module is used for positioning the target physical fragment according to the equipment identifier and executing predicate push-down calculation on the compressed data. The data writing method comprises the following steps: receiving a data writing request, and analyzing a device identifier in the request; positioning a target physical fragment according to the equipment identifier in cooperation with a Hash algorithm; and adding the data to the column type storage file in the corresponding physical fragment according to a timestamp sequence, and synchronously updating the metadata mapping relationship of the independent index layer. According to the method, the data query performance and query efficiency can be improved, and the data storage cost is reduced.

A computerized method is provided for using large language models to integrate data from multiple sources including technical metadata (e.g. column and table names) with a dictionary of standard abbreviations employed in the metadata, representative data from the columns rows of the table, a business glossary of terms in the data, and representative queries used to interrogate the data, to create a non-technical or natural language description of business value of the table and its columns and data.

Methods, systems, and devices for data management are described. A method may include obtaining a new snapshot of a relational database of a source computing environment, comparing the new snapshot of the relational database to a prior snapshot of the relational database, updating, in response to the comparison indicating that data of a row of a relational database table changed between the prior snapshot and the new snapshot, a field of a second versioning control column of a corresponding row of a backup database table of the set of backup database tables to indicate a version of the new snapshot, and adding, in response to the comparison indicating that the data of the row of the relational database table changed, an additional row to the backup database table, where a field of a first versioning control column of the additional row includes an indication of the version of the new snapshot.

Some embodiments include information retrieval through query history insights by accessing query history of a first user, processing the query history of the first user using a first machine learning model to identify naming characteristics of the query history specific for the first user, and enriching a database comprising data associated with the first user with the identified naming characteristics of the query history. The system receives a new search query in natural language from the first user, processes the new search query in the natural language using a second machine learning model to identify embeddings within the new search query, identifies one or more recommended tables and corresponding columns, and causes display of the recommended tables and corresponding columns for each of the recommended tables by a user device of the first user.

The system obtains messages including multiple properties, such as a sender, a receiver, and content. The system creates a database to store the messages and presents the database in a user interface including a table including rows and columns. The database can be customized by including a custom column among the multiple columns. The system customizes the user interface by creating custom views of the database. The system receives an indication to create a view of the multiple messages, where the indication to create the view includes a criterion to select a subset of messages from the messages. The system creates the view including multiple layouts configured to visually provide important information associated with the subset of messages.

The invention discloses an HTAP database query optimization method and device and a storage medium. The query performance is remarkably improved by combining the advantages of row storage and column storage. Comprising the steps of analyzing SQL query to generate a logic query plan; evaluating the execution cost of each logic operator in different storage modes by using a mixed cost model, wherein hyper-parameters of the mixed cost model are optimized through Bayesian search; selectively adopting a dynamic programming algorithm or a greedy algorithm to generate a hybrid physical query plan with the minimum execution cost according to the table number; and executing the plan to obtain a query result. In addition, an MCTS algorithm is further utilized, the optimal column subset is selected in the preset memory budget limit based on the query historical record and loaded into the memory column storage, and the subsequent query performance is further optimized. The problem that row storage and column storage in an existing HTAP system cannot work cooperatively is solved, operator-level mixed execution is achieved, and meanwhile memory resource utilization is optimized.

The invention discloses a system and a method for supporting multi-version concurrency control in a column storage database, belongs to the technical field of databases, and aims to solve the technical problem of how to provide efficient and flexible concurrency control and data operation management. Comprising a multi-version management module for creating a transaction ID and a timestamp for each transaction and maintaining a version linked list for each data segment in a column storage database; the operation optimization module is used for performing data operation on the data in the column storage database based on the submitted transaction, performing operation according to a comparison result of a version number and a transaction timestamp when the data operation is executed, and storing the data before modification into a revocation buffer area; the transaction conflict processing module is used for detecting and processing transaction conflicts based on a preset conflict processing rule; the storage space management module divides the storage of the column storage database into table storage and local storage; and the version linked list management module is used for deleting the version information which is no longer referenced by any active transaction based on the transaction state and the version information.

The invention provides an NL2SQL database table dynamic screening method and system based on multi-strategy fusion. The method comprises the steps that a preliminary candidate set is generated, a semantic encoder is used for encoding a user natural language problem and table and column information in a database into vectors, the semantic similarity is calculated, a preset number of tables are screened out from a whole database table based on the similarity, and a preliminary candidate table set is formed; performing deep analysis and reordering, performing multi-dimensional analysis on the tables in the preliminary candidate table set, and calculating a graph relationship compactness score and a content correlation score of each table; and performing fusion decision, inputting the semantic similarity score, the graph relationship compactness score and the content correlation score of each table into a fusion model to obtain a final comprehensive score of each table, and screening out a final correlation table set based on the final comprehensive score. The method has high accuracy and high recall rate, the generation speed and response efficiency of the NL2SQL system are improved, meanwhile, the robustness is high, and the practicability and generalization ability of the system are improved.

The invention discloses a row-level data permission authorization method and device, equipment and a medium, and relates to the technical field of computers. Comprising the following steps: registering basic data into a preset database; configuring a corresponding field management user with a field management authority for the preset service field; creating or updating a record in a user data permission table based on the user permission application to determine a permission relationship between the user identifier and the entity data; registering a service data table and a mapping relation between an entity column and an entity type in the service data table in a preset service field based on a user instruction of a field management user; and dynamically generating a permission filtering condition based on the query SQL and the user identifier so as to generate an SQL statement with permission filtering, and querying corresponding service data from the service data table based on the mapping relationship so as to generate a corresponding query result based on the service data. Therefore, the efficiency and flexibility of line-level data permission authorization can be improved on the premise of ensuring data security.

An automated method is provided for comparing a first database table and a second database table with each other to produce a table of differences therebetween. The first database table and the second database table include one or more subsets of data that are intended to match each other. The first database table and the second database table each include multiple rows, each row having multiple columns. A first set of indicia is generated comprising indicia for a subset of the columns of each of the multiple rows of a subset of the first database table, and a second set of indicia is generated comprising indicia for a subset of the columns of each of the multiple rows of a subset of the second database table. The second set of indicia are generated without regard to an original order of the columns in the second database. The indicia in the first set of indicia are compared with the indicia in the second set of indicia on a row-by-row basis, wherein the comparing occurs without regard to an original order of the rows in the second database table. Any indicia in the first set of indicia that are not present in the second set of indicia are identified, thereby identifying a first set of mismatched indicia. Likewise, any indicia in the second set of indicia that are not present in the first set of indicia are identified, thereby identifying a second set of mismatched indicia. The table of differences is populated with the identified indicia. The second database table may be repaired by using at least one of the first set of mismatched indicia and the second set of mismatched indicia.

The invention discloses a query acceleration system of a column storage database and a self-adaptive index construction method, and the system comprises a data compression module which is used for compressing data of the column storage database; the input end of the parallel processing module is electrically connected with the output end of the data compression module; the input end of the query load analysis module is electrically connected with the output end of the parallel processing module; the input end of the self-adaptive index module is electrically connected with the output end of the query load analysis module and the output end of the parallel processing module; and the input end of the machine learning prediction module is electrically connected with the output end of the query load analysis module. The system comprises a data compression module and the like, through multi-module cooperation and a five-step index construction method, the static index problem is solved, and the park monitoring data query efficiency and the system adaptability are improved.

The invention discloses a multi-modal retrieval method and device, and relates to the technical field of multi-modal retrieval, and the method comprises the steps: obtaining a query vector corresponding to a user query text, and carrying out the query in a vector database according to the query vector, and generating a query result; vector codes corresponding to the multi-level content block structures corresponding to the multiple multi-modal table documents are stored in the vector database; the multi-level content block structure of any multi-modal table document comprises an atomic layer, a semantic layer and a relation layer; the atomic block comprises target element content and structure information of any cell in the multi-modal table document; the semantic block comprises aggregation content corresponding to any row, any column or any whole table in the multi-modal table document; the relation block comprises a text for describing the incidence relation between the multiple atomic blocks and the incidence relation between the multiple semantic blocks; the structure information comprises row and column indexes corresponding to each cell and a cross-modal association relationship among the multi-modal elements. According to the method, the multi-modal retrieval accuracy can be improved.

The invention provides an SQL query interpretation generation method and system based on a heterogeneous graph, and belongs to the technical field of databases. The method comprises the steps that after data cleaning is conducted on an SQL query, a source tokens is extracted; constructing an SQL abstract syntax tree based on the source tokens, wherein the abstract syntax tree comprises keyword nodes, table name nodes, column name nodes and query condition nodes of SQL query; based on the abstract syntax tree, introducing six different types of edges among father and child nodes, brother nodes and leaf nodes, and constructing a heterogeneous graph of SQL query; inputting the heterogeneous graph into an improved heterogeneous graph neural network, and extracting heterogeneous features of SQL query through a two-stage aggregation process based on an attention mechanism; the learned SQL features are fused into an encoder-decoder framework, and natural language interpretation of SQL query is generated; according to the method provided by the invention, heterogeneous features of SQL query can be fused and extracted, and more accurate SQL explanation is generated.

The embodiment of the invention provides a large-model-based natural language query analysis and database field matching method, which comprises the following steps of: performing intention analysis and entity recognition on natural language query of a user based on a large language model (LLM), and extracting a potential keyword set; for each extracted keyword, binding an entity type through an STAM mechanism, and constructing an entity-type mapping relationship; a column description vector library is constructed, efficient retrieval is achieved in combination with an approximate nearest neighbor ANN algorithm, and results of semantic vector matching, editing distance matching and traceability enhancement matching are combined through a multi-strategy fusion matching mechanism to form a final candidate column set; a database metadata interface is connected, information is analyzed, a basic structure is constructed, semantic enhancement field description is generated, and output is organized in a tetrad structure; and reversely deducing the affiliated table through column matching, judging the table structure value by combining the information density in the table and the connectivity with other tables, eliminating redundant tables, and optimizing database representation.

The invention relates to a method and a device for realizing synchronization of an oversized transaction to an openGauss database based on Debezium. The method comprises the following steps: configuring the size of a file cache slice; starting an incremental synchronization service, and obtaining a data change record (EVENT) by analyzing an Oracle source library log; dividing the EVENT into an independent file cache queue according to the transaction ID, and triggering dynamic segmentation (the storage capacity exceeds the standard or is overtime); when the service is abnormal, positioning a site through an index to realize breakpoint resume; after the transactions are submitted, the file cache queue is read in sequence and pushed to the downstream, and incomplete transactions are rolled back when the transactions are abnormal; events containing LOB / XML columns are combined into DML statements, and the DML statements are issued according to a primary key strategy; and converting the downstream system into a standard format for the consumer to subscribe. According to the scheme, memory caching is replaced by file caching, OOM is avoided, dynamic segmentation and breakpoint resume improve the large transaction processing capacity, the LOB / XML merging strategy guarantees the data consistency, general transaction optimization is supported, and the oversized transaction synchronization efficiency and stability are remarkably improved.

The embodiment of the invention provides an SQL statement processing method and device, electronic equipment and a computer readable storage medium, and relates to the field of databases. The method comprises the steps of obtaining at least one SQL statement to be processed; based on at least one of an SPJG rule, a public expression recognition rule and a time column recognition rule, extracting a reference SQL clause with at least one of business semantics, reusability and time from each SQL statement to be processed; inputting each reference SQL clause and a preset cue word into a large language model to obtain at least one first SQL clause; the preset cue word is used for indicating to screen out a first SQL clause from each reference SQL clause by combining natural language description information of each reference SQL based on at least one of multiplexing potential, calculation overhead, semantic rationality and incremental support. According to the embodiment of the invention, the SQL clauses which are more suitable for being packaged into the virtual table can be selected.

The invention discloses a user ad-hoc query report realization method and device based on a large language model, and relates to the technical field of natural language processing. The method comprises the following steps: constructing an ad hoc query report system based on the large language model, wherein the ad hoc query report system comprises a semantic analysis module, an analysis engine module and a query statistics module; a natural language query statement input by a user is input into a semantic analysis module to be analyzed through a fine-tuned large language model, a structured query condition set and a structured query result index column set are output and input into an analysis engine module, and according to permission information of the current user, through pre-constructed index metadata and SQL semantic analysis rules, the query result index column set is analyzed, and the query result index column set is analyzed. Generating a query SQL statement; and inputting the query SQL statement into a query statistics module, connecting a target database to execute query operation to obtain a query result set, and performing data formatting processing on the query result set to obtain a final query result. By adopting the method, complex query requirements can be met.

The invention discloses a method and device for realizing integrity protection of sensitive data of a database, and belongs to the technical field of database security, and the method comprises the following steps: identifying sensitive data of a key column of a data table in a target database through an encryption system, and recording the sensitive data as key data; integrity protection is configured for the key data, and a digital signature is applied; the method comprises the following steps: calculating an abstract value of key data by using an SM3 hash algorithm, and calculating related sensitive information of the current key data and an MAC value of a system key at the same time; the database service system accesses the key data through the proxy connection of the encryption system, data modification or new data insertion is carried out, that is, a service user carries out data entry and storage; the encryption system compares whether the MAC value of the related sensitive information of the current key data is consistent with the MAC value of the related sensitive information of the last key data, and returns a comparison result to the system; and displaying a data state corresponding to a comparison result returned by the system on a system web page.

A database system operates by: generating an initial query operator execution flow for a corresponding query for execution that includes an extend operation for execution serially before an aggregation operation indicating grouping performance of a corresponding aggregation by new column values of a new column generated via the extend operation via performance of a corresponding extend function of at least one input column; converting the initial query operator execution flow into an updated query operator execution flow semantically equivalent to the initial query operator execution flow based on: generating an updated aggregation operation to group by input column values of the at least one input column, and pushing the updated aggregation operation for execution serially before the extend operation; and executing the corresponding query by applying the updated query operator execution flow based on: executing the updated aggregation operation upon a plurality of rows to generate a plurality of aggregation values for a plurality of row groups via grouping by the at least one input column, and executing the extend operation after performing the updated aggregation operation by generating the new column from the at least one input column.

Embodiments of the present disclosure provide a method and apparatus for database query, an electronic device, and a medium. The method includes: determining, in response to receiving a structured query language (SQL) query for a database, a query condition and a target column of the SQL query that relates to a number of distinct values (NDV) estimation; selecting, from sampled data of the database, data of the target column that meets the query condition; generating feature data of the selected data of the target column; and determining an estimated NDV of the target column based on the feature data using a model for NDV estimation.

The invention belongs to the technical field of databases, and particularly discloses an index column fusion storage method and system.The method comprises the steps that a target column is determined, the target column is a data column configured with a Pstort index in a column storage table, and the character length of the field type of original data of the target column is larger than that of an index position identifier; the column storage table is a data table adopting a column storage mode, and the index position identifier is used for indicating the position of the original data in the index file; and configuring the field type of the target column as an index position identifier type, so that each row in the target column stores an index position identifier. By means of the method and device, high-performance and high-compression index column data fusion is achieved, and the occupation amount of the data files and the index files in the disk space can be reduced under the condition that the query efficiency is considered.