478 results about "Column (database)" patented technology

A database system generates a prompt for an LLM or other machine learning (ML) model to narrow the search space to highly relevant information about a database. A distinct instance of a classifier, a clustering algorithm, or a topic modeling model can be trained based on information from ML automation within the database system, respectively for each column or table in the database. Model instances can then be used during generative LLM inferencing to identify relevant sources of data to answer the user's question. Thus, the prompt generation combines ML automation and other ML models or an LLM for topic modeling and schema description.

The invention provides a Text2SQL generation method and system driven by a large language model, and the method comprises the steps: recognizing key entities and relationships in a natural language question through the large language model, and carrying out the construction to obtain a query graph; constructing a hierarchical graph representation model based on the database mode of the target database; searching candidate sub-image sets which have the same structure as the query graph and are in semantic association with the query graph from the constructed graph representation model; and constructing a structured Prompt template based on the candidate sub-image set and the user question, generating an initial SQL statement, and outputting a final query result after grammar and semantic detection correction. According to the method, a traditional database is converted into the graph knowledge base, so that the SQL generation accuracy and performance are improved; guiding the large model to generate an SQL (Structured Query Language) meeting business requirements through knowledge graph storage and retrieval table and column information; and the accuracy of the SQL is continuously optimized by utilizing a self-adaptive feedback mechanism, so that the method is particularly suitable for a large-scale complex query scene, and self-improvement and customization of the model are realized.

The invention provides an SQL statement analysis method and device, equipment and a storage medium, and relates to the technical field of databases. According to the specific scheme, the method comprises the steps that SQL statements and metadata thereof are obtained, wherein the metadata comprises table names, column names and predicates of the SQL statements; target statistical information corresponding to the metadata is obtained from a preset statistical information base, and the target statistical information comprises table statistical information, column statistical information and index statistical information. And loading the statistical information to a test environment database to obtain a simulation database corresponding to the SQL statement. And based on the statistical information, determining an execution plan of the SQL statement, and executing the SQL statement in the simulation database according to the execution plan to obtain a statement analysis result. The method can effectively solve the problem of analysis distortion caused by the distribution difference of test and production environment data, and can obtain a more accurate SQL analysis result without complex data preparation.

The invention relates to the technical field of natural language processing, and discloses an intelligent slot filling and mode linking method based on RAG and M-Schema, and the method comprises the steps: rapidly matching a preset FAQ knowledge base through an RAG-FAQ module, and extracting a predefined medical intention and a slot structure; if the matching fails, starting an NLU module to perform intention classification and slot position extraction; performing entity linking and normalization on the slot value by utilizing an RAG-medical field knowledge base context enhancement module, performing multi-dimensional verification in combination with M-Schema metadata of a target database, and outputting a matching column and a confidence score; aiming at uncertainty types such as medical term ambiguity and business rule conflict, a clarification problem is dynamically generated, user feedback is received, SQL query conforming to business rules is generated after slot position information is corrected, and finally the SQL query is transmitted to a medical database execution module to output an analysis result. According to the invention, the availability and accuracy of the general framework in the scene facing the hospital management field can be improved.

Techniques for priority-based masking policy selection in a database environment are described. Masking policies are defined and attached to columns of relational data for particular users or roles. The attachment of a masking policy to a column includes a user-specified priority value. When multiple policies could apply to a particular query, the conflict can be easily resolved and understood by use of the priority values, for example, by selecting a candidate policy having a highest priority value.

The invention discloses a method and a system for generating an SQL (Structured Query Language) query based on a natural language question. The method comprises the following steps: converting mode information of a target database and the natural language question into semantic vector representation; based on the semantic vector representation, a simplified database mode set related to SQL query is screened out through an attention mechanism; based on the natural language problem, the simplified database mode set and preset database constraint information, generating an SQL structural skeleton, and filling specific elements of the SQL structural skeleton to form a preliminary SQL query; and performing dynamic correction and verification on the initial SQL query by utilizing a large language model, and outputting a final SQL query. According to the method, association mining between user query and a database mode is effectively enhanced by utilizing a context-aware cross-encoder mechanism, and an implicit corresponding relation between a natural language problem and a database table / column can be more accurately identified and utilized.

A computerized method is provided for using large language models to integrate data from multiple sources including technical metadata (e.g. column and table names) with a dictionary of standard abbreviations employed in the metadata, representative data from the columns rows of the table, a business glossary of terms in the data, and representative queries used to interrogate the data, to create a non-technical or natural language description of business value of the table and its columns and data.

Methods, systems, and devices for data management are described. A method may include obtaining a new snapshot of a relational database of a source computing environment, comparing the new snapshot of the relational database to a prior snapshot of the relational database, updating, in response to the comparison indicating that data of a row of a relational database table changed between the prior snapshot and the new snapshot, a field of a second versioning control column of a corresponding row of a backup database table of the set of backup database tables to indicate a version of the new snapshot, and adding, in response to the comparison indicating that the data of the row of the relational database table changed, an additional row to the backup database table, where a field of a first versioning control column of the additional row includes an indication of the version of the new snapshot.

Some embodiments include information retrieval through query history insights by accessing query history of a first user, processing the query history of the first user using a first machine learning model to identify naming characteristics of the query history specific for the first user, and enriching a database comprising data associated with the first user with the identified naming characteristics of the query history. The system receives a new search query in natural language from the first user, processes the new search query in the natural language using a second machine learning model to identify embeddings within the new search query, identifies one or more recommended tables and corresponding columns, and causes display of the recommended tables and corresponding columns for each of the recommended tables by a user device of the first user.

The system obtains messages including multiple properties, such as a sender, a receiver, and content. The system creates a database to store the messages and presents the database in a user interface including a table including rows and columns. The database can be customized by including a custom column among the multiple columns. The system customizes the user interface by creating custom views of the database. The system receives an indication to create a view of the multiple messages, where the indication to create the view includes a criterion to select a subset of messages from the messages. The system creates the view including multiple layouts configured to visually provide important information associated with the subset of messages.

The invention provides an NL2SQL database table dynamic screening method and system based on multi-strategy fusion. The method comprises the steps that a preliminary candidate set is generated, a semantic encoder is used for encoding a user natural language problem and table and column information in a database into vectors, the semantic similarity is calculated, a preset number of tables are screened out from a whole database table based on the similarity, and a preliminary candidate table set is formed; performing deep analysis and reordering, performing multi-dimensional analysis on the tables in the preliminary candidate table set, and calculating a graph relationship compactness score and a content correlation score of each table; and performing fusion decision, inputting the semantic similarity score, the graph relationship compactness score and the content correlation score of each table into a fusion model to obtain a final comprehensive score of each table, and screening out a final correlation table set based on the final comprehensive score. The method has high accuracy and high recall rate, the generation speed and response efficiency of the NL2SQL system are improved, meanwhile, the robustness is high, and the practicability and generalization ability of the system are improved.

The invention discloses a row-level data permission authorization method and device, equipment and a medium, and relates to the technical field of computers. Comprising the following steps: registering basic data into a preset database; configuring a corresponding field management user with a field management authority for the preset service field; creating or updating a record in a user data permission table based on the user permission application to determine a permission relationship between the user identifier and the entity data; registering a service data table and a mapping relation between an entity column and an entity type in the service data table in a preset service field based on a user instruction of a field management user; and dynamically generating a permission filtering condition based on the query SQL and the user identifier so as to generate an SQL statement with permission filtering, and querying corresponding service data from the service data table based on the mapping relationship so as to generate a corresponding query result based on the service data. Therefore, the efficiency and flexibility of line-level data permission authorization can be improved on the premise of ensuring data security.

The invention discloses a query acceleration system of a column storage database and a self-adaptive index construction method, and the system comprises a data compression module which is used for compressing data of the column storage database; the input end of the parallel processing module is electrically connected with the output end of the data compression module; the input end of the query load analysis module is electrically connected with the output end of the parallel processing module; the input end of the self-adaptive index module is electrically connected with the output end of the query load analysis module and the output end of the parallel processing module; and the input end of the machine learning prediction module is electrically connected with the output end of the query load analysis module. The system comprises a data compression module and the like, through multi-module cooperation and a five-step index construction method, the static index problem is solved, and the park monitoring data query efficiency and the system adaptability are improved.

The invention discloses a multi-modal retrieval method and device, and relates to the technical field of multi-modal retrieval, and the method comprises the steps: obtaining a query vector corresponding to a user query text, and carrying out the query in a vector database according to the query vector, and generating a query result; vector codes corresponding to the multi-level content block structures corresponding to the multiple multi-modal table documents are stored in the vector database; the multi-level content block structure of any multi-modal table document comprises an atomic layer, a semantic layer and a relation layer; the atomic block comprises target element content and structure information of any cell in the multi-modal table document; the semantic block comprises aggregation content corresponding to any row, any column or any whole table in the multi-modal table document; the relation block comprises a text for describing the incidence relation between the multiple atomic blocks and the incidence relation between the multiple semantic blocks; the structure information comprises row and column indexes corresponding to each cell and a cross-modal association relationship among the multi-modal elements. According to the method, the multi-modal retrieval accuracy can be improved.

The embodiment of the invention provides a large-model-based natural language query analysis and database field matching method, which comprises the following steps of: performing intention analysis and entity recognition on natural language query of a user based on a large language model (LLM), and extracting a potential keyword set; for each extracted keyword, binding an entity type through an STAM mechanism, and constructing an entity-type mapping relationship; a column description vector library is constructed, efficient retrieval is achieved in combination with an approximate nearest neighbor ANN algorithm, and results of semantic vector matching, editing distance matching and traceability enhancement matching are combined through a multi-strategy fusion matching mechanism to form a final candidate column set; a database metadata interface is connected, information is analyzed, a basic structure is constructed, semantic enhancement field description is generated, and output is organized in a tetrad structure; and reversely deducing the affiliated table through column matching, judging the table structure value by combining the information density in the table and the connectivity with other tables, eliminating redundant tables, and optimizing database representation.

The invention relates to a method and a device for realizing synchronization of an oversized transaction to an openGauss database based on Debezium. The method comprises the following steps: configuring the size of a file cache slice; starting an incremental synchronization service, and obtaining a data change record (EVENT) by analyzing an Oracle source library log; dividing the EVENT into an independent file cache queue according to the transaction ID, and triggering dynamic segmentation (the storage capacity exceeds the standard or is overtime); when the service is abnormal, positioning a site through an index to realize breakpoint resume; after the transactions are submitted, the file cache queue is read in sequence and pushed to the downstream, and incomplete transactions are rolled back when the transactions are abnormal; events containing LOB / XML columns are combined into DML statements, and the DML statements are issued according to a primary key strategy; and converting the downstream system into a standard format for the consumer to subscribe. According to the scheme, memory caching is replaced by file caching, OOM is avoided, dynamic segmentation and breakpoint resume improve the large transaction processing capacity, the LOB / XML merging strategy guarantees the data consistency, general transaction optimization is supported, and the oversized transaction synchronization efficiency and stability are remarkably improved.

The embodiment of the invention provides an SQL statement processing method and device, electronic equipment and a computer readable storage medium, and relates to the field of databases. The method comprises the steps of obtaining at least one SQL statement to be processed; based on at least one of an SPJG rule, a public expression recognition rule and a time column recognition rule, extracting a reference SQL clause with at least one of business semantics, reusability and time from each SQL statement to be processed; inputting each reference SQL clause and a preset cue word into a large language model to obtain at least one first SQL clause; the preset cue word is used for indicating to screen out a first SQL clause from each reference SQL clause by combining natural language description information of each reference SQL based on at least one of multiplexing potential, calculation overhead, semantic rationality and incremental support. According to the embodiment of the invention, the SQL clauses which are more suitable for being packaged into the virtual table can be selected.

The invention discloses a user ad-hoc query report realization method and device based on a large language model, and relates to the technical field of natural language processing. The method comprises the following steps: constructing an ad hoc query report system based on the large language model, wherein the ad hoc query report system comprises a semantic analysis module, an analysis engine module and a query statistics module; a natural language query statement input by a user is input into a semantic analysis module to be analyzed through a fine-tuned large language model, a structured query condition set and a structured query result index column set are output and input into an analysis engine module, and according to permission information of the current user, through pre-constructed index metadata and SQL semantic analysis rules, the query result index column set is analyzed, and the query result index column set is analyzed. Generating a query SQL statement; and inputting the query SQL statement into a query statistics module, connecting a target database to execute query operation to obtain a query result set, and performing data formatting processing on the query result set to obtain a final query result. By adopting the method, complex query requirements can be met.

A database system operates by: generating an initial query operator execution flow for a corresponding query for execution that includes an extend operation for execution serially before an aggregation operation indicating grouping performance of a corresponding aggregation by new column values of a new column generated via the extend operation via performance of a corresponding extend function of at least one input column; converting the initial query operator execution flow into an updated query operator execution flow semantically equivalent to the initial query operator execution flow based on: generating an updated aggregation operation to group by input column values of the at least one input column, and pushing the updated aggregation operation for execution serially before the extend operation; and executing the corresponding query by applying the updated query operator execution flow based on: executing the updated aggregation operation upon a plurality of rows to generate a plurality of aggregation values for a plurality of row groups via grouping by the at least one input column, and executing the extend operation after performing the updated aggregation operation by generating the new column from the at least one input column.

The invention belongs to the technical field of databases, and particularly discloses an index column fusion storage method and system.The method comprises the steps that a target column is determined, the target column is a data column configured with a Pstort index in a column storage table, and the character length of the field type of original data of the target column is larger than that of an index position identifier; the column storage table is a data table adopting a column storage mode, and the index position identifier is used for indicating the position of the original data in the index file; and configuring the field type of the target column as an index position identifier type, so that each row in the target column stores an index position identifier. By means of the method and device, high-performance and high-compression index column data fusion is achieved, and the occupation amount of the data files and the index files in the disk space can be reduced under the condition that the query efficiency is considered.

The invention discloses a traceable data conversion method and system based on declarative rules, and relates to the technical field of data processing. The method comprises the following steps: acquiring a conversion rule file written by adopting a PTL (Program Transfer Language), and defining a mapping relationship between a target data column and a source data field in a declarative manner; analyzing the rule file to generate an operation instruction set, and processing the source data line by line based on the instruction set to generate a target data file; and meanwhile, outputting the traceability metadata, and recording the corresponding relationship between the target data and the source fields and the rule statements in a machine readable form. Flexible and accurate conversion between various data sources and target formats is achieved, complexity of manual operation and script development is avoided, data processing efficiency and reliability are improved, a complete traceable mechanism is established, transparency and compliance of the data exchange process are enhanced, the method is suitable for conversion of various formats such as Excel, databases, XML and JSON, and the method has wide application prospects. The method has universality and application prospects.

The invention relates to the technical field of query characterization, and discloses a database query feature vector generation method, which comprises the following steps of: obtaining a connection bitmap and a database mode graph of a database queried by a target query statement; determining a database mode sub-graph related to the target query statement according to the database mode graph; encoding the database mode sub-graph through a preset gating graph neural network to generate a mode item vector; and determining a query feature vector corresponding to the target query statement according to the connection bitmap and the mode item vector. According to the method, by obtaining the database mode graph and the connection bitmap, the structure association information between the table and the column in the database and the predicate condition features of the query can be synthesized, the semantic association represented by the features is enhanced based on the coding capability of the graph neural network, and the expression capability of the query feature vector for query intention and data association is effectively improved.

The invention discloses a multi-source time series data query method and device, and relates to the technical field of database management, and the method comprises the steps: defining virtual tables according to business requirements, and establishing a mapping relation between a time series data column label in each virtual table and a table name, a column label and a physical storage position of a pre-stored original table; when a query request is received, determining an original table corresponding to each time sequence data column label in the virtual table according to the mapping relation; an independent table scanning operator is generated for an original table, so that after each table scanning operator concurrently extracts time sequence data in each original table, each table scanning operator concurrently extracts the time sequence data in each original table and transmits the time sequence data to a virtual table scanning operator through an exchange operator; and merging the time sequence data in each original table according to a timestamp through the virtual table scanning operator to obtain a query result containing the time sequence data of the plurality of original tables.

The scanning range of an inner table can be limited in the join processing irrespective of the existence of search conditions or a partition key, and the performance of the join processing can be improved. There is a column range index representing a range of values stored in a table. When a join result of a current stage is an outer table related to a join of a next stage in a join of each stage of the join processing, a database management apparatus identifies whether there is a possibility that an inner table related to a join of a next stage includes a hit record in an inner table related to a join of a current stage based on a column range index corresponding to a join column of a current stage and a column range index corresponding to a join column of a next stage.

The invention discloses a Text-to-SQL (Structured Query Language) method based on a large language model. The method comprises a database mode linking module, a context information enriching module, a dynamic decomposition SQL (Structured Query Language) generation module, an execution feedback optimization module and a selection module based on self-consistency. Performing multi-path recall screening on related tables, columns and value fields in a database through a mode link; the mode information of the database is expressed through context information enrichment, key information is extracted, and a question is re-described; aiming at the problem of different complexity, different SQL generation schemes are designed, and a plurality of candidate SQL statements are generated; checking and correcting grammar and semantic errors existing in the SQL statement by executing feedback optimization; and through voting and a binary selection model, selecting the most accurate SQL query as a final SQL. The method can effectively solve the problems of enterprise data increase, difficulty in complex SQL generation and the like, improves the conversion accuracy and efficiency, enhances the adaptability to different database modes, and has important practical value.

The embodiment of the invention provides a data query method, a database system, equipment, a storage medium and a program product. The method comprises the following steps: when a query request is received, determining a first column table storage mode of a to-be-queried table according to a table name of the to-be-queried table carried in the query request; obtaining a column type storage file of the to-be-queried table; determining a second column storage table mode used when the column storage file is generated according to mode version information correspondingly recorded by the column storage file; obtaining first column data of the to-be-queried column from the column type storage file according to the second column table storage mode and the column name of the to-be-queried column carried in the query request; according to the first column table storage mode and the second column table storage mode, converting the first column data into second column data adaptive to the first column table storage mode; and determining a query result of the query request according to the second column of data. According to the technical scheme provided by the embodiment of the invention, the influence of database mode change on database service can be reduced.

A method, system, and computer program product are configured to: extract, from metadata of a table in a database, a table name abbreviation of the table and plural column name abbreviations of plural columns in the table; automatically generate a prompt for an artificial intelligence (AI) model, the prompt including the table name abbreviation, the plural column name abbreviations, and an instruction to expand the table name abbreviation and the plural column name abbreviations; provide the prompt to the AI model using a single call to the AI model; receive an output generated by the AI model based on the prompt; and process the output.

The invention provides a heterogeneous database query method, device and equipment based on large model driving. The method comprises the following steps: receiving a natural language query instruction of a user for a target service; performing semantic analysis on the query instruction by using a large language model to obtain a query keyword set, performing mapping processing on the keyword set and a heterogeneous database mode corresponding to a target service, and determining a condition column and a target column corresponding to the query instruction; determining a query task type corresponding to the query instruction according to a data type of a target column; constructing a semantic operator execution path according to the query task type; according to the keyword set and the condition column and the target column obtained through matching, corresponding semantic operators are called to execute the semantic operator execution path so as to obtain a query result corresponding to the query instruction, and by means of the technical scheme, the efficiency and reliability of heterogeneous database query can be improved.

Systems and methods are disclosed for converting natural language queries to a query instruction set for searching a data warehouse. To generate a query instruction set from a natural language query, a system iteratively uses a generative artificial intelligence (AI) model and database query tools to generate a query instruction set in a stepwise manner. The system and generative AI model do not require a priori knowledge of data table contents in the data warehouse, which may include sensitive information. In addition, the system does not require access to the data warehouse to generate the query instruction set. Instead, the system is implemented to use structure information from the data warehouse, including table lists (such as table names) and table format information (such as column names) of tables in the data warehouse, and the generative AI model is a generally trained model to generate the query instruction set.

The invention relates to the technical field of database query optimization and distributed system data processing, and discloses an intelligent routing optimization method and system for multi-table association query, and the method comprises the steps: a query topology analysis unit analyzes an object relation mapping configuration file to construct a query topological graph; the environment parameter detection unit collects real-time state data of the server cluster based on the query topological graph, wherein the real-time state data comprises a pre-estimated cache hit rate, associated sparseness and a target table data line number estimation value; the routing strategy decision-making unit compares the real-time state data with a preset threshold value, and selects a cache-driven step-by-step query strategy, a column-type compression transmission strategy or an asynchronous intermediate table unloading strategy; and the dynamic execution control unit executes data acquisition and assembly operation according to a selected strategy. According to the method, the problems of low database computing resource utilization rate and application server memory overflow caused by the adoption of a static strategy in a traditional framework are solved, and the system response speed and the service availability of multi-table association query are effectively improved.