Natural language-to-SQL conversion method based on data platform and large language model

Abstract

The present invention relates to the technical field of data processing, and provides a natural language-to-SQL conversion method based on a data platform and a large language model. The method comprises: vectorizing and storing metadata, lineage, metrics, and historical query information; performing intent recognition and vector matching on a user query request; determining a query condition on the basis of the matching result; and constructing an association path on the basis of data lineage; optimizing a table join order using historical queries; and generating an optimal SQL statement by means of index matching. The present invention can realize the intelligent conversion from natural language to efficient SQL, improve query efficiency, and reduce the use threshold of users.

Description

A Natural Language SQL Transformation Method Based on Data Platform and Large Language Model Technical Field

[0001] This invention relates to data processing technology, and more particularly to a natural language SQL conversion method based on a data platform and a large language model. Background Technology

[0002] With the rapid development of big data technology, enterprises have accumulated a large amount of data assets. To fully utilize these data resources for analysis and decision-making, data analysts frequently need to write SQL queries to retrieve the required data. However, for business personnel unfamiliar with SQL syntax, directly writing SQL queries presents significant challenges. Although some natural language to SQL conversion tools are available on the market, these tools often only handle simple single-table queries and still face many challenges when dealing with complex multi-table joins.

[0003] The existing technology has the following main shortcomings:

[0004] Existing natural language to SQL tools lack a deep understanding of business semantics and cannot accurately identify the business rules and indicator definitions implied in the user's query intent, resulting in deviations between the generated SQL query results and the user's actual needs.

[0005] When handling multi-table join queries, existing technologies typically employ fixed table join strategies, failing to fully utilize historical query experience and database index information to optimize query performance, resulting in low query execution efficiency.

[0006] Existing technologies lack effective interaction mechanisms to clarify and supplement query conditions when processing incomplete or ambiguous query requests, which fails to guarantee the accuracy and completeness of the generated SQL query statements and affects the usability of query results. Summary of the Invention

[0007] This invention provides a natural language SQL conversion method based on a data platform and a large language model, which can solve the problems in the prior art.

[0008] A first aspect of the present invention,

[0009] Provides natural language SQL conversion methods based on data platforms and large language models, including:

[0010] The acquired metadata, data lineage, business metrics, and historical query information are converted into vector form and stored in a knowledge vector library via a data service API. Natural language processing (NLP) is used to determine the intent of user-inputted queries, and the extracted intent information is converted into query vectors. These query vectors are then matched for similarity with the metadata vectors, lineage vectors, metric vectors, and historical query vectors in the knowledge vector library. Based on the matching results, the data table, fields, and relationships to be queried are determined. When the matching results indicate incomplete query conditions, supplementary queries are generated based on the business metrics information in the knowledge vector library. User feedback on the supplementary queries is obtained, and the query vectors are updated accordingly. This similarity matching process is repeated until complete query conditions are obtained.

[0011] Based on the query vector and matching results, the structural information and field definitions of the target data table are extracted from the knowledge vector base; an initial association path is constructed according to the data flow path in the data lineage information; based on the data tables in the initial association path, historical query records containing the same data table combination are retrieved from the historical query information; the historical query records are sorted according to the query execution time, and the table join order in the historical query record with the shortest execution time is selected as the optimized table join order;

[0012] Based on the index information in the metadata, index matching is performed on the relevant data filtering conditions and table join conditions, and the query method with the highest index coverage is selected as the index matching result; based on the table association order and the index matching result, a complete SQL query statement containing data filtering conditions, multi-table join relationships and target fields is generated; the SQL query statement is executed and the query result is recorded; the SQL query statement, the query vector, the query execution time, the association path and the query result are updated to the historical query information.

[0013] The metadata information includes the data table structure, field definitions, index information, constraints, data types, and field comments;

[0014] The data lineage information includes data flow paths, upstream and downstream relationships of data objects, and task dependencies;

[0015] The business metrics information includes sales statistics, conversion rate calculation, and inventory turnover rate;

[0016] The historical query information includes SQL query statements, query frequency, and query execution time.

[0017] Based on the matching results, determine the data table, fields, and relationships to be queried; when the matching results show that the query conditions are incomplete, generate supplementary queries based on the business indicator information in the knowledge vector base; obtain user feedback on the supplementary queries and update the query vector, including:

[0018] The query vector is matched with the vectors in the knowledge vector base using a similarity calculation method. The query vector and the vectors in the knowledge vector base are matched in multiple dimensions through cosine similarity calculation, and the matching result is determined according to the set similarity threshold.

[0019] A query graph is constructed based on the matching results. The query graph includes a vertex set representing data tables and an edge set representing the relationships between tables. The connectivity index of the query graph is calculated. The connectivity index characterizes the integrity of the graph through the relationship between the number of edges and the number of vertices. When the connectivity index is less than a preset connectivity threshold, data table nodes that need to be supplemented with additional association conditions are marked.

[0020] The validity of the fields involved in the query is validated. The validity weight of the fields is calculated based on the field relevance, query frequency and null value rate. The field relevance represents the degree of correlation between the field and the business indicators. The query frequency represents the frequency of use of the field in historical queries. The null value rate represents the data quality of the field.

[0021] A hierarchical query strategy is constructed based on the validity weight of the fields. The hierarchical query strategy includes a core business condition confirmation layer, a relationship supplementation layer, and a query constraint refinement layer. Each layer is assigned a different importance coefficient. The query priority is calculated according to the degree of influence of the business indicators on the supplementary conditions, and corresponding supplementary queries are generated according to the query priority.

[0022] Obtain user feedback information on the supplementary query, calculate vector update strength based on the feedback information, the vector update strength is jointly determined by the number of feedbacks, the importance of feedback and the learning rate; enhance and normalize the query vector according to the vector update strength to generate an updated query vector.

[0023] Based on the query vector and matching results, the structural information and field definitions of the target data table are extracted from the knowledge vector base; the initial association path is constructed according to the data flow path in the data lineage information, including:

[0024] The query vector is decomposed into an entity vector and a relation vector. The entity vector and the relation vector are linearly combined based on a preset weight coefficient to generate a target query vector. The relevance score between the target query vector and each data table in the knowledge vector base is calculated. The relevance score is obtained by weighting the entity matching score, relation matching score and field coverage. Data tables with a relevance score greater than the relevance threshold are selected as candidate data tables.

[0025] A structural similarity matrix is ​​constructed for the candidate data tables. The value of each element in the structural similarity matrix is ​​obtained by calculating the ratio of the intersection to the union of the field sets of the corresponding two data tables. Based on the structural similarity matrix, the semantic feature vector of each field is calculated, and the semantic feature vectors are weighted and summed to obtain the field semantic vector. The field semantic correlation degree between the candidate data tables is calculated based on the field semantic vector.

[0026] A directed weighted graph of data lineage is constructed based on the semantic relevance of the fields. The nodes of the directed weighted graph represent candidate data tables, and the edges represent the relationships between tables. The edge weights are calculated by weighting the data dependency strength, data flow frequency, and time relevance. The path cost of all associated paths in the graph is calculated based on the edge weights and the pre-calculated reliability coefficients.

[0027] The path with the lowest path cost is selected as the candidate association path from the association paths. The quality score of the candidate association path is calculated. The quality score is obtained by the ratio of the cumulative product of the weights of all edges in the path to the sum of squared connection costs. At the same time, the redundancy of the candidate association path is calculated. The redundancy is determined based on the ratio of the required edge set to the actual edge set to maintain the data table association relationship.

[0028] When the quality score is greater than a preset quality threshold and the redundancy is less than a preset redundancy threshold, the candidate association path is determined as the final association path; otherwise, the path with the second smallest path cost is selected from the remaining candidate association paths as a new candidate association path, and the quality score calculation and redundancy calculation are repeated until a final association path that meets the quality and redundancy requirements is found.

[0029] Based on the index information in the metadata, index matching is performed on the relevant data filtering conditions and table join conditions, and the query method with the highest index coverage is selected as the index matching result; based on the table join order and the index matching result, a complete SQL query statement containing data filtering conditions, multi-table join relationships, and target fields is generated, including:

[0030] Obtain the index information and query condition set of the data table, calculate the matching degree between each index and the query condition, calculate the index usage cost based on the index storage overhead, access time and maintenance cost, and select the index combination with the lowest index usage cost and index coverage exceeding the preset coverage threshold as the candidate index set;

[0031] The query conditions are analyzed based on the candidate index set, and the selectivity score of the data filtering conditions is calculated. The selectivity score is determined according to the ratio of the amount of data that meets the filtering conditions to the total amount of data in the table. The table join conditions are evaluated, and the join cost is calculated based on the size of the tables involved in the join, the cardinality of the join index, and the join type. The join condition combinations with a join cost value lower than a preset cost threshold are selected.

[0032] A multi-table connection tree is constructed using the connection condition combination. The connection cost is used as the weight of the connection edge. The size of the intermediate result set of different connection sequences is calculated. All connection sequences are sorted by the selection rate and computation cost of the connection steps. The sequence with the minimum total cost is selected as the optimal connection sequence.

[0033] An initial SQL query statement is generated based on the optimal join sequence and candidate index set. The query integrity score of the query statement is calculated. The integrity score includes three dimensions: query field coverage, join condition integrity, and index utilization. At the same time, a performance prediction model is built based on theoretical execution time and IO overhead to calculate the expected execution efficiency of the query statement.

[0034] When the query integrity score is lower than the preset score threshold or the expected execution efficiency is lower than the preset efficiency threshold, an adaptive tuning process is initiated by adjusting the join sequence, updating index selection, and optimizing the query structure until the performance requirements are met or the maximum number of tuning attempts is reached.

[0035] A multi-table join tree is constructed using the aforementioned join condition combinations. The join cost is used as the weight of the join edge. The size of the intermediate result set for different join sequences is calculated. All possible join sequences are sorted based on the selection rate and computational cost of the join steps. The sequence with the minimum total cost is selected as the optimal join sequence, including:

[0036] A connection correlation evaluation model is constructed based on the number of foreign key associations and data relevance between data tables. The connection correlation score between any two data tables is calculated. The connection correlation score is determined by the ratio of the number of foreign key associations to the number of records in the tables and the data relevance coefficient.

[0037] The weight value of the connection edge is calculated based on the connection relevance score. The weight value is obtained by weighting the connection operation cost, input / output overhead and processor computation overhead, and the weight value is used as the weight attribute of the connection tree edge.

[0038] For all connection sequences, the cumulative execution cost is calculated, which includes the cost of connection operations, the cost of intermediate result transmission, and the cost of storage. The sequence execution efficiency score is calculated by combining the selection rate of connection steps and the processing cost.

[0039] Select the sequence with the highest execution efficiency score from all connection sequences as a candidate sequence, and calculate the expected resource consumption value of the candidate sequence, which includes processor utilization, memory usage and network bandwidth utilization.

[0040] When the expected resource consumption exceeds the system capacity threshold, a sequence optimization process is triggered. This process reduces resource consumption by adjusting the connection order, splitting connection steps, or parallelizing the process until the optimal connection sequence that satisfies the system resource constraints is found.

[0041] A second aspect of the present invention provides a natural language SQL conversion system based on a data platform and a large language model, comprising:

[0042] The first unit is used to convert the acquired metadata, data lineage, business indicator, and historical query information into vector form and store it in a knowledge vector library via a data service API; it uses natural language processing technology to determine the intent of the user's input query request and converts the extracted intent information into a query vector; it performs similarity matching between the query vector and the metadata vector, lineage vector, indicator vector, and historical query vector in the knowledge vector library; it determines the data table, fields, and relationships to be queried based on the matching results; when the matching results show that the query conditions are incomplete, it generates supplementary queries based on the business indicator information in the knowledge vector library; it obtains the user's feedback on the supplementary queries and updates the query vector; it repeats the similarity matching process until complete query conditions are obtained.

[0043] The second unit is used to extract the structural information and field definitions of the target data table from the knowledge vector base based on the query vector and matching results; construct an initial association path according to the data flow path in the data lineage information; retrieve historical query records containing the same data table combination from the historical query information based on the data tables in the initial association path; sort the historical query records according to the query execution time, and select the table join order in the historical query record with the shortest execution time as the optimized table join order;

[0044] The third unit is used to perform index matching on the relevant data filtering conditions and table join conditions based on the index information in the metadata information, and select the query method with the highest index coverage as the index matching result; based on the table association order and the index matching result, generate a complete SQL query statement containing data filtering conditions, multi-table join relationships and target fields; execute the SQL query statement and record the query results; and update the SQL query statement, the query vector, the query execution time, the association path and the query results to the historical query information.

[0045] Third aspect of the present invention

[0046] An electronic device is provided, comprising:

[0047] processor;

[0048] Memory used to store processor-executable instructions;

[0049] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0050] Fourth aspect of the present invention,

[0051] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0052] The beneficial effects of this application are as follows:

[0053] This invention vectorizes and stores metadata, data lineage, business metrics, and historical query information. It then utilizes natural language processing (NLP) technology for intent discrimination and similarity matching, enabling it to accurately understand user query intent and quickly locate relevant data tables and fields. When query conditions are incomplete, the system can generate supplementary queries based on business metric information to ensure complete query conditions are obtained, thereby improving query accuracy and efficiency.

[0054] This invention utilizes data lineage information to construct initial association paths and optimizes table association order by analyzing historical query records, effectively reducing the complexity and execution time of data queries. Furthermore, by matching and optimizing index information and selecting the optimal query method, query performance is further improved.

[0055] This invention updates the generated SQL query statements, query vectors, execution time, association paths, and query results to historical query information, forming a continuously optimizing closed-loop system. This method not only continuously improves query efficiency but also provides valuable references for subsequent queries, thereby achieving continuous optimization of query performance and continuous improvement of system intelligence. Attached Figure Description

[0056] Figure 1 is a flowchart illustrating the natural language SQL conversion method based on a data platform and a large language model according to an embodiment of the present invention;

[0057] Figure 2 is a logical diagram of the natural language SQL conversion method based on a data platform and a large language model according to an embodiment of the present invention.

[0058] Figure 3 is a schematic diagram of the structure of the natural language SQL conversion system based on the data platform and large language model according to an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0061] Figures 1 and 2 are respectively a flowchart and a logic diagram of the natural language SQL conversion method based on a data platform and a large language model according to an embodiment of the present invention. The method includes:

[0062] S101. The acquired metadata, data lineage, business indicator, and historical query information are converted into vector form and stored in a knowledge vector library via a data service API; natural language processing technology is used to determine the intent of the user's input query request, and the extracted intent information is converted into a query vector; the query vector is matched with the metadata vector, lineage vector, indicator vector, and historical query vector in the knowledge vector library for similarity; the data table, fields, and relationships to be queried are determined based on the matching results; when the matching results show that the query conditions are incomplete, a supplementary query is generated based on the business indicator information in the knowledge vector library; the user's feedback on the supplementary query is obtained, and the query vector is updated; the similarity matching process is repeated until complete query conditions are obtained;

[0063] S102. Based on the query vector and matching results, extract the structural information and field definitions of the target data table from the knowledge vector base; construct an initial association path according to the data flow path in the data lineage information; based on the data tables in the initial association path, retrieve historical query records containing the same data table combination from the historical query information; sort the historical query records according to the query execution time, and select the table join order in the historical query record with the shortest execution time as the optimized table join order;

[0064] S103. Based on the index information in the metadata information, perform index matching on the relevant data filtering conditions and table join conditions, and select the query method with the highest index coverage as the index matching result; based on the table association order and the index matching result, generate a complete SQL query statement containing data filtering conditions, multi-table join relationships and target fields; execute the SQL query statement and record the query results; update the SQL query statement, the query vector, the query execution time, the association path and the query results to the historical query information.

[0065] In one optional implementation, the metadata information includes the data table structure, field definitions, index information, constraints, data types, and field comments;

[0066] The data lineage information includes data flow paths, upstream and downstream relationships of data objects, and task dependencies;

[0067] The business metrics information includes sales statistics, conversion rate calculation, and inventory turnover rate;

[0068] The historical query information includes SQL query statements, query frequency, and query execution time.

[0069] First, a metadata acquisition module is built to obtain table structure information from various data sources through a database connector. For relational databases, a JDBC driver is used to read the data dictionary and extract basic information such as table name, field name, field type, default value, and whether it is nullable. Simultaneously, DDL statements are parsed to obtain integrity constraint information such as primary key constraints, foreign key constraints, and uniqueness constraints.

[0070] When obtaining data lineage, the ETL job configuration file is first parsed to extract input and output table information from the data processing task. A directed graph model is constructed to store the dependencies between tables as edges, while also recording transformation rules. For stored procedures and views, their definition statements are parsed to identify the source and target tables involved, thus supplementing and refining the lineage graph.

[0071] The collection of business metrics needs to be integrated with specific business scenarios. Taking sales revenue statistics as an example, by configuring the metric scope, the statistical period, aggregation dimensions, calculation rules, and other elements are determined. A sales fact table is established in the data warehouse to record fields such as order amount, payment status, and refund amount. Sales revenue is calculated according to different dimensions (such as date, region, and category) through pre-aggregated views.

[0072] To improve conversion rates, it's necessary to record the status changes at each stage in the user behavior fact table. For example, in an e-commerce scenario, this includes recording key milestones such as browsing products, adding items to the cart, submitting an order, and successful payment. Through behavior sequence analysis, the conversion ratio between each stage can be calculated. Simultaneously, user segmentation can be combined to analyze the conversion performance of users with different characteristics.

[0073] Inventory turnover rate is calculated based on an inventory snapshot table and an inbound / outbound transaction log. The inventory snapshot table records the daily inventory quantity and value, while the inbound / outbound transaction log calculates the sales outbound volume during the period. Combined with purchase receipt records, the timeliness of replenishment can be analyzed, allowing for the optimization of inventory management strategies.

[0074] Historical query information is collected by deploying a monitoring component on the database server. This records execution details such as the original SQL statement, execution timestamp, response time, and number of rows affected. A query pattern library is established to categorize similar SQL statements and analyze the execution frequency of each type of query. Index optimization suggestions are provided for high-frequency queries to improve query performance.

[0075] This application can achieve the following:

[0076] By comprehensively collecting and analyzing metadata, we can accurately grasp the current status of data assets, providing fundamental support for data governance and improving data quality and utilization efficiency. Based on data lineage analysis, we can quickly locate the source of data anomalies, assess the scope of change impact, reduce operational risks, and improve the reliability and maintainability of data processing. Through standardized indicator systems and historical query analysis, we can effectively support business decisions, optimize system performance, improve user experience, and realize the continuous mining and application of data value.

[0077] In one optional implementation, the data table, fields, and relationships to be queried are determined based on the matching results; when the matching results indicate incomplete query conditions, supplementary queries are generated based on business indicator information in the knowledge vector base; obtaining user feedback on the supplementary queries and updating the query vector includes:

[0078] The query vector is matched with the vectors in the knowledge vector base using a similarity calculation method. The query vector and the vectors in the knowledge vector base are matched in multiple dimensions through cosine similarity calculation, and the matching result is determined according to the set similarity threshold.

[0079] A query graph is constructed based on the matching results. The query graph includes a vertex set representing data tables and an edge set representing the relationships between tables. The connectivity index of the query graph is calculated. The connectivity index characterizes the integrity of the graph through the relationship between the number of edges and the number of vertices. When the connectivity index is less than a preset connectivity threshold, data table nodes that need to be supplemented with additional association conditions are marked.

[0080] The validity of the fields involved in the query is validated. The validity weight of the fields is calculated based on the field relevance, query frequency and null value rate. The field relevance represents the degree of correlation between the field and the business indicators. The query frequency represents the frequency of use of the field in historical queries. The null value rate represents the data quality of the field.

[0081] A hierarchical query strategy is constructed based on the validity weight of the fields. The hierarchical query strategy includes a core business condition confirmation layer, a relationship supplementation layer, and a query constraint refinement layer. Each layer is assigned a different importance coefficient. The query priority is calculated according to the degree of influence of the business indicators on the supplementary conditions, and corresponding supplementary queries are generated according to the query priority.

[0082] Obtain user feedback information on the supplementary query, calculate vector update strength based on the feedback information, the vector update strength is jointly determined by the number of feedbacks, the importance of feedback and the learning rate; enhance and normalize the query vector according to the vector update strength to generate an updated query vector.

[0083] The matching process between query vectors and the knowledge vector base first requires vectorization of the input natural language query. A pre-trained language model converts the query text into a high-dimensional vector representation, typically with 768 or 1024 dimensions. The knowledge vector base stores vector representations of standard descriptions of business metrics, related data tables, field definitions, and other information.

[0084] To improve efficiency during vector matching, a block-based computation approach is employed. The knowledge vector base is divided into blocks according to business domains, with each block containing relevant business indicator information. Preliminary matching results are obtained by calculating the cosine similarity between the query vector and the vectors in each block. A similarity threshold of 0.75 is set, and only matches with similarities greater than the threshold are retained.

[0085] When constructing a query graph based on the matching results, the identified data tables are used as vertices of the graph, and the relationships between the tables are used as edges. For example, if the order table and the user table are related by user ID, an edge is added between the corresponding vertices. The connectivity index is obtained by calculating the ratio of the number of edges to the number of vertices. When this index is less than 0.8, it indicates that the query graph is not complete and the relationship conditions need to be supplemented.

[0086] For the fields involved in the query, their effectiveness is evaluated from multiple dimensions. Field relevance is calculated by the co-occurrence frequency with business metrics, query frequency is based on recent query log statistics, and the null value rate is calculated by sampling data. The weighted average of these three metrics is used as the field's effectiveness weight.

[0087] The tiered query strategy divides supplementary conditions into three layers: the core business condition confirmation layer has an importance coefficient of 0.5, the relational supplementation layer has an importance coefficient of 0.3, and the query constraint refinement layer has an importance coefficient of 0.2. Within each layer, query priority is determined based on field validity weights and the degree of business impact. For example, in an order analysis scenario, the order time range is first confirmed, then user-level relational conditions are supplemented, and finally, constraints such as order status are refined.

[0088] After obtaining user feedback, the vector update strength is calculated. The number of feedback responses represents the number of times a user confirms the same question. The importance of feedback is determined based on the user's selection action. The learning rate is initially set to 0.1 and decreases with the number of feedback responses. The corresponding dimensions of the query vector are adjusted according to the update strength to ensure that the updated vector still satisfies the normalization constraint.

[0089] This application can achieve the following:

[0090] By using multi-dimensional vector matching and dynamic threshold adjustment, the accuracy of query intent understanding is improved, query errors caused by ambiguous understanding are reduced, and the system is able to more accurately identify the user's true query needs.

[0091] By adopting a hierarchical query strategy and a priority sorting mechanism, the cost of user interaction is reduced, redundant confirmation processes are avoided, the efficiency of supplementing query conditions is improved, and the completeness and rationality of the supplemented conditions are ensured.

[0092] The vector dynamic update mechanism based on user feedback enables continuous optimization of query understanding capabilities. The system can adaptively adjust query strategies, improving the accuracy of query results and user satisfaction, while reducing the workload of system maintenance.

[0093] In one optional implementation, based on the query vector and matching results, the structural information and field definitions of the target data table are extracted from the knowledge vector base; the initial association path is constructed according to the data flow path in the data lineage information, including:

[0094] The query vector is decomposed into an entity vector and a relation vector. The entity vector and the relation vector are linearly combined based on a preset weight coefficient to generate a target query vector. The relevance score between the target query vector and each data table in the knowledge vector base is calculated. The relevance score is obtained by weighting the entity matching score, relation matching score and field coverage. Data tables with a relevance score greater than the relevance threshold are selected as candidate data tables.

[0095] A structural similarity matrix is ​​constructed for the candidate data tables. The value of each element in the structural similarity matrix is ​​obtained by calculating the ratio of the intersection to the union of the field sets of the corresponding two data tables. Based on the structural similarity matrix, the semantic feature vector of each field is calculated, and the semantic feature vectors are weighted and summed to obtain the field semantic vector. The field semantic correlation degree between the candidate data tables is calculated based on the field semantic vector.

[0096] A directed weighted graph of data lineage is constructed based on the semantic relevance of the fields. The nodes of the directed weighted graph represent candidate data tables, and the edges represent the relationships between tables. The edge weights are calculated by weighting the data dependency strength, data flow frequency, and time relevance. The path cost of all associated paths in the graph is calculated based on the edge weights and the pre-calculated reliability coefficients.

[0097] The path with the lowest path cost is selected as the candidate association path from the association paths. The quality score of the candidate association path is calculated. The quality score is obtained by the ratio of the cumulative product of the weights of all edges in the path to the sum of squared connection costs. At the same time, the redundancy of the candidate association path is calculated. The redundancy is determined based on the ratio of the required edge set to the actual edge set to maintain the data table association relationship.

[0098] When the quality score is greater than a preset quality threshold and the redundancy is less than a preset redundancy threshold, the candidate association path is determined as the final association path; otherwise, the path with the second smallest path cost is selected from the remaining candidate association paths as a new candidate association path, and the quality score calculation and redundancy calculation are repeated until a final association path that meets the quality and redundancy requirements is found.

[0099] First, we introduce the specific implementation of query vector processing and target data table matching. The input query text is segmented and semantically parsed to extract entity and relation information. Entity information includes specific objects such as table names and field names, while relation information includes operation types such as join and group by. The entity information is transformed into a 300-dimensional entity vector, and the relation information into a 100-dimensional relation vector. In practice, the entity vector weight can be set to 0.7, and the relation vector weight to 0.3. The final target query vector is generated through a weighted combination.

[0100] Let's take a specific example. Suppose the query is "to find the association between user order information and payment records". Through word segmentation, we can obtain entity words such as "user", "order", and "payment", and relational words such as "association". After converting these words into vectors, we perform weighted combinations to obtain the target query vector.

[0101] Next, the relevance between the target query vector and each data table in the knowledge base is calculated. The relevance calculation considers three aspects: entity matching degree, relation matching degree, and field coverage. Entity matching degree reflects the similarity of table names and field names; relation matching degree reflects the matching degree of the association types between tables; field coverage indicates the coverage of the fields required for the query in the target table. The weights for these three items are set to 0.4, 0.3, and 0.3 respectively. When the overall score exceeds 0.75, the table is included in the candidate set.

[0102] After obtaining the candidate data tables, it is necessary to analyze the structural similarity between the tables. Structural similarity is calculated by comparing the field sets of the two tables and calculating the ratio of the number of shared fields to the total number of fields. For example, a user table and an order table may both contain a `user_id` field, but their total number of fields is 10 and 8 respectively; therefore, their structural similarity is 0.1.

[0103] Based on the structural similarity matrix, the semantic relationships between fields are further analyzed. Information such as name, type, and annotations are extracted for each field and transformed into a semantic feature vector. In practical applications, a pre-trained word vector model can be used to extract features, and weights are applied according to field importance. Finally, a correlation matrix reflecting the strength of semantic relationships between fields is obtained.

[0104] When constructing a directed weighted graph of data lineage, candidate data tables are used as nodes, and the relationships between tables are used as edges. The weight of each edge needs to consider multiple factors: data dependency strength (0.4%) reflects the referencing relationship between tables; data flow frequency (0.3%) indicates the frequency of data updates; and time relevance (0.3%) reflects the timeliness of the data. The edge weight is obtained by weighting these factors.

[0105] When searching for the optimal association path, the cost value of each possible path is first calculated. The path cost value considers path length, edge weight, and reliability coefficient. The reliability coefficient is derived from historical data quality assessment and reflects the data's reliability. The path with the lowest cost value is selected as the candidate association path.

[0106] Candidate association paths are evaluated for quality, and their quality score and redundancy are calculated. The quality score reflects the usability of the path and is measured by balancing the cumulative effect of edge weights with the connection costs. Redundancy represents the proportion of redundant connections in the path and is determined by comparing the size of the required edge set with the actual edge set.

[0107] A path is considered the final associated path when its quality score exceeds 0.8 and its redundancy is below 0.2. Otherwise, the path with the second lowest cost needs to be re-evaluated until a path that meets the criteria is found.

[0108] This application can achieve the following:

[0109] Vectorization and multi-dimensional matching improve the accuracy of data table associations, enabling precise identification of query intent and finding the most relevant data tables. Structural similarity and semantic association analysis are used to deeply mine the relationships between data tables, avoiding misjudgments caused by relying solely on surface features. Quality assessment and redundancy control mechanisms are introduced to ensure that the generated association paths meet business needs while maintaining high execution efficiency, thus improving query performance and resource utilization.

[0110] In one optional implementation, based on the index information in the metadata, index matching is performed on the relevant data filtering conditions and table join conditions, and the query method with the highest index coverage is selected as the index matching result; based on the table join order and the index matching result, a complete SQL query statement containing data filtering conditions, multi-table join relationships, and target fields is generated, including:

[0111] Obtain the index information and query condition set of the data table, calculate the matching degree between each index and the query condition, calculate the index usage cost based on the index storage overhead, access time and maintenance cost, and select the index combination with the lowest index usage cost and index coverage exceeding the preset coverage threshold as the candidate index set;

[0112] The query conditions are analyzed based on the candidate index set, and the selectivity score of the data filtering conditions is calculated. The selectivity score is determined according to the ratio of the amount of data that meets the filtering conditions to the total amount of data in the table. The table join conditions are evaluated, and the join cost is calculated based on the size of the tables involved in the join, the cardinality of the join index, and the join type. The join condition combinations with a join cost value lower than a preset cost threshold are selected.

[0113] A multi-table connection tree is constructed using the connection condition combination. The connection cost is used as the weight of the connection edge. The size of the intermediate result set of different connection sequences is calculated. All connection sequences are sorted by the selection rate and computation cost of the connection steps. The sequence with the minimum total cost is selected as the optimal connection sequence.

[0114] An initial SQL query statement is generated based on the optimal join sequence and candidate index set. The query integrity score of the query statement is calculated. The integrity score includes three dimensions: query field coverage, join condition integrity, and index utilization. At the same time, a performance prediction model is built based on theoretical execution time and IO overhead to calculate the expected execution efficiency of the query statement.

[0115] When the query integrity score is lower than the preset score threshold or the expected execution efficiency is lower than the preset efficiency threshold, an adaptive tuning process is initiated by adjusting the join sequence, updating index selection, and optimizing the query structure until the performance requirements are met or the maximum number of tuning attempts is reached.

[0116] This invention provides a SQL query generation method based on index matching and table join optimization. The method first obtains the metadata information of the data table, including index information and the set of query conditions. Then, it calculates the matching degree between each index and the query conditions, and considers the index's storage overhead, access time, and maintenance costs to calculate the index usage cost.

[0117] During the index matching process, the system assigns a matching score to each index. For example, suppose there is a user table (user_table) containing fields id, name, age, and city. This table has two indexes: idx_name_age(name, age) and idx_city(city). If the query condition is "WHERE name = 'John' AND age > 30", the idx_name_age index will have a higher matching score because it completely covers the query condition.

[0118] Next, the system calculates the cost of using each index. This takes into account the index's storage space, access speed, and maintenance difficulty. For example, for the idx_name_age index, if it occupies a large amount of storage space (e.g., 1GB) but significantly improves query speed (e.g., reducing query time from 5 seconds to 0.1 seconds), while maintaining a moderate maintenance cost, then its cost of use might be assessed as moderate.

[0119] The system selects the combination of indexes with the lowest index usage cost and an index coverage rate exceeding a preset coverage threshold as the candidate index set. Assuming the preset coverage threshold is 80%, the idx_name_age index might be selected as a candidate index in the above example because it completely covers the query conditions.

[0120] Based on the candidate index set, the system performs in-depth analysis of the query conditions. First, it calculates the selectivity score of the data filtering conditions. The selectivity score is determined by the ratio of the number of records that satisfy the filtering conditions to the total number of records in the table. For example, if the user_table has a total of 1 million records, and 5,000 records satisfy the condition "name='John'AND age>30", then the selectivity score for this filtering condition is 0.005 (5000 / 1000000).

[0121] For table join conditions, the system calculates the join cost based on the size of the tables involved in the join, the cardinality of the join indexes, and the join type. Suppose there is another table, `order_table`, that needs to be joined with `user_table`. If `user_table` has 1 million records and `order_table` has 10 million records, and the join condition is `user_table.id = order_table.user_id`, and `user_id` has an index, then the join cost might be relatively low. The system will filter out join condition combinations with a join cost lower than a preset cost threshold.

[0122] Using the selected combination of join conditions, the system constructs a multi-table join tree. In this tree, each join operation is represented as an edge, and the weight of the edge is the previously calculated join cost. The system calculates the size of the intermediate result set for different join sequences and sorts all possible join sequences based on the selectivity and computational cost of each join step. Finally, the system selects the sequence with the minimum total cost as the optimal join sequence.

[0123] For example, if there are three tables, user_table, order_table, and product_table, that need to be joined, the system might try the following join sequence:

[0124] 1.(user_table JOIN order_table)JOIN product_table;

[0125] 2.(order_table JOIN product_table)JOIN user_table;

[0126] 3.(user_table JOIN product_table)JOIN order_table;

[0127] If the first connection sequence has the lowest total cost after calculation, then it will be selected as the optimal connection sequence.

[0128] Based on the optimal join sequence and candidate index set, the system generates an initial SQL query statement. Then, the system calculates the query integrity score for this query statement. The integrity score comprises three dimensions: query field coverage, join condition integrity, and index utilization.

[0129] The system will check whether the query covers all the required fields, includes all necessary join conditions, and makes full use of available indexes.

[0130] Simultaneously, the system will build a performance prediction model based on theoretical execution time and I / O overhead to calculate the expected execution efficiency of the query statement. This prediction model may take into account factors such as database hardware configuration, current load, and historical query performance.

[0131] If the query completeness score is lower than a preset score threshold (e.g., 0.8) or the expected execution efficiency is lower than a preset efficiency threshold (e.g., expected execution time exceeds 5 seconds), the system will initiate an adaptive tuning process. During this process, the system may try the following optimization strategies:

[0132] 1. Adjust the join sequence: For example, change the original (user_table JOIN order_table) JOIN product_table to (order_table JOIN product_table) JOIN user_table.

[0133] 2. Update index selection: For example, if it is found that there is no index on the order_date field, it may be recommended to create a new index to improve query efficiency.

[0134] 3. Optimize query structure: For example, rewrite subqueries as JOIN, or break down complex conditions into multiple simple conditions.

[0135] The system will continuously repeat this optimization process until the performance requirements are met or the maximum number of tuning attempts (e.g., 10) is reached. After each optimization, the system will re-evaluate the query's completeness score and expected execution efficiency to determine if further optimization is needed.

[0136] Using this method, the system can generate SQL query statements that both meet query requirements and have high execution efficiency.

[0137] This application can achieve the following:

[0138] By comprehensively analyzing index information and query conditions, the optimal index combination can be selected, significantly improving query efficiency. It not only considers the matching degree between the index and the query conditions but also weighs the index's storage overhead, access time, and maintenance costs, thus achieving a good balance between query performance and system resource utilization.

[0139] By constructing a multi-table join tree and evaluating different join sequences, the optimal table join order can be found. This method considers the size of the tables, the selectivity of the join conditions, and the size of the intermediate result sets, effectively reducing the amount of data processing during the query process and thus significantly improving the execution efficiency of complex queries.

[0140] An adaptive tuning mechanism is introduced, which dynamically optimizes query statements based on query completeness scores and expected execution efficiency. This continuous optimization approach not only handles complex query scenarios but also adapts to changes in the database environment, ensuring long-term stability of query performance. In this way, the method can generate efficient SQL queries in various complex database environments, significantly improving overall database performance and user experience.

[0141] In one optional implementation, a multi-table join tree is constructed using the join condition combination, the join cost is used as the weight of the join edge, the size of the intermediate result set for different join sequences is calculated, and all possible join sequences are sorted based on the selection rate and computational cost of the join steps. The sequence with the minimum total cost is selected as the optimal join sequence, including:

[0142] A connection correlation evaluation model is constructed based on the number of foreign key associations and data relevance between data tables. The connection correlation score between any two data tables is calculated. The connection correlation score is determined by the ratio of the number of foreign key associations to the number of records in the tables and the data relevance coefficient.

[0143] The weight value of the connection edge is calculated based on the connection relevance score. The weight value is obtained by weighting the connection operation cost, input / output overhead and processor computation overhead, and the weight value is used as the weight attribute of the connection tree edge.

[0144] For all connection sequences, the cumulative execution cost is calculated, which includes the cost of connection operations, the cost of intermediate result transmission, and the cost of storage. The sequence execution efficiency score is calculated by combining the selection rate of connection steps and the processing cost.

[0145] Select the sequence with the highest execution efficiency score from all connection sequences as a candidate sequence, and calculate the expected resource consumption value of the candidate sequence, which includes processor utilization, memory usage and network bandwidth utilization.

[0146] When the expected resource consumption exceeds the system capacity threshold, a sequence optimization process is triggered. This process reduces resource consumption by adjusting the connection order, splitting connection steps, or parallelizing the process until the optimal connection sequence that satisfies the system resource constraints is found.

[0147] For example, in optimizing multi-table join queries in a database, the first step is to construct a join relevance evaluation model. This involves analyzing foreign key relationships between tables, counting the number of foreign keys in each pair, and calculating the ratio to the total number of records in each table. Simultaneously, considering the distribution characteristics of field values, a data sampling method is used to calculate the relevance coefficient between fields. Taking the order table and the user table as an example, assuming the order table has 10 million records and the user table has 1 million records, and the two tables are linked by user ID with a relevance coefficient of 0.8, a high join relevance score can be obtained.

[0148] After obtaining the join relevance score, the weight values ​​of the join edges are further calculated. Weight calculation needs to consider several factors: the CPU overhead of the join operation itself, the I / O overhead of data reading, and the storage overhead of intermediate results. Taking the join of two large tables as an example, if the input table sizes are 10GB and 20GB respectively, and indexes are built on the join fields, the CPU overhead of the join operation is relatively small, with the main overhead coming from data reading. Based on this, a reasonable weight value can be calculated.

[0149] After constructing the join tree based on the aforementioned weight values, it is necessary to evaluate the execution efficiency of different join sequences. For each possible join sequence, its total execution cost is calculated cumulatively. Taking a three-table join as an example, assuming the sizes of tables A, B, and C are 1GB, 2GB, and 5GB respectively, the intermediate result set size and transmission cost of the sequences "A→B→C" and "B→C→A" will differ significantly. By simulating the selection rate of each join step (e.g., 0.1 indicates that the result set is only 10% of the input data), the execution efficiency of different sequences can be accurately evaluated.

[0150] After obtaining the candidate sequence with the highest execution efficiency score, it is also necessary to estimate its resource consumption. A resource consumption prediction model is established by analyzing historical execution data. For example, for a connection operation with a total data volume of 50GB, it is estimated that it will require 40% CPU resources, 30GB of memory space, and 2Gbps of network bandwidth. If the predicted value exceeds the system threshold (e.g., memory utilization should not exceed 80%), the connection sequence needs to be optimized.

[0151] Optimization methods include adjusting the connection order, splitting large connection steps into multiple smaller steps, or introducing parallel processing mechanisms. Taking excessive memory usage as an example, a large connection operation can be broken down into multiple small batches, with each batch processing a portion of the data, thus keeping peak memory usage within an acceptable range. Through iterative optimization, the optimal connection sequence that satisfies performance requirements without exceeding resource constraints can be obtained.

[0152] This application can achieve the following:

[0153] By constructing an accurate join relevance evaluation model and a comprehensive cost calculation system, the execution efficiency of multi-table join queries is significantly improved, unnecessary resource waste is reduced, and query performance is greatly optimized. Based on resource consumption prediction and dynamic optimization mechanisms, the system can adaptively adjust join strategies, effectively avoiding resource overload issues, improving system stability and reliability, and ensuring smooth operation in high-concurrency scenarios. Employing flexible join sequence optimization schemes, combined with parallel processing and batch execution techniques, efficient processing of large-scale data is achieved, enhancing the system's scalability and adaptability, and meeting the needs of data processing of different scales.

[0154] Figure 3 is a schematic diagram of the structure of the natural language SQL conversion system based on a data platform and a large language model according to an embodiment of the present invention. As shown in Figure 3, the system includes:

[0155] The first unit is used to convert the acquired metadata, data lineage, business indicator, and historical query information into vector form and store it in a knowledge vector library via a data service API; it uses natural language processing technology to determine the intent of the user's input query request and converts the extracted intent information into a query vector; it performs similarity matching between the query vector and the metadata vector, lineage vector, indicator vector, and historical query vector in the knowledge vector library; it determines the data table, fields, and relationships to be queried based on the matching results; when the matching results show that the query conditions are incomplete, it generates supplementary queries based on the business indicator information in the knowledge vector library; it obtains the user's feedback on the supplementary queries and updates the query vector; it repeats the similarity matching process until complete query conditions are obtained.

[0156] The second unit is used to extract the structural information and field definitions of the target data table from the knowledge vector base based on the query vector and matching results; construct an initial association path according to the data flow path in the data lineage information; retrieve historical query records containing the same data table combination from the historical query information based on the data tables in the initial association path; sort the historical query records according to the query execution time, and select the table join order in the historical query record with the shortest execution time as the optimized table join order;

[0157] The third unit is used to perform index matching on the relevant data filtering conditions and table join conditions based on the index information in the metadata information, and select the query method with the highest index coverage as the index matching result; based on the table association order and the index matching result, generate a complete SQL query statement containing data filtering conditions, multi-table join relationships and target fields; execute the SQL query statement and record the query results; and update the SQL query statement, the query vector, the query execution time, the association path and the query results to the historical query information.

[0158] A third aspect of the embodiments of the present invention,

[0159] An electronic device is provided, comprising:

[0160] processor;

[0161] Memory used to store processor-executable instructions;

[0162] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0163] Fourth aspect of the present invention,

[0164] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0165] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

A natural language SQL conversion method based on a data platform and a large language model is characterized by: include: The acquired metadata, data lineage, business metrics, and historical query information are converted into vector form and stored in the knowledge vector library through the data service API. Natural language processing (NLP) technology is used to determine the intent of user-inputted queries, and the extracted intent information is converted into a query vector. The query vector is then matched for similarity with metadata vectors, lineage vectors, indicator vectors, and historical query vectors in the knowledge vector base. Based on the matching results, the data table, fields, and relationships to be queried are determined. When the matching results indicate that the query conditions are incomplete, supplementary queries are generated based on business indicator information in the knowledge vector base. User feedback on the supplementary queries is obtained, and the query vector is updated. The similarity matching process is repeated until complete query conditions are obtained. Based on the query vector and matching results, the structural information and field definitions of the target data table are extracted from the knowledge vector base; an initial association path is constructed according to the data flow path in the data lineage information; based on the data tables in the initial association path, historical query records containing the same data table combination are retrieved from the historical query information; the historical query records are sorted according to the query execution time, and the table join order in the historical query record with the shortest execution time is selected as the optimized table join order; Based on the index information in the metadata, the relevant data filtering conditions and table join conditions are matched with the index, and the query method with the highest index coverage is selected as the index matching result. Based on the table join order and the index matching result, generate a complete SQL query statement containing data filtering conditions, multi-table join relationships, and target fields; execute the SQL query statement and record the query results. Update the SQL query statement, the query vector, the query execution time, the associated path, and the query results to the historical query information. The method according to claim 1, characterized in that, The metadata information includes the data table structure, field definitions, index information, constraints, data types, and field comments; The data lineage information includes data flow paths, upstream and downstream relationships of data objects, and task dependencies; The business metrics information includes sales statistics, conversion rate calculation, and inventory turnover rate; The historical query information includes SQL query statements, query frequency, and query execution time. The method according to claim 1, characterized in that, The data table, fields, and relationships to be queried are determined based on the matching results; when the matching results show that the query conditions are incomplete, supplementary queries are generated based on the business indicator information in the knowledge vector base. Obtaining user feedback on the supplementary query and updating the query vector includes: The query vector is matched with the vectors in the knowledge vector base using a similarity calculation method. The query vector and the vectors in the knowledge vector base are matched in multiple dimensions through cosine similarity calculation, and the matching result is determined according to the set similarity threshold. A query graph is constructed based on the matching results. The query graph includes a vertex set representing data tables and an edge set representing the relationships between tables. The connectivity index of the query graph is calculated. The connectivity index characterizes the integrity of the graph through the relationship between the number of edges and the number of vertices. When the connectivity index is less than a preset connectivity threshold, data table nodes that need to be supplemented with additional association conditions are marked. The validity of the fields involved in the query is validated. The validity weight of the fields is calculated based on the field relevance, query frequency and null value rate. The field relevance represents the degree of correlation between the field and the business indicators. The query frequency represents the frequency of use of the field in historical queries. The null value rate represents the data quality of the field. A hierarchical query strategy is constructed based on the validity weight of the fields. The hierarchical query strategy includes a core business condition confirmation layer, a relationship supplementation layer, and a query constraint refinement layer. Each layer is assigned a different importance coefficient. The query priority is calculated according to the degree of influence of the business indicators on the supplementary conditions, and corresponding supplementary queries are generated according to the query priority. Obtain user feedback information on the supplementary query, calculate vector update strength based on the feedback information, the vector update strength is jointly determined by the number of feedbacks, the importance of feedback and the learning rate; enhance and normalize the query vector according to the vector update strength to generate an updated query vector. The method according to claim 1, characterized in that, Based on the query vector and matching results, the structural information and field definitions of the target data table are extracted from the knowledge vector base; the initial association path is constructed according to the data flow path in the data lineage information, including: The query vector is decomposed into an entity vector and a relation vector. The entity vector and the relation vector are linearly combined based on a preset weight coefficient to generate a target query vector. The relevance score between the target query vector and each data table in the knowledge vector base is calculated. The relevance score is obtained by weighting the entity matching score, relation matching score and field coverage. Data tables with a relevance score greater than the relevance threshold are selected as candidate data tables. A structural similarity matrix is ​​constructed for the candidate data tables. The value of each element in the structural similarity matrix is ​​obtained by calculating the ratio of the intersection to the union of the field sets of the corresponding two data tables. Based on the structural similarity matrix, the semantic feature vector of each field is calculated, and the semantic feature vectors are weighted and summed to obtain the field semantic vector. The field semantic correlation degree between the candidate data tables is calculated based on the field semantic vector. A directed weighted graph of data lineage is constructed based on the semantic relevance of the fields. The nodes of the directed weighted graph represent candidate data tables, and the edges represent the relationships between tables. The edge weights are calculated by weighting the data dependency strength, data flow frequency, and time relevance. The path cost of all associated paths in the graph is calculated based on the edge weights and the pre-calculated reliability coefficients. The path with the lowest path cost is selected as the candidate association path from the association paths. The quality score of the candidate association path is calculated. The quality score is obtained by the ratio of the cumulative product of the weights of all edges in the path to the sum of squared connection costs. At the same time, the redundancy of the candidate association path is calculated. The redundancy is determined based on the ratio of the required edge set to the actual edge set to maintain the data table association relationship. When the quality score is greater than a preset quality threshold and the redundancy is less than a preset redundancy threshold, the candidate association path is determined as the final association path; otherwise, the path with the second smallest path cost is selected from the remaining candidate association paths as a new candidate association path, and the quality score calculation and redundancy calculation are repeated until a final association path that meets the quality and redundancy requirements is found. The method according to claim 1, characterized in that, Based on the index information in the metadata, the relevant data filtering conditions and table join conditions are matched with the index, and the query method with the highest index coverage is selected as the index matching result. Based on the table join order and the index matching results, a complete SQL query statement is generated that includes data filtering conditions, multi-table join relationships, and target fields, including: Obtain the index information and query condition set of the data table, calculate the matching degree between each index and the query condition, calculate the index usage cost based on the index storage overhead, access time and maintenance cost, and select the index combination with the lowest index usage cost and index coverage exceeding the preset coverage threshold as the candidate index set; The query conditions are analyzed based on the candidate index set, and the selectivity score of the data filtering conditions is calculated. The selectivity score is determined according to the ratio of the amount of data that meets the filtering conditions to the total amount of data in the table. The table join conditions are evaluated, and the join cost is calculated based on the size of the tables involved in the join, the cardinality of the join index, and the join type. The join condition combinations with a join cost value lower than a preset cost threshold are selected. A multi-table connection tree is constructed using the connection condition combination. The connection cost is used as the weight of the connection edge. The size of the intermediate result set of different connection sequences is calculated. All connection sequences are sorted by the selection rate and computation cost of the connection steps. The sequence with the minimum total cost is selected as the optimal connection sequence. An initial SQL query statement is generated based on the optimal join sequence and candidate index set. The query integrity score of the query statement is calculated. The integrity score includes three dimensions: query field coverage, join condition integrity, and index utilization. At the same time, a performance prediction model is built based on theoretical execution time and IO overhead to calculate the expected execution efficiency of the query statement. When the query integrity score is lower than the preset score threshold or the expected execution efficiency is lower than the preset efficiency threshold, an adaptive tuning process is initiated by adjusting the join sequence, updating index selection, and optimizing the query structure until the performance requirements are met or the maximum number of tuning attempts is reached. The method according to claim 1, characterized in that, A multi-table join tree is constructed using the aforementioned join condition combinations. The join cost is used as the weight of the join edge. The size of the intermediate result set for different join sequences is calculated. All possible join sequences are sorted based on the selection rate and computational cost of the join steps. The sequence with the minimum total cost is selected as the optimal join sequence, including: A connection correlation evaluation model is constructed based on the number of foreign key associations and data relevance between data tables. The connection correlation score between any two data tables is calculated. The connection correlation score is determined by the ratio of the number of foreign key associations to the number of records in the tables and the data relevance coefficient. The weight value of the connection edge is calculated based on the connection relevance score. The weight value is obtained by weighting the connection operation cost, input / output overhead and processor computation overhead, and the weight value is used as the weight attribute of the connection tree edge. For all connection sequences, the cumulative execution cost is calculated, which includes the cost of connection operations, the cost of intermediate result transmission, and the cost of storage. The sequence execution efficiency score is calculated by combining the selection rate of connection steps and the processing cost. Select the sequence with the highest execution efficiency score from all connection sequences as a candidate sequence, and calculate the expected resource consumption value of the candidate sequence, which includes processor utilization, memory usage and network bandwidth utilization. When the expected resource consumption exceeds the system capacity threshold, a sequence optimization process is triggered. This process reduces resource consumption by adjusting the connection order, splitting connection steps, or parallelizing the process until the optimal connection sequence that satisfies the system resource constraints is found. A natural language SQL conversion system based on a data platform and a large language model, used to implement the method as described in any one of claims 1-6, characterized in that it includes: The first unit is used to convert the acquired metadata, data lineage, business indicator, and historical query information into vector form and store it in a knowledge vector library via a data service API; it uses natural language processing technology to determine the intent of the user's input query request and converts the extracted intent information into a query vector; it performs similarity matching between the query vector and the metadata vector, lineage vector, indicator vector, and historical query vector in the knowledge vector library; it determines the data table, fields, and relationships to be queried based on the matching results; when the matching results show that the query conditions are incomplete, it generates supplementary queries based on the business indicator information in the knowledge vector library; it obtains the user's feedback on the supplementary queries and updates the query vector; it repeats the similarity matching process until complete query conditions are obtained. The second unit is used to extract the structural information and field definitions of the target data table from the knowledge vector base based on the query vector and matching results; construct an initial association path according to the data flow path in the data lineage information; retrieve historical query records containing the same data table combination from the historical query information based on the data tables in the initial association path; sort the historical query records according to the query execution time, and select the table join order in the historical query record with the shortest execution time as the optimized table join order; The third unit is used to perform index matching on the relevant data filtering conditions and table join conditions based on the index information in the metadata information, and select the query method with the highest index coverage as the index matching result; based on the table association order and the index matching result, generate a complete SQL query statement containing data filtering conditions, multi-table join relationships and target fields; execute the SQL query statement and record the query results; and update the SQL query statement, the query vector, the query execution time, the association path and the query results to the historical query information. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images