Data analysis method and system, and storage medium

Abstract

The present application relates to the field of natural language processing, and discloses a data analysis method and system, and a storage medium. The method of the present application comprises: acquiring a natural language inputted by a user; constructing a semantic data model for converting the natural language into a semantic query statement; constructing a semantic parser on the basis of the semantic data model and custom rules; inputting the natural language into the semantic parser for parsing to obtain a semantic query statement; inputting the semantic query statement into a semantic corrector for correction to obtain a target semantic query statement; converting the target semantic query statement into a structured query statement by means of a semantic translator; and inputting the structured query statement into a database for query to obtain a query result.

Description

A data analysis method, system, and storage medium Technical Field

[0001] This application relates to the field of natural language processing, and more particularly to a data analysis method, system, and storage medium. Background Technology

[0002] With the rapid development of science and technology, user query needs are becoming increasingly diverse and complex. Users expect to obtain the information they need quickly and accurately through query systems. However, traditional query methods often rely on strict grammatical rules and keyword matching, which to some extent limits users' query freedom. When users input query information in natural language, even slight differences in grammar or logic can cause the query system to fail to correctly understand the user's intent, leading to query failures or inaccurate results.

[0003] To address this challenge, existing technologies have incorporated natural language processing (NLP) techniques. This technology can convert user-input natural language into a format that the query system can understand, thereby effectively reducing the query failure rate and improving the accuracy of the returned results.

[0004] However, natural language processing (NLP) technology still has shortcomings in semantic analysis and understanding. Especially when dealing with complex or ambiguous expressions, existing NLP techniques often struggle to accurately capture the true intent of user input. This not only affects the accuracy of query results but also limits the application of query systems and the user experience. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a data analysis method, system, and storage medium.

[0006] The technical solution provided in this application is described below:

[0007] The first aspect of this application provides a data analysis method, the method comprising:

[0008] Obtain natural language input from the user;

[0009] Build the initial data model;

[0010] The natural language is segmented using a pre-configured industry knowledge base and a WordPiece word segmenter to construct a prompt;

[0011] The prompt is encoded using a pre-trained model to obtain the encoded vector for each token;

[0012] The encoding vector of the first classification token output by the pre-trained model is used as the representation of the entire sentence;

[0013] Based on the sentence representation, all the encoded vectors are processed to obtain a local semantic representation;

[0014] By identifying entities and attributes in the natural language and mapping them to schema elements in the database, a mapping relationship can be obtained;

[0015] Based on the mapping relationship and the local semantic representation, a semantic data model is constructed;

[0016] A semantic parser is constructed by combining the semantic data model with custom rules.

[0017] The natural language is input into the semantic parser for parsing to obtain a semantic query statement;

[0018] The semantic query statement is input into the semantic corrector for correction in order to obtain the target semantic query statement;

[0019] The target semantic query statement is converted into a structured query statement using a semantic translator;

[0020] The structured query statement is input into the database for querying to obtain query results; the step of constructing a semantic data model based on the mapping relationship and the local semantic representation includes:

[0021] The mapping relationship is initialized based on the context information of the currently input natural language query.

[0022] Adjust the weights of the database schema elements based on the context information;

[0023] Construct an initial semantic graph, which includes entity nodes, relation nodes, operation nodes, entity relation edges, condition relation edges, and operation relation edges;

[0024] For each node, a comprehensive node feature representation is generated by combining the local semantic representation, the database metadata information, and the user's historical query behavior.

[0025] The features of each edge in the initial semantic graph are initialized to obtain a comprehensive edge feature representation;

[0026] The integrated node feature representation and the integrated edge feature representation are embedded into the initial semantic graph, and the initial semantic graph is dynamically updated based on the sink algorithm for implementing the connector to obtain the input semantic graph;

[0027] The input semantic graph is fed into a pre-constructed initial graph neural network for training. The information of neighboring nodes is aggregated through graph convolution operations in each layer of the initial graph neural network, and the state of each node is updated by combining the features of each node.

[0028] The initial graph neural network was optimized using the Adam optimizer to obtain a converged semantic data model.

[0029] Optionally, the prompt is encoded using a pre-trained model to obtain an encoded vector for each token, including:

[0030] The prompt is converted into the input format of the pre-trained model to obtain input_ids and attention_mask;

[0031] The input_ids and attention_mask are input into the pre-trained model to obtain the encoding vector for each token.

[0032] Optionally, by identifying entities and attributes in the natural language and mapping the entities and attributes to schema elements in the database to obtain mapping relationships, including:

[0033] Extract key entities from the natural language;

[0034] Query the database schema information to obtain the schema elements of the database;

[0035] Match the key entities with the schema elements to obtain the mapping relationship.

[0036] Optionally, the semantic query statement is input into a semantic corrector for correction to obtain the target semantic query statement, including:

[0037] Construct an initial semantic corrector;

[0038] Determine whether the semantic query statement conforms to the pre-configured correction rules;

[0039] If not, the semantic query statement is input into a pre-configured machine learning model to obtain correction suggestions;

[0040] Determine whether the query conditions in the semantic query statement exceed three;

[0041] If so, the semantic query statement is decomposed to obtain a single query condition statement;

[0042] Based on the proposed corrections, the individual query condition statements are semantically corrected to obtain the corrected query statements.

[0043] Evaluate the query statement and obtain the evaluation results;

[0044] The pre-configured machine learning model and the correction rules are improved based on the evaluation results to obtain a target semantic corrector;

[0045] The semantic query statement is input into the target semantic corrector for correction in order to obtain the target semantic query statement.

[0046] Optionally, after converting the target semantic query statement into a structured query statement using a semantic translator, the method further includes:

[0047] The rules engine determines whether the structured query statement contains any errors.

[0048] If so, the structured query statement is input into a pre-configured deep learning model to obtain the statement error type;

[0049] Determine whether the error type of the statement is an incorrect field name;

[0050] If not, prompt the user to make a selection to obtain the result;

[0051] The structured query statement is adjusted based on the selection result to obtain the target structured query statement.

[0052] Optionally, the structured query statement is input into the database to retrieve query results, including:

[0053] The target structured query statement is input into the database to retrieve query data;

[0054] Select a data chart based on the data type of the query data;

[0055] The query data and the data charts are visualized to obtain the query results.

[0056] Optionally, a data chart can be selected based on the data type of the queried data, including:

[0057] Extract the statistical features of the query data;

[0058] The statistical features are input into a pre-configured recognition model to obtain the distribution type;

[0059] Generate data charts based on the distribution type.

[0060] A second aspect of this application provides a data analysis system, the system comprising:

[0061] The acquisition unit is used to acquire natural language input from the user.

[0062] The first building unit is used to construct the initial data model;

[0063] The second construction unit is used to perform word segmentation on the natural language based on a pre-configured industry knowledge base and a WordPiece word segmenter to construct a prompt.

[0064] The encoding unit is used to encode the prompt using a pre-trained model to obtain an encoding vector for each token.

[0065] The first processing unit is used to take the encoded vector of the first classification token output by the pre-trained model as the representation of the entire sentence;

[0066] The second processing unit is used to process all the encoded vectors according to the sentence representation to obtain a local semantic representation;

[0067] The third processing unit is used to identify entities and attributes in the natural language and map the entities and attributes to schema elements in the database to obtain mapping relationships;

[0068] The third construction unit is used to construct a semantic data model based on the mapping relationship and the local semantic representation;

[0069] The fourth building unit is used to combine the semantic data model with custom rules to build a semantic parser;

[0070] The fourth processing unit is used to input the natural language into the semantic parser for parsing to obtain a semantic query statement;

[0071] The fifth processing unit is used to input the semantic query statement into the semantic corrector for correction in order to obtain the target semantic query statement;

[0072] The sixth processing unit is used to convert the target semantic query statement into a structured query statement using a semantic translator;

[0073] The query unit is used to input the structured query statement into the database for querying to obtain query results;

[0074] The third building unit is specifically used for:

[0075] The mapping relationship is initialized based on the context information of the currently input natural language query.

[0076] Adjust the weights of the database schema elements based on the context information;

[0077] Construct an initial semantic graph, which includes entity nodes, relation nodes, operation nodes, entity relation edges, condition relation edges, and operation relation edges;

[0078] For each node, a comprehensive node feature representation is generated by combining the local semantic representation, the database metadata information, and the user's historical query behavior.

[0079] The features of each edge in the initial semantic graph are initialized to obtain a comprehensive edge feature representation;

[0080] The integrated node feature representation and the integrated edge feature representation are embedded into the initial semantic graph, and the initial semantic graph is dynamically updated based on the sink algorithm for implementing the connector to obtain the input semantic graph;

[0081] The input semantic graph is fed into a pre-constructed initial graph neural network for training. The information of neighboring nodes is aggregated through graph convolution operations in each layer of the initial graph neural network, and the state of each node is updated by combining the features of each node.

[0082] The initial graph neural network was optimized using the Adam optimizer to obtain a converged semantic data model.

[0083] A third aspect of this application provides a data analysis system, the system comprising:

[0084] Processor, memory, input / output units, and bus;

[0085] The processor is connected to the memory, the input / output unit, and the bus;

[0086] The memory stores a program, which the processor invokes to execute the first aspect and any one of the optional methods in the first aspect.

[0087] A fourth aspect of this application provides a computer-readable storage medium on which a program is stored, which, when executed on a computer, performs the methods of the first aspect and any one of the first aspects.

[0088] As can be seen from the above technical solutions, this application has the following advantages:

[0089] 1. This application segments natural language using a pre-configured industry knowledge base and WordPiece word segmenter to construct a prompt. The prompt is then encoded using a pre-trained model, accurately capturing semantic information in the natural language and enhancing its analysis and processing capabilities. Next, the first encoded classification token is used as the representation of the entire sentence. All encoded vectors are then processed based on this representation to obtain a local semantic representation. This local semantic representation reflects the complex semantic structure of natural language, helping to better understand user input intent and providing a foundation for subsequent semantic conversion. Furthermore, by mapping entities and attributes in natural language to database schema elements, a mapping relationship is obtained. This step establishes a bridge between natural language and structured data, enabling the effective organization and utilization of unstructured natural language. Finally, a semantic data model is constructed using the mapping relationship and local semantic representation. This model is then combined with custom rules to build a semantic parser. The constructed semantic parser improves the accuracy and efficiency of converting natural language into semantic query statements, providing a foundation for converting semantic query statements into structured query statements and enhancing the system's semantic understanding and analysis capabilities.

[0090] 2. By inputting the semantic query statement into the semantic corrector for correction, the semantic query statement is further optimized, eliminating ambiguity and errors in the semantic query statement, improving the accuracy of subsequent query results, and further enhancing natural language analysis capabilities.

[0091] 3. The target semantic query statement is translated by a semantic translator. This step converts the semantic query statement into a structured query statement that can be recognized by the machine, which is easier for the database to understand and execute, improves query efficiency and accuracy, and provides users with a better query experience. Attached Figure Description

[0092] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0093] Figure 1 is a schematic flowchart of an embodiment of the data analysis method provided in this application;

[0094] Figure 2 is a flowchart illustrating another embodiment of the data analysis method provided in this application;

[0095] Figure 3 is a flowchart illustrating another embodiment of the data analysis method provided in this application;

[0096] Figure 4 is a flowchart illustrating another embodiment of the data analysis method provided in this application;

[0097] Figure 5 is a flowchart illustrating another embodiment of the data analysis method provided in this application;

[0098] Figure 6 is a flowchart illustrating another embodiment of the data analysis method provided in this application;

[0099] Figure 7 is a flowchart illustrating an embodiment of the data analysis system provided in this application;

[0100] Figure 8 is a flowchart of another embodiment of the data analysis system provided in this application. Detailed Implementation

[0101] It should be noted that the data analysis method provided in this application can be applied to terminals, systems, and servers. For example, a terminal can be a smartphone, computer, tablet, smart TV, smartwatch, portable computer, or a desktop computer, etc. For ease of explanation, this application uses a terminal as the execution subject for illustration.

[0102] Please refer to Figure 1. This application first provides an embodiment of a data analysis method, which includes:

[0103] S101. Obtain the natural language input from the user;

[0104] Natural language refers to the language humans use in daily life, including spoken and written language. In this embodiment, natural language refers to the natural question and answer statements entered by the user in the search box.

[0105] In this embodiment, before acquiring the user's natural language input, a user input interface needs to be provided. The user then inputs their question-and-answer requests on this interface using a keyboard, voice, or other input methods. These input requests are captured by the system in data form, which helps in generating query results that meet the user's expectations.

[0106] S102. Construct the initial data model;

[0107] In this embodiment, before building the initial data model, the data source type must first be configured, followed by identifying the data source to be queried and configuring it accordingly. After configuring the data source, the data model can be built, which requires defining its dimensions, measures, metrics, and associated keys.

[0108] In this embodiment, constructing an initial data model provides the foundation for subsequently obtaining a semantic data model. The dimensions, metrics, indicators, and associated keys of the data model ensure the accuracy and relevance of the data.

[0109] S103. Perform word segmentation on the natural language based on the pre-configured industry knowledge base and WordPiece word segmenter to construct a prompt;

[0110] An industry knowledge base is a customized resource library that integrates professional knowledge, terminology, and corresponding data structures within a specific field. It aims to provide convenient, accurate, and comprehensive knowledge support for models or related systems. A WordPiece tokenizer is an algorithm for text processing, primarily used to break down words into subwords or tokens. A prompt is a designed text input, typically referring to a request or question posed to the model.

[0111] In this embodiment, before performing word segmentation on the natural language, it is necessary to integrate domain expertise, terminology, and data structures to configure an industry knowledge base. Then, the WordPiece word segmenter initializes the vocabulary based on the industry knowledge base and calculates the probability of each word in the natural language to obtain the initial likelihood value. Next, the WordPiece word segmenter selects adjacent words with the highest mutual information values ​​for merging, forming new words and updating the vocabulary. After each merging, the likelihood value of the sentence is recalculated to evaluate the impact of the merging on the overall probability of the sentence, until a predetermined vocabulary size is reached, ultimately obtaining a prompt of appropriate length.

[0112] In practical applications, assuming a sentence consists of n words, representing the i-th word, the language model likelihood value of that sentence T can be expressed as the product of the probabilities of all words. Next, adjacent words a and b are merged into a new word c, and the likelihood values ​​of the sentence before and after the merge are calculated. Then, the WordPiece segmenter is used to select the two adjacent sub-words with the highest mutual information values ​​for merging. After merging the two words, they are replaced with a new sub-word, and the vocabulary is updated. This process is iterated until a predetermined vocabulary size is reached. For example, for the query "How many TVs are sold?", after the above preprocessing, the input sequence of the pre-trained model is: [CLS] "TV", "of", "sales", "is", "how many?"

[0113] In this embodiment, by using a pre-configured industry knowledge base and WordPiece word segmenter to segment natural language, words and terms in the text can be accurately identified and segmented. Then, by reconstructing these words and terms, a prompt of appropriate length required by the pre-trained model can be obtained, which helps to encode the prompt using the pre-trained model in the future.

[0114] S104. Encode the prompt using a pre-trained model to obtain the encoding vector for each token;

[0115] Pre-trained models refer to models trained on large datasets using deep learning algorithms. These models learn a large number of general features and knowledge representations during training, and can be used for initialization solutions for various related tasks. In this embodiment, BERT, T5, and CodeBERT pre-trained models were used to train the data model. Three strategies were adopted for selecting pre-trained models: task specificity, model size, and domain adaptability. Task specificity refers to selecting models relevant to the SQL generation task; model size considers the balance between model size and available computing resources—larger models may be more accurate than smaller models, but they also increase computational overhead; domain adaptability refers to selecting models pre-trained in relevant domains (such as finance, healthcare, and e-commerce) to better understand domain-specific terminology and relationships. A token is the smallest unit or element into which a prompt is divided, which can be a word, subword (such as roots or affixes), character, or other text fragment.

[0116] In this embodiment, before encoding the prompt, it needs to be converted into the input format of the pre-trained model, specifically into the input_ids and attention_mask required by the pre-trained model. Here, input_ids is the index of each token, representing its representation in the pre-trained model's vocabulary, and attention_mask indicates which positions are valid. Next, the input_ids and attention_mask are input into the pre-trained model for processing, resulting in the encoded token vector representation, i.e., the token's encoded vector.

[0117] S105. Use the encoding vector of the first classification token output by the pre-trained model as the representation of the entire sentence;

[0118] The classification token, [CLS] token, is a special token primarily used for classification tasks. It is placed at the beginning of the input sequence to represent the category or overall semantics of the entire input sequence. Sentence representation refers to the meaning of the entire sentence.

[0119] In this embodiment, the encoded vector of the first classification token output by the pre-trained model is used as the representation of the entire sentence. The encoded vector of this classification token captures the overall semantic information of the sentence, providing a foundation for subsequent acquisition of local semantic representations.

[0120] S106. Process all the encoded vectors according to the sentence representation to obtain a local semantic representation;

[0121] In this embodiment, based on the obtained sentence representation, the encoded vectors of all tokens in the sentence are processed to extract local semantic information. This process, by analyzing the encoded vector of each token, yields a detailed local semantic representation, providing a foundation for subsequent construction of a semantic data model. This local semantic representation is used for specific field matching and semantic SQL generation.

[0122] S107. By identifying entities and attributes in the natural language and mapping the entities and attributes to schema elements in the database, a mapping relationship is obtained;

[0123] In this embodiment, entities and attributes in natural language refer to specific people, events, and things. For example, in a user-inputted query, "Show all orders with sales exceeding 1000," "sales amount" is both an entity and an attribute of this query. Database schema elements mainly include tables, constraints, views, stored procedures, and data types. These elements collectively constitute the structure and function of the database, defining how data is stored, the relationships between data, and how to manipulate this data. Mapping relationships refer to the correspondence between elements of two pieces of data.

[0124] In this embodiment, before obtaining the mapping relationship, it is first necessary to obtain natural language, then identify entities and attributes in the natural language, then extract key entities from the natural language based on the identified entities and attributes, further parse the database schema to obtain the schema information of the database, list all schema elements, and finally match the key entities in the natural language with the schema elements in the database to obtain the mapping relationship.

[0125] In practical applications, the system retrieves the user input "Show all orders with sales exceeding 1000", then extracts key entities such as "sales" and "orders" by identifying the entities and attributes in "Show all orders with sales exceeding 1000". Next, the database is parsed to obtain all relevant tables and columns. For example, the "orders" table has columns: id, sales, customer_id, and order_date. Furthermore, "sales" is matched to the sales column, and "orders" is matched to the orders table. The relationship between matching key entities to relevant tables or columns is called the mapping relationship.

[0126] This embodiment identifies entities and attributes in natural language and maps them to schema elements in the database, thereby establishing a mapping relationship between natural language and database structure. This allows the system to quickly understand user query information, reduces errors from manual intervention, and provides a foundation for later generating a model that converts natural language into semantic query statements.

[0127] S108. Construct a semantic data model based on the mapping relationship and the local semantic representation;

[0128] In this embodiment, constructing a semantic data model requires obtaining local semantic representations and the mapping relationship between natural language and database schema elements. Then, based on the mapping relationship and local semantic representations, a large model that converts natural language into semantic query statements is constructed, namely the semantic data model.

[0129] In this embodiment, a semantic data model is constructed to convert natural language into semantic query statements. Furthermore, when using the semantic data model, the model can accurately extract key entities from the natural language and then generate corresponding semantic query statements using internally configured mapping relationships and local semantic representations. This greatly improves the efficiency and accuracy of subsequent queries and provides a foundation for building a semantic parser later.

[0130] Specifically, one possible implementation of this step includes:

[0131] Step 1: Initialize the mapping relationship based on the context information of the currently input natural language query;

[0132] In this implementation, the input natural language query is first parsed by the natural language processing (NLP) module to extract key information, such as the target entity of the query, query conditions, attributes, operation type (such as SELECT, WHERE, etc.) and its context.

[0133] For example: "Find people who are older than 30" → Extract "age" as an attribute, "30 years old" as a condition, and "people" as an entity.

[0134] Analyze the context of the query to determine its context and intent. For example, if the query involves complex aggregations or conditional operations (such as AVG, SUM), the weights of different parts need to be adjusted according to the context.

[0135] Contextual information may include user query history, query frequency, and frequency of use of query fields.

[0136] Based on the extracted query elements and context information, natural language elements (such as entities, attributes, and operations) are mapped to schema elements (such as tables and columns) and relationships in the database, and the mapping relationships of each component in the query are initialized.

[0137] Step 2: Adjust the weights of the database schema elements based on the context information;

[0138] The weights of schema elements (such as tables and columns) in the database are dynamically adjusted based on the context of the input query. For example, if a user frequently queries a particular table or column, its importance in the current query is increased. Different initial weights are assigned to each entity, column, and operation, and these weights are further amplified based on the query context. Frequently used tables / columns may be given higher weights so that the model prioritizes these high-weight elements when generating queries. Historical user query behavior is analyzed to determine which schema elements are most relevant, and their weights are adjusted according to user query habits. This helps the model more accurately generate query structures that users are likely to focus on.

[0139] Step 3: Construct an initial semantic graph, which includes entity nodes, relation nodes, operation nodes, entity relation edges, condition relation edges, and operation relation edges.

[0140] This step aims to construct nodes and edges. For different nodes and edges, the following applies:

[0141] Entity nodes: map to database tables and columns, such as Customer and age.

[0142] Relationship node: Represents the relationship between tables, such as Customer.id = Order.customer_id.

[0143] Operation node: Indicates the SQL operation type, such as SELECT, WHERE, JOIN.

[0144] Edge construction:

[0145] Entity relationship edge: For example, Customer.id = Order.customer_id.

[0146] Conditional relationship edge: for example, age>30.

[0147] Operational relationship edges: such as the relationship between SELECT and the target field, and the relationship between WHERE and the conditions.

[0148] Based on the nodes and edges described above, an initial semantic graph is constructed. In this graph, nodes represent the core elements of the query (tables, columns, conditions, etc.), and edges represent the relationships and operations between these elements.

[0149] Step 4: For each node, combine the local semantic representation, the database metadata information, and the user's historical query behavior to generate a comprehensive node feature representation for each node;

[0150] Generate local semantic representations (such as word vectors, embeddings, etc.) for each node (such as table name, column name, conditions, etc.) based on the context of natural language queries.

[0151] For example, word vector representations can be generated for the table name "Customer" and the column name "age". Additional feature information is generated for nodes by combining database metadata (such as field types, table relationships, indexes, etc.). For example, if the "age" column is an integer, an "integer type" tag can be added to its feature vector. Personalized feature representations are generated by incorporating the user's query history. If a user has frequently queried the "age" field in the past, its weight will be higher, influencing the generation of the node's feature representation. By combining local semantics, metadata, and historical behavior, a comprehensive feature representation for each node is generated. These feature representations will then serve as input to the graph neural network.

[0152] Step 5: Initialize the features of each edge in the initial semantic graph to obtain a comprehensive edge feature representation;

[0153] When initializing edge features, the type of relationship they represent needs to be considered. For example, the features of edges in a JOIN relationship and those in a WHERE relationship may differ; a JOIN edge might contain join conditions, while a WHERE edge might contain filter conditions. Edge features can also be adjusted in conjunction with the query context. For instance, if the query involves joining multiple tables, the edge weights can reflect the complexity of the joins. A feature representation is generated for each edge by combining node features, relationship types, and query context. These edge feature representations will be propagated in subsequent graph convolution operations.

[0154] Step 6: Embed the integrated node feature representation and the integrated edge feature representation into the initial semantic graph, and dynamically update the initial semantic graph based on the sink algorithm for implementing the connector to obtain the input semantic graph;

[0155] The combined feature representation of each node and edge is embedded into the initial semantic graph to form the input semantic graph. The feature representation of each node and edge will serve as its input in the graph neural network. This ensures that the graph structure can effectively capture the semantics and relationships of the query.

[0156] In step 6, the semantic graph can be embedded using the TransE embedding model, specifically:

[0157] Map entities h, t and relation r to low-dimensional vectors;

[0158] Define the optimization objective: The optimization objective in the model is T;

[0159] The embedding model is trained by optimizing the function to obtain the embedding vector;

[0160] Based on similarity rules in the embedding space, the semantic graph content is expanded:

[0161] Logical rules based on the embedded space (such as path reasoning) supplement unannotated relationships.

[0162] The newly added data in real time is embedded and mapped, and the new entities and relationships are added to the semantic graph. Specifically:

[0163] Incrementally train the embedding model to avoid reconstructing the entire embedding space;

[0164] Update the semantic graph structure so that it can dynamically reflect new semantics;

[0165] A unified batch processing technology framework can be adopted, which implements the sink algorithm of the connector to write newly added mapping relationship data into the semantic graph in real time or offline, thereby achieving the effect of dynamic updates.

[0166] Step 7: Input the input semantic graph into the pre-constructed initial graph neural network for training. Aggregate the information of neighboring nodes through graph convolution operations in each layer of the initial graph neural network, and update the state of each node by combining the features of each node.

[0167] Each layer of a graph neural network collects information from neighboring nodes through graph convolution operations, aggregating features between nodes. For example, in the JOIN operation, graph convolution aggregates node features related to the Customer and Order tables to update node states and understand the relationships between them. The aggregated neighbor information is then combined to update the state of each node. Through multiple layers of graph convolution, the model can progressively capture more complex semantic relationships.

[0168] Step 8: Optimize the initial graph neural network using the Adam optimizer to obtain a converged semantic data model.

[0169] In this implementation, the Adam optimizer is used to optimize the parameters of the graph neural network and minimize the loss function. The optimization goal is to enable the graph neural network to accurately map the input natural language query to the SQL query statement. Mean squared error can be used for optimization. Through multiple rounds of training, the network weights are updated until the model converges, i.e., the difference between the output SQL query statement and the actual query is minimized.

[0170] S109. Construct a semantic parser by combining the semantic data model with custom rules;

[0171] A semantic parser is an important tool in natural language processing (NLP) technology. In this embodiment, its main function is to convert natural language text into semantic query statements, enabling the system to understand and execute human instructions. Custom rules refer to a series of rules defined manually based on specific needs or business logic. These rules guide the semantic parser on how to understand and process natural language input, and how to convert it into corresponding semantic representations or query statements. Custom rules may include, but are not limited to: defining how to identify key entities and attributes in natural language and map them to schema elements in the database; specifying the syntactic structure and semantic requirements of natural language input to ensure its validity and accuracy; and defining how to convert query conditions in natural language into filtering conditions in database query statements.

[0172] In this embodiment, building a semantic parser first requires obtaining a trained semantic data model, then constructing some custom rules, and finally building a semantic parser based on the semantic data model and custom rules to convert the user's natural language into a standardized semantic query statement. Building this semantic parser facilitates the subsequent conversion of the user's natural language into a standardized semantic query statement, laying the foundation for obtaining accurate query results later.

[0173] Specifically, one possible implementation of this step includes:

[0174] Step a: Construct a standardized query template framework, which includes selection operations, filtering conditions, sorting and grouping operations;

[0175] Define a select operation template to specify the target entity or attribute to be returned in the query. The select operation is the first part of the SQL query.

[0176] Example: For the query "Find customer's name", the template framework will convert it to:

[0177] SELECT name FROM Customer

[0178] Filtering conditions (WHERE):

[0179] Define a filter template to specify the constraints in the query. Filter conditions typically correspond to the WHERE clause and specify which records meet the query criteria.

[0180] Example: For the query "Find customers older than 30", the template framework will convert it to:

[0181] SELECT name FROM Customer WHERE age>30

[0182] Sorting and grouping (ORDER BY, GROUP BY):

[0183] Define sorting rules to map the sorting conditions in the query to the ORDER BY statement in SQL.

[0184] Example: Query "List of customers sorted by age" → SELECT name, age FROM Customer ORDER BY age

[0185] Define grouping rules to map the grouping conditions in the query to the GROUP BY statement in SQL.

[0186] Example: Query "Number of customers grouped by city" → SELECT city, COUNT(*) FROM Customer GROUP BY city

[0187] This standardized framework ensures that natural language queries can be transformed into structured, standardized SQL queries.

[0188] Step b: Construct entity mapping rules to map entities in natural language queries to tables and columns in the semantic data model;

[0189] Use Natural Language Processing (NLP) techniques to identify entities in a query. Entities are typically nouns or noun phrases that represent specific objects in the database (such as customers, products, orders, etc.).

[0190] Example: Query "Find customer's name", where Customer is an entity.

[0191] For each identified entity, define mapping rules to convert it into tables and columns in the database.

[0192] For example, "customer" might be mapped to the Customer table in the database, and "name" might be mapped to the name column. It's also necessary to handle potential ambiguity to ensure the same entity is mapped correctly in different contexts.

[0193] Example: A Customer might have multiple attributes such as name and address, so the specific fields of the entity need to be determined based on the context. Entity mapping rules ensure that entities in a query are correctly mapped to database tables and columns in the semantic data model.

[0194] Step c: Construct condition mapping rules to map conditions in natural language queries to standard operators in the database;

[0195] Based on the query objective, determine the entities or attributes to be selected. For example, the query "find customer name" will generate `SELECT name FROM Customer`. Generate the WHERE clause of the query using condition mapping rules. For example, a query for "age greater than 30" will be mapped to `WHERE age>30`. Use custom sorting and grouping rules to handle the ORDER BY and GROUP BY clauses in the query.

[0196] For example, the query "sort by age" will be mapped to ORDER BY age.

[0197] Step d: Construct sorting and grouping rules to handle sorting and grouping operations in natural language queries, mapping them to ORDER BY and GROUP BY in SQL statements;

[0198] For queries involving multiple tables, use join rules to generate JOIN operations. For example, for the query "find the relationship between customers and orders", a custom rule can be used to determine whether to use JOIN to join the Customer and Order tables, with the condition Customer.id = Order.customer_id.

[0199] Step e: Construct subquery rules to handle complex queries that contain subqueries and generate corresponding query commands;

[0200] For queries containing subqueries, nested query commands are generated using subquery rules. Nested SQL subqueries are generated by decomposing the structured portion of the query.

[0201] Step f: Construct join rules to handle join operations between multiple tables and generate corresponding SQL join conditions;

[0202] By applying all rules to specific natural language queries, the final SQL query statement can be obtained. The outputs of custom rules (such as entities, conditions, sorting, etc.) are combined into an SQL query structure that conforms to the semantic data model.

[0203] For example, for the natural language query "find all customers older than 30 and sorted in ascending order of age", the final SQL query could be:

[0204] SELECT name FROM Customer WHERE age>30ORDER BY age ASC

[0205] Step g: Construct a filling relationship that embeds the output of the custom rule into the standardized query template framework to obtain a semantic parser.

[0206] Embedding the output of your custom rule into the standardized query template framework, filling in placeholders, generates a complete SQL query. Specifically, the output of your custom rule (e.g., table name, column name, conditions, etc.) is filled into placeholders within the standardized query template framework. Assume the custom rule output is as follows:

[0207] columns→name,age

[0208] table→Customer

[0209] conditions→age>30

[0210] group_by → empty (no grouping)

[0211] order_by→ORDER BY age ASC

[0212] SQL query after filling:

[0213] SELECT name, age

[0214] FROM Customer

[0215] WHERE age>30

[0216] ORDER BY age ASC;

[0217] A mapping mechanism for filling relationships also needs to be created for different types of placeholders (such as {columns}, {table}, {conditions}, etc.).

[0218] The core of a semantic parser is query parsing and population based on predefined rules and templates. Flexible query parsing is achieved through the design of appropriate data structures (such as rule lists, mapping tables, and template engines).

[0219] The parser first maps the natural language query to the corresponding rule output (such as entity, condition, sort, etc.), and then populates these outputs into a standardized query template framework.

[0220] Generate a complete SQL query statement based on the populated query template.

[0221] S110. Input the natural language into the semantic parser for parsing to obtain a semantic query statement;

[0222] In this embodiment, before obtaining the semantic query statement, it is necessary to obtain the natural language input by the user, and then input the natural language into the semantic parser. The semantic parser calls the semantic data model according to the custom rules set internally, and inputs the natural language into the semantic data model. Then, the semantic data model processes the natural language to obtain the semantic query statement.

[0223] S111. Input the semantic query statement into the semantic corrector for correction, so as to obtain the target semantic query statement;

[0224] Semantic correctors are primarily used to correct semantic errors in semantic queries, ensuring that the queries are expressed more accurately and naturally. By deeply analyzing the semantic meaning of the query, semantic correctors identify and correct semantic deviations, making the query expression more consistent with expectations.

[0225] In this embodiment, before correcting the semantic query statement, an initial semantic corrector is first constructed. Then, it is determined whether the statement conforms to preset correction rules. If it does not conform, it is input into a pre-trained machine learning model to obtain correction suggestions. If it conforms, the target semantic query statement is directly output. Next, it is checked whether the semantic query statement contains more than three query conditions. If it does, the structure of the semantic query statement is decomposed to obtain the query statement for each individual query condition. Subsequently, semantic correction is performed on the query statement for each individual query condition according to the correction suggestions. If there are no more than three conditions, semantic correction is performed directly to obtain the corrected query statement. Further, the corrected query statement is evaluated, and the machine learning model and correction rules are optimized and improved based on the evaluation results until the target semantic corrector is constructed. Finally, the original semantic query statement is input into this target semantic corrector to obtain the optimized target semantic query statement.

[0226] In this embodiment, inputting the semantic query statement into the semantic corrector for semantic correction can significantly improve the accuracy of the query statement, ensure that the query results are more in line with user expectations, and reduce resource waste caused by erroneous queries.

[0227] S112. The target semantic query statement is converted into a structured query statement using a semantic translator;

[0228] A semantic translator is a tool that converts semantic queries into structured queries. Based on natural language processing and semantic understanding technologies, it deeply analyzes the semantic information in queries and accurately converts it into structured queries that databases or other systems can understand and execute. A structured query is a programming language specifically designed for interacting with databases, providing a set of tools for manipulating databases, including functions for querying, inserting, updating, and deleting data.

[0229] In this embodiment, before obtaining the structured query statement, it is necessary to construct a conversion relationship between the semantic query statement and the semantic data model. Then, based on this conversion relationship, a semantic translator is built to convert the semantic query statement into a structured query statement. Next, the target semantic query statement is input into the semantic translator for processing. Further, the target semantic query statement is parsed into a semantic representation containing key entities, attributes, and relationships. Finally, the semantic translator, based on these semantic representations and combined with the preset conversion relationship, gradually constructs the corresponding structured query statement. In practical applications, this process accurately converts the semantic query statement into a structured query statement, greatly improving the accuracy of subsequent queries.

[0230] S113. Input the structured query statement into the database to perform a query and obtain the query results.

[0231] In this embodiment, the obtained structured query statement is input into the database, and then the database executes the structured query statement to obtain the query data. Next, different data charts are selected according to the data type of the query data, and finally the query data and data charts are visualized to obtain the query results.

[0232] In practical applications, inputting structured query statements into a database not only quickly yields accurate data results but also intelligently matches data charts based on data type, enabling data visualization. This process makes the data results more intuitive and easier to understand, helping users make informed decisions.

[0233] In this embodiment, this application uses a pre-configured industry knowledge base and a WordPiece word segmenter to segment natural language to construct a prompt. The prompt is then encoded using a pre-trained model, accurately capturing semantic information in the natural language and improving natural language analysis and processing capabilities. Next, the first classification token obtained after encoding is used as the representation of the entire sentence. All encoded vectors are then processed based on this sentence representation to obtain a local semantic representation. This local semantic representation reflects the complex semantic structure of natural language, helping to better understand the user's input intent and providing a foundation for subsequent semantic conversion. Furthermore, by mapping entities and attributes in the natural language to database schema elements to obtain mapping relationships, this step establishes a bridge between natural language and structured data, enabling unstructured natural language to be effectively organized and utilized. Finally, a semantic data model is constructed through mapping relationships and local semantic representations. This model is then combined with custom rules to build a semantic parser. The constructed semantic parser improves the accuracy and efficiency of converting natural language into semantic query statements, providing a foundation for subsequent conversion of semantic query statements into structured query statements, and enhancing the system's semantic understanding and analysis capabilities. By inputting the semantic query statement into a semantic corrector for correction, the semantic query statement is further optimized, eliminating ambiguity and errors, improving the accuracy of subsequent query results, and further enhancing natural language analysis capabilities. Finally, a semantic translator translates the target semantic query statement, converting it into a machine-recognizable structured query statement, facilitating database understanding and execution, improving query efficiency and accuracy, and providing users with a better query experience.

[0234] Please refer to Figure 2, which illustrates another embodiment of a data analysis method provided in this application. This embodiment includes:

[0235] S201. Convert the prompt into the input format of the pre-trained model to obtain input_ids and attention_mask;

[0236] In this embodiment, the given prompt is preprocessed, including steps such as removing irrelevant characters and word segmentation, to obtain an input that meets the requirements of the pre-trained model. Next, the processed prompt is converted into an input format that the pre-trained model can recognize, namely, input_ids and attention_mask. Here, input_ids represents the index of each token, and attention_mask indicates which positions are valid. In practical applications, converting the processed prompt into the input_ids and attention_mask required by the pre-trained model provides the foundation for the pre-trained model to encode the prompt.

[0237] S202. Input the input_ids and the attention_mask into the pre-trained model to obtain the encoding vector of each token.

[0238] In this embodiment, the preprocessed input_ids and attention_mask are used as inputs to the pre-trained model. The pre-trained model then searches for the corresponding vector in the vocabulary based on the token index in the input_ids, and applies a weighted processing to the input sequence using the attention_mask to identify valid positions. Further, the model encodes the input_ids and attention_mask, ultimately outputting an encoded vector for each token.

[0239] In this embodiment, converting the prompt into the input format of the pre-trained model ensures that the pre-trained model can accurately understand the input data, thereby improving the efficiency and accuracy of natural language processing. After inputting the obtained input formats `input_ids` and `attention_mask` into the pre-trained model, we can obtain the encoded vector of each token. These vectors capture contextual information and semantic features, enhancing the model's ability to understand text and also helping to obtain the intent in natural language during subsequent natural language analysis.

[0240] Please refer to Figure 3, which illustrates another embodiment of a data analysis method provided in this application. This embodiment includes:

[0241] S301. Extract key entities from the natural language;

[0242] In this embodiment, before obtaining key entities in natural language, it is necessary to obtain the natural language input by the user, then identify the entities and attributes in the natural language, and then extract the key entities in the natural language based on the identified entities and attributes.

[0243] In practical applications, when a user inputs a query in natural language to inquire about detailed information about a product, the system first obtains this input. Then, the system analyzes, processes, and identifies key entities, such as product name, model number, and color. These key entities are then used to match product information in the database. Extracting key entities from natural language improves the accuracy of data processing, enabling rapid location and extraction of core information from the language, thus providing strong support for subsequent mapping relationships.

[0244] S302. Query the database schema information to obtain the schema elements of the database;

[0245] Database schema information refers to metadata that describes the structure and organization of a database. In a database, database schema information defines the structure and relationships of elements such as tables, fields, relational models, views, and indexes, as well as data integrity constraints. This information is usually stored in a data dictionary to ensure the consistency, integrity, and reliability of the database.

[0246] In this embodiment, before parsing the database schema information, it is necessary to connect to the target database and then use the metadata query function provided by the database management system to retrieve the database schema information. Next, the retrieved schema elements are extracted for subsequent matching. This process, by connecting to the target database and retrieving schema elements, allows the system to understand the database's structure and characteristics, providing reliable data support for subsequent matching operations.

[0247] S303. Match the key entities with the schema elements to obtain the mapping relationship.

[0248] In this embodiment, before obtaining the mapping relationship, it is necessary to search for key entities in the extracted schema elements based on the natural language. Then, based on the search results, the key entities and the found schema elements are matched to obtain the mapping relationship between them. This mapping relationship enables the system to more accurately understand the user's query intent, providing support for the subsequent construction of a semantic data model.

[0249] In this embodiment, by first extracting key entities from natural language, the core content of the natural language input by the user can be identified, providing a foundation for subsequent matching of information in the database. Then, by obtaining the schema elements in the database, the structure of the database can be accurately understood, which helps to match natural language to database fields. Furthermore, the key entities are matched with schema elements to obtain mapping relationships, which not only enhances the accuracy of subsequent queries but also strengthens the connection between natural language and database data.

[0250] Please refer to Figure 4, which illustrates another embodiment of a data analysis method provided in this application. This embodiment includes:

[0251] S401, Construct the initial semantic modifier;

[0252] In this embodiment, when constructing the initial semantic corrector, it is necessary to define pattern matching rules, logical consistency rules, and semantic consistency rules. The pattern matching rules are used to check whether the fields in the query statement match the database schema. The logical consistency rules are used to ensure that the logical structure of the query statement conforms to the rules of the structured query language. The semantic consistency rules are used to ensure that the query statement matches the input natural language query.

[0253] S402. Determine whether the semantic query statement conforms to the pre-configured correction rules;

[0254] In this embodiment, the obtained semantic query statements are matched one by one with the pattern matching rules, logical consistency rules, and semantic consistency rules. If the semantic query semantics all conform to the pattern matching rules, logical consistency rules, and semantic consistency rules, then step S403 is executed. If the semantic query semantics do not conform to one or more of the rules, then step S404 is executed.

[0255] S403, Output the target semantic query statement;

[0256] In this embodiment, if the semantic query statement conforms to the pre-configured correction rules, no semantic correction is required, and the semantic corrector can directly output the semantic query statement.

[0257] S404. Input the semantic query statement into a pre-configured machine learning model to obtain correction suggestions;

[0258] A machine learning model is an algorithm or set of algorithms that learns from training data. It can identify patterns, relationships, or regularities in the data and use this learned knowledge to predict or classify future data.

[0259] In this embodiment, before obtaining correction suggestions, an initial machine learning model and a training dataset need to be constructed. The training dataset is built from a large number of query statements and their corresponding correction results. Then, a supervised learning method is used in conjunction with the training dataset to train the machine learning model to identify errors in the query statements and output corresponding correction suggestions. Furthermore, the output correction suggestions are compared with preset correction suggestions to obtain comparison results. The machine learning model is then optimized based on the comparison results to obtain the target machine learning model. Finally, the semantic query statement is input into the target machine learning model to obtain correction suggestions.

[0260] S405. Determine whether the query conditions in the semantic query statement exceed three;

[0261] In this embodiment, to determine whether there are more than three query conditions in a semantic query statement, it is necessary to identify all query conditions in the semantic query statement. Then, the total number of query conditions in the semantic query statement is calculated. Further, the obtained total number of query conditions is compared with the set judgment conditions. If there are no more than three query conditions, step S406 is executed; if there are more than three query conditions, step S407 is executed. Besides the above judgment conditions, other conditions can be set according to actual circumstances, and no specific limitations are made here.

[0262] S406. Perform semantic correction on the semantic query statement according to the correction suggestion to obtain the corrected query statement;

[0263] In this embodiment, if the semantic query statement contains no more than three query conditions, the semantic query statement is directly combined with the correction suggestions given by the machine learning model to perform semantic correction, so as to obtain the corrected semantic query statement.

[0264] S407. Decompose the semantic query statement to obtain a single query condition statement;

[0265] In this embodiment, semantic query statements with more than three query conditions are parsed to obtain keywords and conditions. Then, based on the structure and semantic relationship of the semantic query statements, the keywords and conditions are parsed and grouped to form a single independent query condition. For example, the original query statement is "find mobile phones priced below 100 yuan and rated above 4 stars". Through the above steps, it is decomposed into two query condition statements: "mobile phones priced below 100 yuan" and "mobile phones rated above 4 stars".

[0266] S408. Perform semantic correction on the individual query condition statement according to the correction suggestion to obtain the corrected query statement;

[0267] In this embodiment, before obtaining the corrected query statement, each individual query condition statement needs to be corrected according to the correction suggestions output by the pre-configured machine learning model. Then, the corrected individual query condition statements are summarized to obtain the corrected query statement. This process can automatically discover and correct potential errors or unreasonable aspects in individual query condition statements, thereby avoiding the tediousness and error-proneness of manual correction, speeding up the correction of query statements, and improving the reliability of query results.

[0268] S409. Evaluate the query statement and obtain the evaluation result;

[0269] In this embodiment, before obtaining the evaluation results, it is necessary to calculate metrics such as precision and recall of the query results. These metrics are used to measure the performance improvement of the query statement. Then, qualitative methods, such as manual review of query results and logs, are used to evaluate the semantic accuracy and logical rationality of the query statement. Finally, the overall evaluation effect of the modified query statement is obtained by combining the metrics and qualitative evaluation results. This comprehensive evaluation method helps ensure the optimization effect of the query statement and provides some assistance in obtaining accurate semantic query statements subsequently.

[0270] S410. Based on the evaluation results, improve the pre-configured machine learning model and the correction rules to obtain a target semantic corrector;

[0271] In this embodiment, to obtain the target semantic corrector, the performance of the pre-configured machine learning model and the correction rules need to be further optimized based on the evaluation results. This process needs to be iterated until the preset correction effect is achieved before the target semantic corrector can be obtained.

[0272] In this embodiment, by improving the machine learning model and correction rules based on the evaluation results to obtain the target semantic corrector, this process can obtain objective feedback on the correction effect, optimize the machine learning model and correction rules, improve the accuracy and efficiency of semantic correction, ensure that the corrected query statement can accurately reflect the user's true intent, improve the intelligence level of semantic processing, and provide users with a more accurate and efficient query experience.

[0273] S411. Input the semantic query statement into the target semantic corrector for correction to obtain the target semantic query statement.

[0274] In this embodiment, when a semantic query is input into the target semantic corrector, the semantic corrector determines whether it conforms to the internally pre-configured correction rules. If it does not conform, it is input into a pre-configured machine learning model to obtain correction suggestions for the semantic query. Then, it determines whether the semantic query contains more than three query conditions. If it does, the semantic query is directly decomposed into statements of individual query conditions, and these individual query conditions are corrected and integrated according to the correction suggestions to obtain the target semantic query. If it contains no more than three query conditions, the semantic query is directly corrected according to the correction suggestions to obtain the target semantic query.

[0275] In this embodiment, by determining whether the query statement conforms to the pre-configured correction rules, statements requiring optimization can be quickly filtered out. For semantic query statements that do not conform to the correction rules, a pre-configured machine learning model is used to analyze and process them to obtain correction suggestions, enhancing the intelligent correction capability of the semantic corrector. Next, by decomposing semantic query statements that do not conform to the correction rules and have more than three internal query conditions, individual query condition statements are obtained, reducing the complexity of semantic query statements and improving the efficiency of correction processing. Furthermore, individual query condition statements are corrected according to the correction suggestions, improving the efficiency of correction and the accuracy of the corrected query statement. The corrected query statement is evaluated to obtain the evaluation effect, ensuring the quality of the modified target semantic query statement and providing a feedback mechanism for improving the semantic corrector.

[0276] Please refer to Figure 5, which illustrates another embodiment of a data analysis method provided in this application. This embodiment includes:

[0277] S501. Determine whether there are any statement errors in the structured query statement based on the rule engine;

[0278] A rules engine is a software system that uses predefined rules to evaluate data and perform actions. These rules are typically written in a specialized language called a rules language.

[0279] In this embodiment, before determining whether there are statement errors in the structured query statement, a rule engine needs to be constructed. This rule engine includes SQL syntax rules, database schema rules, and logical consistency rules to ensure the accuracy of the syntax, structure, and logic of the structured query statement. The structured query statement is then input into the constructed rule engine for judgment. If the structured query statement satisfies one or more rules in the rule engine, it indicates that there are no statement errors in the structured query statement, and step S502 is executed. If the structured query statement does not satisfy one or more rules in the rule engine, it indicates that there are statement errors in the structured query statement, and step S503 is executed.

[0280] S502, Output the target structured query statement;

[0281] In this embodiment, if the structured query statement conforms to the SQL syntax rules, database schema rules, and logical consistency rules set by the rule engine, and no errors or anomalies are identified, the rule engine will directly use the structured query statement as the target structured query statement for subsequent data querying.

[0282] S503. Input the structured query statement into a pre-configured deep learning model to obtain the statement error type;

[0283] Deep learning models are machine learning models based on artificial neural networks. Their core idea is to learn the feature representations and patterns of data through multi-layered neural network structures, thereby achieving the understanding and prediction of data.

[0284] In this embodiment, before inputting the structured query statement into the pre-configured deep learning model, a model needs to be built using a deep learning algorithm, and a large dataset of features and annotations from the structured query statements needs to be collected. Then, the model is trained using this dataset, which records the error conditions and error types for each structured query statement. Further, through training, the model can obtain a deep learning model capable of recognizing and classifying query errors. After the deep learning model is built, the structured query statement can be input into it, and the model will output the corresponding error type, such as syntax errors, logical errors, missing fields, etc.

[0285] S504. Determine whether the error type of the statement is a field name error;

[0286] In this embodiment, the system will determine whether the statement error type is a field name error based on the obtained statement error type. If the statement error type is a field name error, then step S505 is executed; if the statement error type is not a field name error, then step S506 is executed.

[0287] S505. Adjust the errors in the statement to obtain the target structured query statement;

[0288] In this embodiment, if the statement error type is a field name error, the system will automatically provide similar field names based on the semantics and content of the structured query statement, and then correct the statement based on the similar field names to obtain the target structured query statement.

[0289] S506. Prompt the user to make a selection to obtain the selection result;

[0290] In this embodiment, if the statement error type is not a field name error, the system will ask the user a question through the interface and provide the user with a choice, such as, "Do you want to use field A or field B for the query?", and then obtain the user's selection result through the user's selection and confirmation.

[0291] S507. Adjust the structured query statement according to the selection result to obtain the target structured query statement.

[0292] In this embodiment, the structured query statement is analyzed based on the user's selection, and the parts that need to be modified are identified to obtain modification suggestions. Then, the structured query statement is modified according to the modification suggestions, such as modifying its internal query conditions or sorting. After the modification is completed, the target structured query statement can be obtained.

[0293] In this embodiment, by using a rule engine and machine learning model to identify and classify structured query statements, the system can automatically identify potential errors and error types in structured query statements, improving the accuracy and efficiency of detection. By determining the error type of the statement, errors in the structured query statement can be quickly located, reducing unnecessary detection steps and improving the speed and accuracy of problem solving. If the statement error type is not a field error, a prompt request is issued to the user. This step is used to provide the user with a choice when the best adjustment direction cannot be determined. Adjusting the query statement based on the user's selection result semantically conforms to the user's query intent, realizing personalized queries and ensuring the accuracy of the query statement.

[0294] Please refer to Figure 6, which illustrates another embodiment of a data analysis method provided in this application. This embodiment includes:

[0295] S601. Input the target structured query statement into the database to query and obtain query data;

[0296] In this embodiment, when performing a database query, the system needs to establish a connection with the database. Then, the adjusted target structured query statement is input into the target database. Next, the database retrieves all data within the database according to the query conditions in the target structured query statement. Furthermore, the database organizes all data that meets the query conditions to obtain query data. For example, the obtained query statement "select * from customers where age>30" is input into the database. Then, the database retrieves all customer data with an age greater than 30 according to this query statement instruction. Further, the database sorts and adjusts this customer data and returns query data containing this customer data.

[0297] S602. Extract the statistical features of the query data;

[0298] In this embodiment, before extracting the statistical characteristics of the query data, the mean of the query data is first calculated. Then, the query data is sorted from smallest to largest, and the median is determined based on the parity of the number of query data points. Next, the dispersion of the query data, i.e., variance, is calculated. Further, the asymmetry of the query data distribution, i.e., skewness, is calculated. Further still, the sharpness of the query data distribution, i.e., kurtosis, is evaluated. Finally, the obtained mean, median, variance, skewness, and kurtosis are collected and organized to obtain the statistical characteristics of the query data.

[0299] S603. Input the statistical features into the pre-configured recognition model to obtain the distribution type;

[0300] In statistics, distribution type refers to the shape or distribution pattern that data or random variables may exhibit. Different distribution types have different characteristics and application scenarios, and can describe and characterize the probability distribution characteristics of data under different conditions. Common distribution types include normal distribution, uniform distribution, and skewed distribution.

[0301] In this embodiment, before inputting statistical features into the pre-configured recognition model, a recognition model needs to be constructed. This model can be built using decision trees, random forests, or deep learning algorithms. Next, a dataset is constructed by collecting various data distribution types and their corresponding visualization methods. Further, the dataset and statistical features are input into the recognition model for training, resulting in a target recognition model capable of identifying data distribution types. This target recognition model primarily analyzes the statistical features and automatically identifies and outputs the corresponding distribution type based on the analysis results.

[0302] S604. Generate a data chart based on the distribution type;

[0303] Data charts are a way of organizing and presenting data using visual elements such as graphs, tables, and symbols. By encoding and transforming data, they present it to the audience in a more intuitive and easy-to-understand form, thereby helping people to better analyze, understand, and communicate data.

[0304] In this embodiment, it is necessary to identify whether the obtained distribution type is normal, uniform, or skewed. Then, the identified distribution type is mapped to a visualization chart to finally generate a data chart. Taking a normal distribution as an example, if the distribution type is normal, the system recommends selecting a histogram or density plot.

[0305] S605. Visualize the query data and the data chart to obtain the query results.

[0306] Visual presentations utilize techniques and methods such as graphic design, data visualization, information graphics, and interactive design to transform complex data, information, or knowledge into visual forms that are easy to understand and intuitively perceive.

[0307] In this embodiment, the query data obtained from the database and the generated data charts need to be integrated, and then the integrated results are displayed on the user interface in a visual way. The displayed results are the query results.

[0308] In this embodiment, by executing a target structured query statement in the database, query data that meets the query conditions can be obtained. Next, the statistical features of this query data are extracted and input into a pre-configured recognition model. The recognition model can automatically determine the data distribution type, reducing human error. Furthermore, based on the identified distribution type, corresponding data charts are automatically generated, enhancing data visualization. Finally, the query data and data charts are combined and visualized, providing users with an intuitive data analysis result. This enables users to make more informed decisions based on this intuitive data, further improving data processing capabilities.

[0309] Please refer to Figure 7, which illustrates a data analysis system provided in this application. The system includes:

[0310] Acquisition unit 701 is used to acquire natural language input from the user;

[0311] The first building unit 702 is used to build the initial data model;

[0312] The second construction unit 703 is used to perform word segmentation on the natural language based on a pre-configured industry knowledge base and a WordPiece word segmenter to construct a prompt.

[0313] Encoding unit 704 is used to encode the prompt using a pre-trained model to obtain an encoding vector for each token;

[0314] The first processing unit 705 is used to use the encoded vector of the first classification token output by the pre-trained model as the representation of the entire sentence;

[0315] The second processing unit 706 is used to process all the encoded vectors according to the sentence representation to obtain a local semantic representation;

[0316] The third processing unit 707 is used to obtain a mapping relationship by recognizing entities and attributes in the natural language and mapping the entities and attributes to schema elements in the database;

[0317] The third construction unit 708 is used to construct a semantic data model based on the mapping relationship and the local semantic representation;

[0318] The fourth construction unit 709 is used to combine the semantic data model with custom rules to construct a semantic parser;

[0319] The fourth processing unit 710 is used to input the natural language into the semantic parser for parsing to obtain a semantic query statement;

[0320] The fifth processing unit 711 is used to input the semantic query statement into the semantic corrector for correction in order to obtain the target semantic query statement;

[0321] The sixth processing unit 712 is used to convert the target semantic query statement into a structured query statement through a semantic translator;

[0322] The query unit 713 is used to input the structured query statement into the database to perform a query and obtain query results.

[0323] Optionally, the encoding unit 704 is also used for:

[0324] The prompt is converted into the input format of the pre-trained model to obtain input_ids and attention_mask;

[0325] The input_ids and attention_mask are input into the pre-trained model to obtain the encoding vector for each token.

[0326] Optionally, the third processing unit 707 is also used for:

[0327] Extract key entities from the natural language;

[0328] Query the database schema information to obtain the schema elements of the database;

[0329] Match the key entities with the schema elements to obtain the mapping relationship.

[0330] Optionally, the fifth processing unit 711 is also used for:

[0331] Construct an initial semantic corrector;

[0332] Determine whether the semantic query statement conforms to the pre-configured correction rules;

[0333] If not, the semantic query statement is input into a pre-configured machine learning model to obtain correction suggestions;

[0334] Determine whether the query conditions in the semantic query statement exceed three;

[0335] If so, the semantic query statement is decomposed to obtain a single query condition statement;

[0336] Based on the proposed corrections, the individual query condition statements are semantically corrected to obtain the corrected query statements.

[0337] Evaluate the query statement and obtain the evaluation results;

[0338] The pre-configured machine learning model and the correction rules are improved based on the evaluation results to obtain a target semantic corrector;

[0339] The semantic query statement is input into the target semantic corrector for correction in order to obtain the target semantic query statement.

[0340] Optionally, a seventh processing unit 714 is also included, for:

[0341] The rules engine determines whether the structured query statement contains any errors.

[0342] If so, the structured query statement is input into a pre-configured deep learning model to obtain the statement error type;

[0343] Determine whether the error type of the statement is an incorrect field name;

[0344] If not, prompt the user to make a selection to obtain the result;

[0345] The structured query statement is adjusted based on the selection result to obtain the target structured query statement.

[0346] Optionally, query unit 713 is also used for:

[0347] The target structured query statement is input into the database to retrieve query data;

[0348] Select a data chart based on the data type of the query data;

[0349] The query data and the data charts are visualized to obtain the query results.

[0350] Optionally, query unit 713 is also used for:

[0351] Extract the statistical features of the query data;

[0352] The statistical features are input into a pre-configured recognition model to obtain the distribution type;

[0353] Generate data charts based on the distribution type.

[0354] Optionally, the third building block is specifically used for:

[0355] The mapping relationship is initialized based on the context information of the currently input natural language query.

[0356] Adjust the weights of the database schema elements based on the context information.

[0357] Construct an initial semantic graph, which includes entity nodes, relation nodes, operation nodes, entity relation edges, condition relation edges, and operation relation edges;

[0358] For each node, a comprehensive node feature representation is generated by combining the local semantic representation, the database metadata information, and the user's historical query behavior.

[0359] The features of each edge in the initial semantic graph are initialized to obtain a comprehensive edge feature representation.

[0360] The integrated node feature representation and the integrated edge feature representation are embedded into the initial semantic graph, and the initial semantic graph is dynamically updated based on the sink algorithm for implementing the connector to obtain the input semantic graph;

[0361] The input semantic graph is fed into a pre-constructed initial graph neural network for training. The information of neighboring nodes is aggregated through graph convolution operations in each layer of the initial graph neural network, and the state of each node is updated by combining the features of each node.

[0362] The initial graph neural network was optimized using the Adam optimizer to obtain a converged semantic data model;

[0363] Optionally, the fourth building block is specifically used for:

[0364] A standardized query template framework is constructed, which includes selection operations, filtering conditions, sorting and grouping operations;

[0365] Construct entity mapping rules to map entities in natural language queries to tables and columns in the semantic data model;

[0366] Construct condition mapping rules to map conditions in natural language queries to standard operators in the database;

[0367] Construct sorting and grouping rules to handle sorting and grouping operations in natural language queries, mapping them to ORDER BY and GROUP BY in SQL statements;

[0368] Construct subquery rules to process complex queries that contain subqueries and generate corresponding query commands;

[0369] Build join rules to handle join operations between multiple tables and generate corresponding SQL join conditions;

[0370] A semantic parser is obtained by constructing a population relation that embeds the output of the custom rules into the standardized query template framework.

[0371] Referring to Figure 8, this application also provides another data analysis system, which includes:

[0372] Processor 801, memory 802, input / output unit 803, bus 804;

[0373] The processor 801 is connected to the memory 802, the input / output unit 803, and the bus 804;

[0374] The memory 802 stores a program, and the processor 801 calls the program to execute any of the methods described above.

[0375] This application also relates to a computer-readable storage medium on which a program is stored, which, when run on a computer, causes the computer to perform any of the methods described above.

[0376] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0377] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0378] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0379] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0380] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data analysis method, characterized in that, The method includes: Obtain natural language input from the user; Build the initial data model; The natural language is segmented using a pre-configured industry knowledge base and a WordPiece word segmenter to construct a prompt; The prompt is encoded using a pre-trained model to obtain the encoded vector for each token; The encoding vector of the first classification token output by the pre-trained model is used as the representation of the entire sentence; Based on the sentence representation, all the encoded vectors are processed to obtain a local semantic representation; By identifying entities and attributes in the natural language and mapping them to schema elements in the database, a mapping relationship can be obtained; Based on the mapping relationship and the local semantic representation, a semantic data model is constructed; A semantic parser is constructed by combining the semantic data model with custom rules. The natural language is input into the semantic parser for parsing to obtain a semantic query statement; The semantic query statement is input into the semantic corrector for correction in order to obtain the target semantic query statement; The target semantic query statement is converted into a structured query statement using a semantic translator; The structured query statement is input into the database to retrieve the query results; The step of constructing a semantic data model based on the mapping relationship and the local semantic representation includes: The mapping relationship is initialized based on the context information of the currently input natural language query; Adjust the weights of the database schema elements based on the context information; Construct an initial semantic graph, which includes entity nodes, relation nodes, operation nodes, entity relation edges, condition relation edges, and operation relation edges; For each node, a comprehensive node feature representation is generated by combining the local semantic representation, the database metadata information, and the user's historical query behavior. The features of each edge in the initial semantic graph are initialized to obtain a comprehensive edge feature representation; The integrated node feature representation and the integrated edge feature representation are embedded into the initial semantic graph, and the initial semantic graph is dynamically updated based on the sink algorithm for implementing the connector to obtain the input semantic graph; The input semantic graph is fed into a pre-constructed initial graph neural network for training. The information of neighboring nodes is aggregated through graph convolution operations in each layer of the initial graph neural network, and the state of each node is updated by combining the features of each node. The initial graph neural network was optimized using the Adam optimizer to obtain a converged semantic data model.

2. The data analysis method according to claim 1, characterized in that, The prompt is encoded using a pre-trained model to obtain an encoded vector for each token, including: The prompt is converted into the input format of the pre-trained model to obtain input_ids and attention_mask; The input_ids and attention_mask are input into the pre-trained model to obtain the encoding vector for each token.

3. The data analysis method according to claim 1, characterized in that, By identifying entities and attributes in the natural language and mapping them to schema elements in the database to obtain mapping relationships, including: Extract key entities from the natural language; Query the database schema information to obtain the schema elements of the database; Match the key entities with the schema elements to obtain the mapping relationship.

4. The data analysis method according to claim 1, characterized in that, The semantic query statement is input into a semantic corrector for correction to obtain the target semantic query statement, including: Construct an initial semantic corrector; Determine whether the semantic query statement conforms to the pre-configured correction rules; If not, the semantic query statement is input into a pre-configured machine learning model to obtain correction suggestions; Determine whether the query conditions in the semantic query statement exceed three; If so, the semantic query statement is decomposed to obtain a single query condition statement; Based on the proposed corrections, the individual query condition statements are semantically corrected to obtain the corrected query statements. Evaluate the query statement and obtain the evaluation results; The pre-configured machine learning model and the correction rules are improved based on the evaluation results to obtain a target semantic corrector; The semantic query statement is input into the target semantic corrector for correction in order to obtain the target semantic query statement.

5. The data analysis method according to claim 1, characterized in that, After converting the target semantic query statement into a structured query statement using a semantic translator, the method further includes: The rules engine determines whether the structured query statement contains any errors. If so, the structured query statement is input into a pre-configured deep learning model to obtain the statement error type; Determine whether the error type of the statement is an incorrect field name; If not, prompt the user to make a selection to obtain the result; The structured query statement is adjusted based on the selection result to obtain the target structured query statement.

6. The data analysis method according to claim 5, characterized in that, The structured query statement is input into the database to retrieve query results, including: The target structured query statement is input into the database to retrieve query data; Select a data chart based on the data type of the query data; The query data and the data charts are visualized to obtain the query results.

7. The data analysis method according to claim 6, characterized in that, Select a data chart based on the data type of the query data, including: Extract the statistical features of the query data; The statistical features are input into a pre-configured recognition model to obtain the distribution type; Generate data charts based on the distribution type.

8. A data analysis system, characterized in that, The system includes: The acquisition unit is used to acquire natural language input from the user. The first building unit is used to construct the initial data model; The second construction unit is used to perform word segmentation on the natural language based on a pre-configured industry knowledge base and a WordPiece word segmenter to construct a prompt. The encoding unit is used to encode the prompt using a pre-trained model to obtain an encoding vector for each token. The first processing unit is used to take the encoded vector of the first classification token output by the pre-trained model as the representation of the entire sentence; The second processing unit is used to process all the encoded vectors according to the sentence representation to obtain a local semantic representation; The third processing unit is used to identify entities and attributes in the natural language and map the entities and attributes to schema elements in the database to obtain mapping relationships; The third construction unit is used to construct a semantic data model based on the mapping relationship and the local semantic representation; The fourth building unit is used to combine the semantic data model with custom rules to build a semantic parser; The fourth processing unit is used to input the natural language into the semantic parser for parsing to obtain a semantic query statement; The fifth processing unit is used to input the semantic query statement into the semantic corrector for correction in order to obtain the target semantic query statement; The sixth processing unit is used to convert the target semantic query statement into a structured query statement using a semantic translator; The query unit is used to input the structured query statement into the database for querying to obtain query results; The third building unit is specifically used for: The mapping relationship is initialized based on the context information of the currently input natural language query. Adjust the weights of the database schema elements based on the context information; Construct an initial semantic graph, which includes entity nodes, relation nodes, operation nodes, entity relation edges, condition relation edges, and operation relation edges; For each node, a comprehensive node feature representation is generated by combining the local semantic representation, the database metadata information, and the user's historical query behavior. The features of each edge in the initial semantic graph are initialized to obtain a comprehensive edge feature representation; The integrated node feature representation and the integrated edge feature representation are embedded into the initial semantic graph, and the initial semantic graph is dynamically updated based on the sink algorithm for implementing the connector to obtain the input semantic graph; The input semantic graph is fed into a pre-constructed initial graph neural network for training. The information of neighboring nodes is aggregated through graph convolution operations in each layer of the initial graph neural network, and the state of each node is updated by combining the features of each node. The initial graph neural network was optimized using the Adam optimizer to obtain a converged semantic data model.

9. A data analysis system, characterized in that, The system includes: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor invokes to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, the program performing the method as claimed in any one of claims 1 to 7 when executed on a computer.

Images