233 results about "Query plan" patented technology

The invention provides a data synchronization method and device based on Spring Batch and a large model. Synchronously initializing a large model micro service and a Spring Batch framework, and establishing a model service hot standby mechanism; obtaining data source features through a metadata scanning engine driven by a large model, and generating an adaptive synchronization strategy; optimizing a query plan based on the synchronization strategy, dynamically controlling data fragmentation and executing streaming preprocessing and semantic mapping conversion; a target table structure is adapted, a three-level conflict detection mechanism is adopted to eliminate data conflicts, and meanwhile transaction processing and batch writing logic are optimized; the data quality is verified through three-level verification, performance parameters are dynamically adjusted and optimized, and fault-tolerant recovery and resource elastic expansion and contraction are achieved; metadata are synchronized, an execution report is generated, knowledge base self-learning is completed based on task full-link indexes, and a data consanguinity map and a reusable processing case library are constructed. According to the scheme, diversified and dynamic enterprise data synchronization requirements can be met.

The invention discloses a communication network multi-source big data analysis method based on distributed database collaboration, and relates to the technical field of communication data analysis, and the method comprises the steps: 1, generating a session contract number, a table snapshot number, a flow site set and a transaction time point at the start of a session, registering a collaborative metadata table, judging semantic compatibility, and setting a state; step 2, pushing down three anchors to a lake-warehouse, a message bus and a transaction library according to the serial number, performing freezing reading according to the anchors to obtain a lake-warehouse segment, a streaming segment and a transaction segment, performing alignment extraction on a joint feature set in the step 3, generating output through a deterministic operator chain, and storing the extracted joint feature set in a database; the method comprises the following steps: step 1, setting a session contract number, a field source and an operator relationship, reading an anchor code as a blood relationship code and writing the blood relationship code into a result table, and step 2, resetting three anchors and multiplexing a query plan during reproduction verification and cross-domain migration, and integrating blood relationship equivalence, result equivalence, freezing consistency, registration difference and migration fingerprints to realize session-level consistent reading, recalculation and auditing.

The invention relates to the technical field of data processing, and particularly provides a heterogeneous data source query method, system and device based on a Dores database and a medium, and the method comprises the following steps: obtaining a query text submitted by a user, and analyzing query parameters of the query text; obtaining metadata of each data source, and generating a plurality of candidate global logic plans based on the metadata and the query parameters; screening a global optimal query plan from the plurality of candidate global logic plans by using a cost model; and converting the global optimal query plan into a federated query statement supported by Dores, and executing the federated query statement by using a Dores federated query function to obtain target data. According to the method, a plurality of candidate global logic plans are automatically generated based on heterogeneous data source metadata and query parameters, and the optimal plan is accurately screened by using the cost model, so that the execution efficiency of cross-source query is remarkably improved, and low-efficiency scanning and network transmission bottlenecks are avoided.

The invention discloses a multi-language data analysis-oriented extensible system, method, equipment and medium, and relates to the technical field of multi-language analysis, the multi-language data analysis-oriented extensible system comprises a grammar analysis module used for analyzing a query language and generating a language-specific syntax tree structure after analysis; the grammar adaptation module is used for converting a language-specific grammar tree structure into an abstract grammar tree AST in a uniform format; the abstract syntax tree processing module injects a permission control strategy, a field desensitization rule and an alarm exception expression into the AST; the business processing module is used for executing business-related rule check and label injection; the query plan generation module is used for constructing a logic query path and a corresponding distributed execution scheme; and the task scheduling module issues the distributed execution scheme to the corresponding execution node. According to the method, by constructing a unified multi-language grammar analysis framework and a standardized abstract syntax tree AST representation mechanism, compatibility and fusion of multiple query languages such as SQL, SPL and natural language are achieved.

The invention belongs to the field of artificial intelligence and big data analysis, and particularly relates to an AI-based drive type data analysis method and system.The method comprises the steps that S1, on the basis of natural language query input by a user, candidate SQL and / or DSL statements are generated through a pre-trained big language model; s2, generating a parallel query plan and optimizing the parallel query plan based on the candidate SQL and / or DSL statements and the metadata knowledge graph; s3, extracting authorization data; s4, fusing the extracted data to generate an analysis feature wide table, and identifying key features, abnormal modes and text abstracts in the analysis feature wide table; s5, executing controlled retrieval in the business knowledge graph based on the key features and the abnormal mode, and generating a security evidence sub-graph; s6, inputting the text abstract of the analysis feature wide table and the security evidence sub-graph into an AI inference engine, and outputting a structured report; and S7, displaying the structured report. The problems of high technical threshold and weak safety and interpretability in a data analysis process are solved.

The invention discloses a statistical data downloading method for supply chain finance, which relates to the technical field of finance and comprises an acceptance permission judgment module, a query resource control module and a desensitization packaging evidence storage module. The acceptance permission judgment module is used for uniformly accessing the statistical application, completing identity verification, main body range binding, index range binding, time range binding and area range binding, and generating an acceptance bill and a judgment bill; the query resource control module is used for generating a query plan under the index aperture mapping dictionary, the partition information and the resource quota, binding a concurrent quota, a throttling quota and a priority queue, and outputting an executable instruction stream; and the desensitization packaging evidence storage module is used for executing differential desensitization according to the main body type and the sensitivity level, performing aggregation calculation and deduplication execution, completing result set packaging, digital watermarking and downloading delivery, generating a downloading certificate and writing a certificate abstract and an evidence index into an evidence storage chain. The method has the characteristic of high consistency.

A system may include a plurality of processing nodes in communication with a storage device configured to store a plurality of data. The processing nodes may receive a query on at least a portion of the data and may generate a query plan in natural language format. The processing nodes may generate a large language model (“LLM”) input based on the natural language format of the query plan and may execute an LLM on the LLM input. The processing nodes may generate, in response to execution of the LLM, a plurality of recommended actions to perform to improve the query plan. The processing nodes may receive input to execute at least one of the plurality of recommended actions and may alter the query plan in accordance with the at least one of the plurality of recommended actions. A method and computer-readable medium are also disclosed.

In some embodiments, there is provided generating a logical query plan for a query that includes a single fact table and multiple reducer tables; optimizing the logical query plan for query execution; and executing the optimized logical query plan.

IO scheduling method and device operating the same, are provided. A device connected to a host device through a CXL protocol, the device comprises a device memory which stores a query plan table and query plan status information synchronized with the host device and a device processor which processes a task according to the query plan table and updates query plan status information when receiving an operation command from the host.

The invention discloses a data query method and device based on a natural language. The method comprises the following steps: receiving natural language input of a user; performing semantic analysis on the natural language input, and determining a user intention and a plurality of semantic elements; based on the user intention, target query operators corresponding to the semantic elements respectively are determined in a query operator sequence, and the query operator sequence comprises a plurality of query operators arranged according to the execution sequence; generating a query plan based on the target query operators, the semantic elements corresponding to the target query operators and the execution sequence of the multiple target query operators; a query statement is generated based on the query plan and target database object identifiers corresponding to the semantic elements respectively, and the target database object identifiers comprise at least one of table names and field names; and performing data query based on the query statement. According to the embodiment of the invention, the accuracy and the interpretability of the query statement can be improved, so that the accuracy and the reliability of the query result are improved.

Examples detect equivalent subexpressions within a computational workload. Examples include converting a query plan tree associated with a first subexpression into a matrix. The first subexpression is a portion of a database query from the computational workload. Each node in the query plan tree is represented as a row of the matrix. The matrix is converted into a first vector. The first subexpression is determined to be equivalent to a second subexpression by comparing the first vector to a second vector associated with the second subexpression. The comparison includes computing a distance between the first and second vectors that is lower than a distance threshold. The computational workload is modified, based on the determining, to perform the first subexpression and exclude performance of the second subexpression as duplicative.

The invention discloses a cardinal number estimation method based on a bidirectional long-short term memory network and ensemble learning, and aims to solve the limitation of a traditional cardinal number estimation method and an existing learning type method in accuracy and efficiency. According to the method, the structure, connection, operation and filtering condition information of the query plan tree is efficiently extracted through the four sub-encoders, the extracted feature sequence is compressed by using the bidirectional long-short-term memory network, the context dependency relationship between the nodes is effectively captured, and the model learning difficulty is reduced. Besides, a Bayesian neural network is introduced to be combined with an active learning strategy to screen and construct a plurality of high-value training data subsets, and a robust integrated model is trained on the basis to be used for cardinality estimation. According to the method, the cardinality estimation accuracy is remarkably improved on multiple data sets, and when complex query and multi-table connection scenes are processed, the method shows better comprehensive performance compared with other methods.

A method of incorporating a bloom filter (BF) into cost-based bottom-up query optimization includes: receiving a query including a request to join at least two data tables that include a first data table and a second data table; generating, in response to the request, at least two query plans that includes a first query plan including subplans to: create the BF based on the second data table, apply the BF to the first data table for filtering data of the first data table before the first data table is joined with another data table, and join the first data table with the another data table; submitting the at least two query plans to a cost-based bottom-up optimizer to obtain a target query plan; and providing, as a response to the received query, a set of data that is retrieved from a data retrieval system by executing the target query plan.

A distributed database identifies classifications of risk associated with stages of a query plan. The distributed database generates an execution plan in which incompatible risk classifications are assigned to separate stages of an execution plan that is derived from the query plan. The stages are assigned to computing nodes for execution based, at least in part, on the risk classifications. A result for the query is generated based on execution of the stages on the assigned computing nodes.

A database execution engine initiates generation of a query plan on a projection pipeline for a received query. In response to initiating generation of the query plan, the database execution engine determines a first estimate of total work that will be parallelized by a scheduling operator for the query plan. Also, the database execution engine determines a second estimate of a final result size at the projection pipeline. Next, the database execution engine calculates a target number of rows based on a pause threshold and one or more configuration parameters. Then, the database execution engine calculates a task size upper bound by multiplying the target number of rows by the first estimate divided by the second estimate. During execution of the query plan, the database execution engine applies the task size upper bound to task sizes for result streaming threads scheduled by the scheduling operator on the projection pipeline.

The application discloses a distributed database query splitting method and device, relates to the technical field of distributed databases, and applies distributed data and comprises the following steps: S1, data sharding storage: table data sharding storage: respectively existing on multiple nodes, selecting hash distribution or random distribution; S2, query splitting steps; comprising: S2.1, generating a query execution plan; S2.2, splitting data into data slots, when a client is connected to a data slot service of a specified number, reading to a data transmission channel; S2.3. Grouping data slots: the coordinator operation in the query plan obtains data from the specified data slot; S3, cluster expansion: expanding the distributed database cluster. The mechanism of the data slot is used to unbond the redistribution data and the nodes and only associate the data slot. Because the data slot can be associated to any node in the cluster, the optimal node can be selected according to the machine load to execute a task.

A database system is operable to execute a query operator execution flow via a plurality of nodes each assigned to participate in a corresponding level of a hierarchical query plan. A first subset of nodes participating in at least one lower level of the hierarchical query plan generate a plurality of first output. A second node participating in at least one upper level of the hierarchical query plan generates second output based on processing the plurality of first output. The second output is segregated into a plurality of second output portions, and the plurality of second output portions are dispersed across the first subset of nodes for processing. The first subset of nodes generates a plurality of third output based on processing corresponding second output portions of the plurality of second output portions. The second node generates fourth output based on processing the plurality of third output.

Provided herein are systems and methods for hash-join broadcast decision making. For example, a method includes generating a query plan for a received query. The query plan includes a plurality of join operations with a plurality of hash-join-build (HJB) operations and a plurality of hash-join-probe (HJP) operations. A decision node of a plurality of decision nodes of the query plan is configured as a primary decision node. Build-side data information associated with build-side data and received from the plurality of HJB operations is decoded by the primary decision node. A data distribution method is determined by the primary decision node for each HJB operation of the plurality of HJB operations based on the build-side data information. The query plan is executed based on distributing the build-side data to the plurality of HJP operations using the data distribution method for each HJB operation of the plurality of HJB operations.

The invention discloses a query intention understanding and execution path optimization system and method based on a multi-modal deep learning model, and the method comprises the steps: a multi-modal intention analysis module is used for receiving unstructured query input from a user, and uniformly extracting the unstructured query input as a standardized SQL; the cross-model intention recognition module is used for receiving the standardized SQL output by the multi-mode intention analysis module, recognizing a data entity, an operation type and a data model to which the data entity and the operation type relate in a query intention, and outputting an entity structure table and a query plan tree; the access mode analysis module is used for using the query plan tree, identifying a connection relationship between node objects in the tree by contrasting with an entity structure table, performing cost estimation on access of each node in the tree, and constructing a logic plan tree; and the path optimization generation module is used for executing generation, evaluation and screening of candidate execution paths on the basis of the logic plan tree output by the analysis module, and finally determining an optimal execution plan. According to the method, the problems of semantic analysis, cross-model query intention recognition and execution path intelligent selection of complex inputs such as natural languages in a multi-mode database can be solved, and the query accuracy and the execution efficiency are improved.

The invention discloses an enterprise-level knowledge base information query system and method based on HMAC, and relates to the technical field of information security, the system comprises a client, an authentication module, a large language interaction module and a knowledge base module; the client is used for acquiring an account, a password and a retrieval request of a user; the authentication module is connected with the client and is used for receiving the account and the password and generating a user token; the large language interaction module is connected with the client and used for receiving the retrieval request, generating a query plan and signing the query plan and the user token to obtain a first signature packet; the knowledge base module is connected with the client and is used for receiving the first signature packet and signing again after verification is passed to obtain a second signature packet; and the big language interaction module is also used for receiving the second signature packet, and generating a query result through a second big language model in the big language interaction module after the verification is passed. According to the invention, the security of information transmission between the client and the knowledge base can be improved.

A computer implemented method can detect performance regression of executing a query using a current query plan. Responsive to detecting the performance regression, the method can generate an alternative query plan as a candidate solution for resolving the performance regression, start a test execution session in a designated execution thread of a query processing engine to execute the query using the alternative query plan, and monitor an elapsed duration after starting the test execution session in a timer thread of the query processing engine. The timer thread is independent of the designated execution thread. Responsive to detecting that the elapsed duration exceeds a timeout value and the test execution session has not been completed, the method can terminate the test execution session in the designated execution thread. Related systems and software for implementing the method are also disclosed.

A method of intelligent query plan cache size management can be implemented. The method can measure query locality during execution of a plurality of incoming queries in a database management system. The database management system includes a query execution plan cache having a size capable of storing at least some of query execution plans generated for the plurality of incoming queries. Based on the measured query locality, the method can adjust the size of the query execution plan cache.

A distributed database keeps an abstract syntax tree (AST) and an included user code from interacting with a query engine coordinator during the performance of query execution processes. The query engine coordinator is separated into a frontend and backend, where a compiler on the frontend generates an AST and serializes it to be sent to the backend. The backend is implemented in a security environment separate from the front end. The backend deserializes the AST, sanitizes it, and generates a query plan based on the sanitized AST.

This invention relates to the field of database management technology, providing a streaming aggregation method, system, and device in a database. The method includes: establishing a mapping relationship in the database; responding to a user creating a streaming view containing aggregation operations, determining the corresponding final calculation function based on the mapping relationship, and creating a materialized table for storing intermediate aggregation states; rewriting the streaming view as a query that executes the final calculation function on the materialized table; responding to data being written to the streaming table, generating a first query plan based on the original query of the streaming view, generating a second query plan and a third query plan based on the mapping relationship, and executing them sequentially to first obtain the intermediate states of the current write operation, then merging the current write intermediate states with the existing intermediate states in the materialized table and updating the materialized table; responding to a user querying the streaming view, executing the rewritten streaming view, obtaining and returning the aggregation result. This invention can fully implement streaming aggregation functionality without modifying the database kernel.

In some implementations, there is provided receiving a query request including a top k query operator for query plan generation, optimization, and execution; generating a query plan that includes at least one pipeline of a plurality of operators, wherein the at least one pipeline is associated with a directed acyclic graph; detecting in the query plan a first operator in the at least one pipeline that causes an early exit; and in response to the early exit by the first operator during query execution, processing back through the directed acyclic graph to identify at least one preceding operator that should or should not run given the early exit of the first operator.

The application discloses a generation method and device for database-based remote subquery pushdown execution plan. In view of the defect that the existing postgres_fdw cannot participate in the upper query plan generation of the subquery path, resulting in the intermediate result being forced to pull back to the local, a new callback interface is introduced in the FDW framework, so that the FDW plug-in can intervene in the subquery planning in the path generation stage; the layered security check is used to ensure the correctness of the pushdown semantics; the FDW state is inherited based on the subquery path abstract information, and the remote scanning path is generated; the subquery is embedded into the upper remote SQL through remote SQL reverse analysis reconstruction; the JOIN path retry optimization is combined with the EPQ path compatibility check, and the path matching problem in different connection sequences and concurrent update scenarios is solved. The application realizes the overall pushdown of the subquery to the remote execution, significantly reduces the network transmission under the premise of guaranteeing the semantic correctness, and improves the cross-database federation query efficiency.

Embodiments of the present specification provide a data query method, device, big data platform, equipment, medium and program product, wherein the data query method comprises: receiving a data query request, determining at least one data source corresponding to the data query request; determining a corresponding security query operator based on the data query request; obtaining data description information of the data source, determining a target query plan from candidate query plans of the at least one data source based on the data description information and the security query operator, and obtaining a query result corresponding to the data query request based on the target query plan, wherein the candidate query plan is used to indicate an execution order of the at least one data source under the security query operator. By using the security query operator and combining the data description information to dynamically select the target query plan, the operator execution order is optimized while ensuring data security, the efficiency bottleneck of traditional non-security scene operators in an encrypted environment is solved, and the effective improvement of query performance in a security protection scene is realized.

This invention relates to the field of intelligent document processing technology, specifically to a method and system for document processing in power transmission and transformation scenarios based on a brain-like memory mechanism. The method includes the following steps: generating a query plan based on query intent; recalling relevant documents from a hierarchically modeled power transmission and transformation document knowledge base; and extracting and reasoning information from the recalled documents using a dual-path parallel processing mechanism and a hierarchical abstraction processing mechanism; filling in a structured form according to a predefined template to generate preliminary structured output; simultaneously transforming the extracted and reasoned information into interpretable reasoning chain records; generating memory cases based on the reasoning chain records and confidence assessment results; and storing and optimizing the memory cases using a brain-like memory mechanism. This method possesses the ability to handle complex multi-step reasoning tasks, significantly improves reasoning accuracy, automates document information extraction, and greatly reduces manual processing time.

The application relates to a reasoning query re-optimization method and device based on a proxy model, equipment and a medium. Current batch data is input into a reasoning model for processing based on a first query plan to obtain data required by a query. The reasoning model comprises a proxy model and a machine learning model. Statistical information is monitored during execution of the first query plan. The statistical information comprises system resources or a query plan selection rate. When a change in the statistical information exceeds a threshold value, historical data is input into the proxy model for retraining based on a second query plan. The historical data comprises data carrying a label after being input into the reasoning model for processing before the current batch data. The method reduces the calculation overhead generated by the re-optimization reasoning query method and improves the re-optimization efficiency.