395 results about "Query plan" patented technology

The invention provides a Dores front-end node-based service proxy method and system, and relates to the technical field of database access request processing, and the method comprises the steps: collecting front-end node performance indexes, extracting node performance characteristics through employing multi-head recursive attention and causal convolution solution, extracting inter-node correlation characteristics through combining graph attention convolution, and obtaining a database access request; the method comprises the following steps: constructing a node state vector, optimizing a multi-layer connection pool parameter by utilizing an online learning algorithm, establishing a node incidence matrix based on comparative learning, constructing a node performance probability graph evolution model through variational reasoning and Monte Carlo sampling, predicting a node performance change trend, and receiving a database access request; and generating a query plan by adopting an edge attention message passing mechanism, carrying out resource evaluation in combination with performance trend prediction, carrying out scheduling decision making by utilizing multi-agent reinforcement learning and Monte Carlo tree search, carrying out fault detection and migration according to node performance deviation, and updating a probability graph evolution model.

A method is provided for caching a query submitted to a graph data storage system, including receiving a query from a query caller directed to the graph data storage system, and examining the query, via a cache suitability determination engine, to determine if the query is suitable for caching under a first graph query caching policy based on the query caller being a trusted caller and / or the query scenario being a trusted query scenario. If suitable, generating a query execution plan for the query via a graph query plan generating engine and examining the query plan to determine if the query plan is suitable for caching under the first policy. If not suitable, caching the query under a second or a third graph caching policy based on the geographic location of a memory system on which the query plan is stored, and providing the query execution plan.

The invention provides an operator unloading method and system of a vectorization execution engine based on a DPU heterogeneous architecture, the DPU heterogeneous architecture comprises a CPU and a DPU network card, and the DPU heterogeneous architecture is applied to an open source distributed computing system; the open source distributed computing system performs grammar analysis on a query language input by a user and adds physical execution stage operation to obtain a physical execution plan, and the Spark plug-in intercepts the physical execution plan and converts the physical execution plan into a query plan based on a set rule; transmitting to a vectorization engine execution path and calling a vectorization execution engine to carry out data processing on a plurality of operators in the query plan; a performance function of a DPU heterogeneous calculation version in a function system of the vectorization execution engines is called, calculation resource consumption of operators capable of being processed by the multiple vectorization execution engines in the query plan is calculated, the operators higher than a preset standard are unloaded to a DPU network card from an original CPU, and the data processing performance of the query task is improved through operator unloading.

A computer-based system and corresponding computer-implemented method process input data queries in a manner enabling advanced functionality for data analytics. The method transforms a query plan tree into a query strategy tree. The query plan tree is constructed from an input data query associated with a computation workload. The method compiles the query strategy tree into dataflow graph(s) (DFG(s)). The method transmits the DFG(s) for execution via a virtual platform. The method monitors execution of the DFG(s). The method outputs, based on a result of the execution monitored, a response to the input data query. The result is received from the virtual platform and represents at least one computational result of processing the computation workload by the virtual platform. The system and method enable rapid and efficient retrieval and analysis of data stored in data lakes and other data storage systems, in response to user queries.

Techniques and solutions are provided for improved query optimization, including for sub-portions of a query plan. A query is submitted to an inline query optimizer and at least one auxiliary query optimizer. The query is optimized by the inline query optimizer and the auxiliary query optimizer. A query processor can evaluate costs associated with query plans produced by the inline query optimizer and the auxiliary query optimizer and select a plan for execution that is most performant.

Dynamic selection of topologies for database data may be performed for performing queries. Different possible query plans for different assignments of topologies are determined for a query. One of the query plans is selected according to a cost-based model that assigns costs to different uses of the different topologies. The query is then performed using the selected query plan.

The present disclosure provides techniques and solutions for improved query optimization. A query plan is received, and at least a portion of the query plan is identified to be analyzed for logically equivalent query plans. A signature is generated for the at least a portion of the query plan. One or more query plans are identified that have signature that matches the signature of the at least a portion of the query plan, but where such query plans are logically equivalent, but not identical, to the at least a portion of the query plan. A query plan of the one or more query plans is substituted in the query plan for the at least a portion of the query plan.

The embodiment of the invention provides a computing resource configuration method and device of FPGA equipment and the FPGA equipment. The method comprises the steps that a heterogeneous query execution plan is generated according to a query request of a user, and the heterogeneous query plan comprises heterogeneous operators and non-heterogeneous operators; and determining a first number of resource blocks required to be configured for executing the heterogeneous operator based on the calculation throughput for processing a non-heterogeneous operator adjacent to the heterogeneous operator and the calculation throughput which can be provided by a single resource block in a programmable array logic FPGA (Field Programmable Gate Array) device. Through the embodiment of the invention, the problems that the throughput of the FPGA equipment is not matched with the calculation throughput of the CPU and the resources of the FPGA equipment cannot be reasonably utilized in the current dynamic configuration scheme in the related technology are solved, and the effect of ensuring that the resources of the FPGA equipment are reasonably utilized in the query acceleration process of the database is achieved.

A future workload may be predicted for a database system using analysis of queries submitted for execution. A feature vector for a query may be determined according to a query plan for the query. If the feature vector has not been previously seen, or has not been sufficiently seen, by the database system, a machine learning inference may be used to predict performance characteristics of the query, the machine learning system trained using previous feature vectors and performance characteristics of executed queries. If the feature vector has been sufficiently seen previously by the database system, a history of performance characteristics of previous queries with similar or the same feature vector may be used to predict the performance characteristics. The predictions may then be used to configure or reconfigure processing cluster(s) of the database system.

The invention provides a data synchronization method and device based on Spring Batch and a large model. Synchronously initializing a large model micro service and a Spring Batch framework, and establishing a model service hot standby mechanism; obtaining data source features through a metadata scanning engine driven by a large model, and generating an adaptive synchronization strategy; optimizing a query plan based on the synchronization strategy, dynamically controlling data fragmentation and executing streaming preprocessing and semantic mapping conversion; a target table structure is adapted, a three-level conflict detection mechanism is adopted to eliminate data conflicts, and meanwhile transaction processing and batch writing logic are optimized; the data quality is verified through three-level verification, performance parameters are dynamically adjusted and optimized, and fault-tolerant recovery and resource elastic expansion and contraction are achieved; metadata are synchronized, an execution report is generated, knowledge base self-learning is completed based on task full-link indexes, and a data consanguinity map and a reusable processing case library are constructed. According to the scheme, diversified and dynamic enterprise data synchronization requirements can be met.

The invention discloses a database query plan optimization method and system, and belongs to the technical field of databases and artificial intelligence. AI model training: training the model by using the preprocessed historical query data according to a specific model architecture, a training mode and resource and time arrangement, and dynamically adjusting the model according to the change of the database; query analysis: evaluating the complexity of the query and the possibly involved data volume, and allocating a complexity level and a data access volume range for the query; query plan generation: generating and evaluating a plurality of candidate query plans based on prediction and query analysis results of the AI model, and selecting an optimal query plan as a final execution scheme; and performing query execution and monitoring. According to the method, the query execution time can be greatly shortened, and the database response speed is increased; system resources are utilized more effectively; dynamic changes of the database can be adapted, and the query plan is automatically updated according to structure and data distribution adjustment.

Systems and methods provide for managing changes to federated systems. In accordance with some aspects, a change to a federated system is received. A historical query is accessed from a query store storing information regarding queries received by the federated system. A query plan evaluation is performed for the historical query based on the change to the federated system, and an action is caused to be performed for the change based on the query plan evaluation.

The invention discloses a communication network multi-source big data analysis method based on distributed database collaboration, and relates to the technical field of communication data analysis, and the method comprises the steps: 1, generating a session contract number, a table snapshot number, a flow site set and a transaction time point at the start of a session, registering a collaborative metadata table, judging semantic compatibility, and setting a state; step 2, pushing down three anchors to a lake-warehouse, a message bus and a transaction library according to the serial number, performing freezing reading according to the anchors to obtain a lake-warehouse segment, a streaming segment and a transaction segment, performing alignment extraction on a joint feature set in the step 3, generating output through a deterministic operator chain, and storing the extracted joint feature set in a database; the method comprises the following steps: step 1, setting a session contract number, a field source and an operator relationship, reading an anchor code as a blood relationship code and writing the blood relationship code into a result table, and step 2, resetting three anchors and multiplexing a query plan during reproduction verification and cross-domain migration, and integrating blood relationship equivalence, result equivalence, freezing consistency, registration difference and migration fingerprints to realize session-level consistent reading, recalculation and auditing.

The invention relates to the technical field of data processing, and particularly provides a heterogeneous data source query method, system and device based on a Dores database and a medium, and the method comprises the following steps: obtaining a query text submitted by a user, and analyzing query parameters of the query text; obtaining metadata of each data source, and generating a plurality of candidate global logic plans based on the metadata and the query parameters; screening a global optimal query plan from the plurality of candidate global logic plans by using a cost model; and converting the global optimal query plan into a federated query statement supported by Dores, and executing the federated query statement by using a Dores federated query function to obtain target data. According to the method, a plurality of candidate global logic plans are automatically generated based on heterogeneous data source metadata and query parameters, and the optimal plan is accurately screened by using the cost model, so that the execution efficiency of cross-source query is remarkably improved, and low-efficiency scanning and network transmission bottlenecks are avoided.

The invention discloses an HTAP database query optimization method and device and a storage medium. The query performance is remarkably improved by combining the advantages of row storage and column storage. Comprising the steps of analyzing SQL query to generate a logic query plan; evaluating the execution cost of each logic operator in different storage modes by using a mixed cost model, wherein hyper-parameters of the mixed cost model are optimized through Bayesian search; selectively adopting a dynamic programming algorithm or a greedy algorithm to generate a hybrid physical query plan with the minimum execution cost according to the table number; and executing the plan to obtain a query result. In addition, an MCTS algorithm is further utilized, the optimal column subset is selected in the preset memory budget limit based on the query historical record and loaded into the memory column storage, and the subsequent query performance is further optimized. The problem that row storage and column storage in an existing HTAP system cannot work cooperatively is solved, operator-level mixed execution is achieved, and meanwhile memory resource utilization is optimized.

The present disclosure involves systems, software, and computer implemented methods for managing database query plans with dynamic conditions. One example method includes receiving a request to process a query based on at least one dynamic condition. A serialized abstract query plan is identified, deserialized, and converted into a query optimizer tree. A condition provider procedure is invoked to generate a first filter string for each view in the query optimizer tree. Comparisons are made to determine matches between first filter strings and corresponding second filter string stored in the query optimizer tree. In response to determining that a first filter string does not match the corresponding second filter string, query plan compilation is triggered. In response to determining that each filter string matches each corresponding second filter string, a query plan is generated based on the converted query optimizer tree and the query is executed using the query plan.

Discloses are a method for detecting a scheduling deadlock in data query. A query plan tree generated by a query statement includes a common temporary table production operator node, a common temporary table consumption operator node, and a hash join operator node. At least one right table chain in the tree is determined. Operator nodes in the right table chain are traversed from bottom to top starting from the leaf node in the right table chain, and the other operator nodes in the tree are traversed, to analyze an execution order relationship between the operator nodes traversed in the tree. When it is found based on the execution order relationship that there are two identical common temporary table consumption operator nodes in the tree that have scheduling dependency, and their upstream operators are the same common temporary table production operator, it is determined that there is a scheduling deadlock.

The invention discloses a multi-language data analysis-oriented extensible system, method, equipment and medium, and relates to the technical field of multi-language analysis, the multi-language data analysis-oriented extensible system comprises a grammar analysis module used for analyzing a query language and generating a language-specific syntax tree structure after analysis; the grammar adaptation module is used for converting a language-specific grammar tree structure into an abstract grammar tree AST in a uniform format; the abstract syntax tree processing module injects a permission control strategy, a field desensitization rule and an alarm exception expression into the AST; the business processing module is used for executing business-related rule check and label injection; the query plan generation module is used for constructing a logic query path and a corresponding distributed execution scheme; and the task scheduling module issues the distributed execution scheme to the corresponding execution node. According to the method, by constructing a unified multi-language grammar analysis framework and a standardized abstract syntax tree AST representation mechanism, compatibility and fusion of multiple query languages such as SQL, SPL and natural language are achieved.

The invention relates to the technical field of data query, in particular to a database query optimization method and device, equipment and a medium. The method comprises the following steps: acquiring a query plan corresponding to a query statement, applying available resources for executing the query plan by the query statement in the process of executing the query plan, if the application fails, acquiring N execution paths corresponding to continuing to execute the query plan, N being an integer greater than zero, evaluating the execution cost of the query plan under each execution path, the execution cost of each execution path is obtained, the execution path corresponding to the minimum execution cost value is selected as the optimal execution path, and the query plan is executed according to the optimal execution path. According to the method and the device, when resources are insufficient, the execution cost of the query statement under different execution paths is evaluated, the execution path with the minimum execution cost is determined as the optimal path, the query statement is executed by using the optimal path, and the execution efficiency is improved.

The present disclosure involves systems, software, and computer implemented methods for using worst-case predicate selectivity for database query plans. One example method includes identifying a query that includes an initial set of parameter values for a set of predicates. An order of predicate evaluation is determined for a query plan by determining a worst-case selectivity for each predicate and ordering the predicates based on the worst-case selectivities. A second instance of the query is received that includes a second set of parameter values. The second instance of the query is executed, to generate a query result, using the query plan and the second set of parameters by evaluating the set of predicates based on the second set of parameters and the order of predicate evaluation determined based on the worst-case selectivities. The query result is provided in response to the second instance of the query.

The invention discloses a data query method, device and equipment for a distributed database, a storage medium and a program product, and the method comprises the steps: building a query plan evaluation model based on the data transmission time delay between every two working nodes in a distributed cluster, and the computing power resource and data storage layout information of each working node; under the condition that a query request is received, calling a query plan evaluation model, respectively evaluating a plurality of query plans generated based on the query request, and determining a target query plan from the plurality of query plans based on an evaluation result; determining at least one target working node of the to-be-queried data in the distributed cluster; and on the basis of a preset push-down strategy, pushing down each query operator contained in the target query plan to each target working node where the to-be-queried data is located, so that each target working node executes each query operator on the basis of the target query plan to obtain a data query result. By adopting the method, the data query efficiency of the distributed database can be improved.

The invention belongs to the field of artificial intelligence and big data analysis, and particularly relates to an AI-based drive type data analysis method and system.The method comprises the steps that S1, on the basis of natural language query input by a user, candidate SQL and / or DSL statements are generated through a pre-trained big language model; s2, generating a parallel query plan and optimizing the parallel query plan based on the candidate SQL and / or DSL statements and the metadata knowledge graph; s3, extracting authorization data; s4, fusing the extracted data to generate an analysis feature wide table, and identifying key features, abnormal modes and text abstracts in the analysis feature wide table; s5, executing controlled retrieval in the business knowledge graph based on the key features and the abnormal mode, and generating a security evidence sub-graph; s6, inputting the text abstract of the analysis feature wide table and the security evidence sub-graph into an AI inference engine, and outputting a structured report; and S7, displaying the structured report. The problems of high technical threshold and weak safety and interpretability in a data analysis process are solved.

The subject technology receives a query. The subject technology generates a set of query blocks based on parsing the query. The subject technology stores query block metadata for each query block from the set of query blocks. The subject technology restores a set of logical query block boundaries. The subject technology performs a hash-based query block matching. The subject technology generates, after performing the hash-based query block matching, a final query plan. In an example, if a (same) query is executed again the same logical query blocks would produce the same metadata (e.g., identifier, name and hash). This metadata can be used to match logical query blocks between multiple executions of the same query.

The invention provides a multi-data-source-oriented federal query method and device, equipment and a medium, and the method comprises the steps: obtaining a data analysis request of a user, converting a data query rule into a relational algebraic node format according to configuration item information and metadata information, obtaining at least one target query plan, and sending the at least one target query plan to a server; generating a query request corresponding to the data source according to the data query parameter of the at least one target query plan to query a corresponding underlying database to obtain target data, receiving the target data of each data source, preprocessing each target data, and performing statistical analysis according to the preprocessed target data to obtain a statistical analysis result; the preprocessing comprises data fusion and alignment; according to the method, the isomerism of different underlying databases can be shielded, the query process oriented to multiple data sources is simplified, the problems of complex cross-data-source analysis process and low efficiency can be effectively solved, and subsequent statistical analysis requirements can be supported by fusing and aligning the queried target data.

The invention discloses a statistical data downloading method for supply chain finance, which relates to the technical field of finance and comprises an acceptance permission judgment module, a query resource control module and a desensitization packaging evidence storage module. The acceptance permission judgment module is used for uniformly accessing the statistical application, completing identity verification, main body range binding, index range binding, time range binding and area range binding, and generating an acceptance bill and a judgment bill; the query resource control module is used for generating a query plan under the index aperture mapping dictionary, the partition information and the resource quota, binding a concurrent quota, a throttling quota and a priority queue, and outputting an executable instruction stream; and the desensitization packaging evidence storage module is used for executing differential desensitization according to the main body type and the sensitivity level, performing aggregation calculation and deduplication execution, completing result set packaging, digital watermarking and downloading delivery, generating a downloading certificate and writing a certificate abstract and an evidence index into an evidence storage chain. The method has the characteristic of high consistency.

In some implementations, there is provided pipeline bypassing of certain operators. In some implementations, a method includes generating a query plan including at least one pipeline of operators; determining whether the at least one pipeline of operators includes a first operator that requires a complete result set as an input and further includes a second operator that supports providing the complete result set using a state identified by a state reference; and in response to determining the at least one pipeline of operators includes the first operator that requires the complete result set as the input and further includes the second operator that supports providing the complete result set using the state identified by the state reference, bypassing in the query plan pipelining between the first operator and the second operator.

A method includes receiving a database query requesting a database to conditionally return one or more data blocks. The database is stored on memory hardware in communication with the data processing hardware and the database query includes a plurality of parameters characterizing the database query. The method includes generating a set of query plans. Each query plan in the set of query plans is configured to execute the database query using a different order of operations. The method includes training a model using historical database queries and generating, using the trained model, a query plan score for each query plan in the set of query plans. The method includes selecting, using the query plan score of each query plan in the set of query plans, a query plan from the set of query plans. The method also includes executing the database query using the selected query plan.

A system may include a plurality of processing nodes in communication with a storage device configured to store a plurality of data. The processing nodes may receive a query on at least a portion of the data and may generate a query plan in natural language format. The processing nodes may generate a large language model (“LLM”) input based on the natural language format of the query plan and may execute an LLM on the LLM input. The processing nodes may generate, in response to execution of the LLM, a plurality of recommended actions to perform to improve the query plan. The processing nodes may receive input to execute at least one of the plurality of recommended actions and may alter the query plan in accordance with the at least one of the plurality of recommended actions. A method and computer-readable medium are also disclosed.

The present disclosure involves systems, software, and computer implemented methods for managing database query plans with dynamic conditions. One example method includes receiving a request to process a query based on a dynamic condition. In response to determining that a plan cache does not include a cached query plan for the query, query plan compilation is triggered to generate a query plan. In response to determining that a cached query plan exists, for each query view, a condition provider procedure is invoked to generate a first filter string. First filter strings are compared to corresponding second filter strings in the cached plan. In response to determining that a first filter string does not match the corresponding second filter string, query plan compilation is triggered. In response to determining that each filter string matches each corresponding second filter string, the database query is executed using the cached query plan.

In some implementations, there is provided a method that includes receiving a query request including a top k query operator for query plan generation, optimization, and execution, wherein k defines a threshold limit of query results for the top k query operator; inserting into a query plan a check operator associated with the top k query operator; in response to executing the query plan, checking, by the check operator, whether an early exit occurs due to the top k query operator reaching the threshold limit; in response to the early exit occurring due to the top k query operator reaching the threshold limit, stopping processing, by the check operator, including opening of another fragment of a database table; and in response to the early exit not occurring, allowing, by the check operator, the opening of the other fragment of the database table.