173 results about "Query plan" patented technology

The invention provides a data synchronization method and device based on Spring Batch and a large model. Synchronously initializing a large model micro service and a Spring Batch framework, and establishing a model service hot standby mechanism; obtaining data source features through a metadata scanning engine driven by a large model, and generating an adaptive synchronization strategy; optimizing a query plan based on the synchronization strategy, dynamically controlling data fragmentation and executing streaming preprocessing and semantic mapping conversion; a target table structure is adapted, a three-level conflict detection mechanism is adopted to eliminate data conflicts, and meanwhile transaction processing and batch writing logic are optimized; the data quality is verified through three-level verification, performance parameters are dynamically adjusted and optimized, and fault-tolerant recovery and resource elastic expansion and contraction are achieved; metadata are synchronized, an execution report is generated, knowledge base self-learning is completed based on task full-link indexes, and a data consanguinity map and a reusable processing case library are constructed. According to the scheme, diversified and dynamic enterprise data synchronization requirements can be met.

The invention discloses a communication network multi-source big data analysis method based on distributed database collaboration, and relates to the technical field of communication data analysis, and the method comprises the steps: 1, generating a session contract number, a table snapshot number, a flow site set and a transaction time point at the start of a session, registering a collaborative metadata table, judging semantic compatibility, and setting a state; step 2, pushing down three anchors to a lake-warehouse, a message bus and a transaction library according to the serial number, performing freezing reading according to the anchors to obtain a lake-warehouse segment, a streaming segment and a transaction segment, performing alignment extraction on a joint feature set in the step 3, generating output through a deterministic operator chain, and storing the extracted joint feature set in a database; the method comprises the following steps: step 1, setting a session contract number, a field source and an operator relationship, reading an anchor code as a blood relationship code and writing the blood relationship code into a result table, and step 2, resetting three anchors and multiplexing a query plan during reproduction verification and cross-domain migration, and integrating blood relationship equivalence, result equivalence, freezing consistency, registration difference and migration fingerprints to realize session-level consistent reading, recalculation and auditing.

The invention relates to the technical field of data processing, and particularly provides a heterogeneous data source query method, system and device based on a Dores database and a medium, and the method comprises the following steps: obtaining a query text submitted by a user, and analyzing query parameters of the query text; obtaining metadata of each data source, and generating a plurality of candidate global logic plans based on the metadata and the query parameters; screening a global optimal query plan from the plurality of candidate global logic plans by using a cost model; and converting the global optimal query plan into a federated query statement supported by Dores, and executing the federated query statement by using a Dores federated query function to obtain target data. According to the method, a plurality of candidate global logic plans are automatically generated based on heterogeneous data source metadata and query parameters, and the optimal plan is accurately screened by using the cost model, so that the execution efficiency of cross-source query is remarkably improved, and low-efficiency scanning and network transmission bottlenecks are avoided.

The invention discloses a multi-language data analysis-oriented extensible system, method, equipment and medium, and relates to the technical field of multi-language analysis, the multi-language data analysis-oriented extensible system comprises a grammar analysis module used for analyzing a query language and generating a language-specific syntax tree structure after analysis; the grammar adaptation module is used for converting a language-specific grammar tree structure into an abstract grammar tree AST in a uniform format; the abstract syntax tree processing module injects a permission control strategy, a field desensitization rule and an alarm exception expression into the AST; the business processing module is used for executing business-related rule check and label injection; the query plan generation module is used for constructing a logic query path and a corresponding distributed execution scheme; and the task scheduling module issues the distributed execution scheme to the corresponding execution node. According to the method, by constructing a unified multi-language grammar analysis framework and a standardized abstract syntax tree AST representation mechanism, compatibility and fusion of multiple query languages such as SQL, SPL and natural language are achieved.

The invention belongs to the field of artificial intelligence and big data analysis, and particularly relates to an AI-based drive type data analysis method and system.The method comprises the steps that S1, on the basis of natural language query input by a user, candidate SQL and / or DSL statements are generated through a pre-trained big language model; s2, generating a parallel query plan and optimizing the parallel query plan based on the candidate SQL and / or DSL statements and the metadata knowledge graph; s3, extracting authorization data; s4, fusing the extracted data to generate an analysis feature wide table, and identifying key features, abnormal modes and text abstracts in the analysis feature wide table; s5, executing controlled retrieval in the business knowledge graph based on the key features and the abnormal mode, and generating a security evidence sub-graph; s6, inputting the text abstract of the analysis feature wide table and the security evidence sub-graph into an AI inference engine, and outputting a structured report; and S7, displaying the structured report. The problems of high technical threshold and weak safety and interpretability in a data analysis process are solved.

The invention discloses a statistical data downloading method for supply chain finance, which relates to the technical field of finance and comprises an acceptance permission judgment module, a query resource control module and a desensitization packaging evidence storage module. The acceptance permission judgment module is used for uniformly accessing the statistical application, completing identity verification, main body range binding, index range binding, time range binding and area range binding, and generating an acceptance bill and a judgment bill; the query resource control module is used for generating a query plan under the index aperture mapping dictionary, the partition information and the resource quota, binding a concurrent quota, a throttling quota and a priority queue, and outputting an executable instruction stream; and the desensitization packaging evidence storage module is used for executing differential desensitization according to the main body type and the sensitivity level, performing aggregation calculation and deduplication execution, completing result set packaging, digital watermarking and downloading delivery, generating a downloading certificate and writing a certificate abstract and an evidence index into an evidence storage chain. The method has the characteristic of high consistency.

In some embodiments, there is provided generating a logical query plan for a query that includes a single fact table and multiple reducer tables; optimizing the logical query plan for query execution; and executing the optimized logical query plan.

The invention discloses a data query method and device based on a natural language. The method comprises the following steps: receiving natural language input of a user; performing semantic analysis on the natural language input, and determining a user intention and a plurality of semantic elements; based on the user intention, target query operators corresponding to the semantic elements respectively are determined in a query operator sequence, and the query operator sequence comprises a plurality of query operators arranged according to the execution sequence; generating a query plan based on the target query operators, the semantic elements corresponding to the target query operators and the execution sequence of the multiple target query operators; a query statement is generated based on the query plan and target database object identifiers corresponding to the semantic elements respectively, and the target database object identifiers comprise at least one of table names and field names; and performing data query based on the query statement. According to the embodiment of the invention, the accuracy and the interpretability of the query statement can be improved, so that the accuracy and the reliability of the query result are improved.

A database execution engine initiates generation of a query plan on a projection pipeline for a received query. In response to initiating generation of the query plan, the database execution engine determines a first estimate of total work that will be parallelized by a scheduling operator for the query plan. Also, the database execution engine determines a second estimate of a final result size at the projection pipeline. Next, the database execution engine calculates a target number of rows based on a pause threshold and one or more configuration parameters. Then, the database execution engine calculates a task size upper bound by multiplying the target number of rows by the first estimate divided by the second estimate. During execution of the query plan, the database execution engine applies the task size upper bound to task sizes for result streaming threads scheduled by the scheduling operator on the projection pipeline.

The application discloses a distributed database query splitting method and device, relates to the technical field of distributed databases, and applies distributed data and comprises the following steps: S1, data sharding storage: table data sharding storage: respectively existing on multiple nodes, selecting hash distribution or random distribution; S2, query splitting steps; comprising: S2.1, generating a query execution plan; S2.2, splitting data into data slots, when a client is connected to a data slot service of a specified number, reading to a data transmission channel; S2.3. Grouping data slots: the coordinator operation in the query plan obtains data from the specified data slot; S3, cluster expansion: expanding the distributed database cluster. The mechanism of the data slot is used to unbond the redistribution data and the nodes and only associate the data slot. Because the data slot can be associated to any node in the cluster, the optimal node can be selected according to the machine load to execute a task.

The invention discloses a query intention understanding and execution path optimization system and method based on a multi-modal deep learning model, and the method comprises the steps: a multi-modal intention analysis module is used for receiving unstructured query input from a user, and uniformly extracting the unstructured query input as a standardized SQL; the cross-model intention recognition module is used for receiving the standardized SQL output by the multi-mode intention analysis module, recognizing a data entity, an operation type and a data model to which the data entity and the operation type relate in a query intention, and outputting an entity structure table and a query plan tree; the access mode analysis module is used for using the query plan tree, identifying a connection relationship between node objects in the tree by contrasting with an entity structure table, performing cost estimation on access of each node in the tree, and constructing a logic plan tree; and the path optimization generation module is used for executing generation, evaluation and screening of candidate execution paths on the basis of the logic plan tree output by the analysis module, and finally determining an optimal execution plan. According to the method, the problems of semantic analysis, cross-model query intention recognition and execution path intelligent selection of complex inputs such as natural languages in a multi-mode database can be solved, and the query accuracy and the execution efficiency are improved.

The invention discloses an enterprise-level knowledge base information query system and method based on HMAC, and relates to the technical field of information security, the system comprises a client, an authentication module, a large language interaction module and a knowledge base module; the client is used for acquiring an account, a password and a retrieval request of a user; the authentication module is connected with the client and is used for receiving the account and the password and generating a user token; the large language interaction module is connected with the client and used for receiving the retrieval request, generating a query plan and signing the query plan and the user token to obtain a first signature packet; the knowledge base module is connected with the client and is used for receiving the first signature packet and signing again after verification is passed to obtain a second signature packet; and the big language interaction module is also used for receiving the second signature packet, and generating a query result through a second big language model in the big language interaction module after the verification is passed. According to the invention, the security of information transmission between the client and the knowledge base can be improved.

A computer implemented method can detect performance regression of executing a query using a current query plan. Responsive to detecting the performance regression, the method can generate an alternative query plan as a candidate solution for resolving the performance regression, start a test execution session in a designated execution thread of a query processing engine to execute the query using the alternative query plan, and monitor an elapsed duration after starting the test execution session in a timer thread of the query processing engine. The timer thread is independent of the designated execution thread. Responsive to detecting that the elapsed duration exceeds a timeout value and the test execution session has not been completed, the method can terminate the test execution session in the designated execution thread. Related systems and software for implementing the method are also disclosed.

A method of intelligent query plan cache size management can be implemented. The method can measure query locality during execution of a plurality of incoming queries in a database management system. The database management system includes a query execution plan cache having a size capable of storing at least some of query execution plans generated for the plurality of incoming queries. Based on the measured query locality, the method can adjust the size of the query execution plan cache.

This invention relates to the field of database management technology, providing a streaming aggregation method, system, and device in a database. The method includes: establishing a mapping relationship in the database; responding to a user creating a streaming view containing aggregation operations, determining the corresponding final calculation function based on the mapping relationship, and creating a materialized table for storing intermediate aggregation states; rewriting the streaming view as a query that executes the final calculation function on the materialized table; responding to data being written to the streaming table, generating a first query plan based on the original query of the streaming view, generating a second query plan and a third query plan based on the mapping relationship, and executing them sequentially to first obtain the intermediate states of the current write operation, then merging the current write intermediate states with the existing intermediate states in the materialized table and updating the materialized table; responding to a user querying the streaming view, executing the rewritten streaming view, obtaining and returning the aggregation result. This invention can fully implement streaming aggregation functionality without modifying the database kernel.

In some implementations, there is provided receiving a query request including a top k query operator for query plan generation, optimization, and execution; generating a query plan that includes at least one pipeline of a plurality of operators, wherein the at least one pipeline is associated with a directed acyclic graph; detecting in the query plan a first operator in the at least one pipeline that causes an early exit; and in response to the early exit by the first operator during query execution, processing back through the directed acyclic graph to identify at least one preceding operator that should or should not run given the early exit of the first operator.

The application discloses a generation method and device for database-based remote subquery pushdown execution plan. In view of the defect that the existing postgres_fdw cannot participate in the upper query plan generation of the subquery path, resulting in the intermediate result being forced to pull back to the local, a new callback interface is introduced in the FDW framework, so that the FDW plug-in can intervene in the subquery planning in the path generation stage; the layered security check is used to ensure the correctness of the pushdown semantics; the FDW state is inherited based on the subquery path abstract information, and the remote scanning path is generated; the subquery is embedded into the upper remote SQL through remote SQL reverse analysis reconstruction; the JOIN path retry optimization is combined with the EPQ path compatibility check, and the path matching problem in different connection sequences and concurrent update scenarios is solved. The application realizes the overall pushdown of the subquery to the remote execution, significantly reduces the network transmission under the premise of guaranteeing the semantic correctness, and improves the cross-database federation query efficiency.

Embodiments of the present specification provide a data query method, device, big data platform, equipment, medium and program product, wherein the data query method comprises: receiving a data query request, determining at least one data source corresponding to the data query request; determining a corresponding security query operator based on the data query request; obtaining data description information of the data source, determining a target query plan from candidate query plans of the at least one data source based on the data description information and the security query operator, and obtaining a query result corresponding to the data query request based on the target query plan, wherein the candidate query plan is used to indicate an execution order of the at least one data source under the security query operator. By using the security query operator and combining the data description information to dynamically select the target query plan, the operator execution order is optimized while ensuring data security, the efficiency bottleneck of traditional non-security scene operators in an encrypted environment is solved, and the effective improvement of query performance in a security protection scene is realized.

This invention relates to the field of intelligent document processing technology, specifically to a method and system for document processing in power transmission and transformation scenarios based on a brain-like memory mechanism. The method includes the following steps: generating a query plan based on query intent; recalling relevant documents from a hierarchically modeled power transmission and transformation document knowledge base; and extracting and reasoning information from the recalled documents using a dual-path parallel processing mechanism and a hierarchical abstraction processing mechanism; filling in a structured form according to a predefined template to generate preliminary structured output; simultaneously transforming the extracted and reasoned information into interpretable reasoning chain records; generating memory cases based on the reasoning chain records and confidence assessment results; and storing and optimizing the memory cases using a brain-like memory mechanism. This method possesses the ability to handle complex multi-step reasoning tasks, significantly improves reasoning accuracy, automates document information extraction, and greatly reduces manual processing time.

The application relates to a reasoning query re-optimization method and device based on a proxy model, equipment and a medium. Current batch data is input into a reasoning model for processing based on a first query plan to obtain data required by a query. The reasoning model comprises a proxy model and a machine learning model. Statistical information is monitored during execution of the first query plan. The statistical information comprises system resources or a query plan selection rate. When a change in the statistical information exceeds a threshold value, historical data is input into the proxy model for retraining based on a second query plan. The historical data comprises data carrying a label after being input into the reasoning model for processing before the current batch data. The method reduces the calculation overhead generated by the re-optimization reasoning query method and improves the re-optimization efficiency.

The application discloses a kind of query plan result dataset fast generation method, this method is first in a plurality of representative data interval actual execution query, record its total time overhead, data characteristics and the processing result of each data;Then based on these samples, construct the prediction model about data interval characteristics and total time overhead. Again by random combination generates a plurality of different data intervals, and then estimates the execution result of query on these intervals, to realize the fast low-cost generation of a large number of available data.The method of the application can greatly reduce the time-consuming of calculation while achieving similar effects to actual query execution, can produce a large number of available data in a short time, has the advantages of strong universality, simple to use and the like.In addition, the application introduces data characteristics (such as text length) and system environment configuration to refine time overhead estimation, and can generate large-scale high-quality data sets for training learning-based query optimizer.

A method for transforming structured query language queries to expedite query execution includes detecting an explode operator in a structured query language query plan that is designed to execute a structured query language query against a database, identifying a query fragment within the structured query language query plan that starts with the explode operator and ends with a second operator, and replacing the query fragment in the structured query language query plan with an optimized query fragment in which the second operator operates directly on vectors that are a target of the second operator.

The present application belongs to the field of database query optimization task, and relates to an uncertainty quantification query cost estimation method under generalization of cross-database. The uncertainty of cost estimation can be quantified, and the method can be accurately and effectively generalized to the database which has not been learned. The method mainly comprises a cross-database representation module CDR and an uncertainty cost estimation module CEU. The CDR module encodes the minimum set of cross-database query plan features, and uses Tree-LSTM to ignore the database-specific features. Then, the CEU module estimates the cost of a given set of query plans in the database and outputs the estimated uncertainty. Compared with the traditional cost estimation, the method can quantify the uncertainty of cost estimation, and can be accurately and effectively generalized to the database which has not been learned.

The application relates to the technical field of knowledge graph data processing, and discloses an electronic component supply-demand matching transaction system based on a knowledge graph, which comprises a component ontology speed labeling module, a constraint speed analysis module and a dynamic decoupling query planning module. The dynamic decoupling query planning module comprises a query cost estimation submodule. The system is used for configuring speed labels for knowledge graph data, analyzing a query into a static constraint subgraph and a dynamic constraint subgraph, and estimating the execution costs of the two subgraphs by using the query cost estimation submodule before execution. Based on the cost comparison result, it is determined and confirmed whether the static constraint subgraph is preferentially executed or the dynamic constraint subgraph is preferentially executed. The application introduces a dynamic inversion mechanism based on cost, solves the query plan mismatching problem caused by the dependence of static priority rules under the condition of extremely skewed data distribution, and enables the system to self-adaptively select an optimal execution path.

A database management system (DBMS) traverses an operator topology of a query plan to identify a pipeline within the query plan. The query plan may include at least a first shared sub-plan reference, where the first shared sub-plan reference refers to a first shared sub-plan, and where the first shared sub-plan is associated with a shared node in the operator topology. In addition, the DBMS assigns a pipeline index to pipelines in the query plan based on the execution order. In addition, the DBMS tracks pipeline dependencies of the query plan. For each shared sub-plan reference, the database management system specifies the shared sub-plan reference as a pipeline interrupter if the pipeline of the shared sub-plan reference has a dependency on a pipeline having an index greater than the corresponding shared sub-plan. Otherwise, the shared sub-plan reference is designated as a non-pipeline interrupter. The query plan is then updated based on the shared sub-plan reference specifications.

The application discloses a kind of plan task execution method and device based on blockchain.Therein, the method includes: when the first node in target blockchain needs to produce new first target block, query the plan task list in the state database of the first node, wherein the nodes in target blockchain include target node set, the nodes in target node set are set to allow to produce new block;In the case where it is found that there is target plan task meeting target execution condition in the query plan task list, execute target plan task on the first node, wherein target execution condition, target smart contract called, target execution parameter of target smart contract are specified in target plan task;First transaction record generated by executing target plan task is packed into first target block on the first node.The present application solves the technical problem of low security in the process of executing plan task in the related art.

Example solutions for optimizing an initial query execution plan for a federated database includes identifying a group of operations in the initial query execution plan that conform to a recognized pattern including fetching data from a data store. Based on detection of the pattern and of the presence of a condition, an optimized query execution plan is created from the initial query execution plan by replacing the group of operations with a first execution branch and a second execution branch, wherein the optimized query execution plan executes at least one of the first and second execution branches. The first execution branch includes first operations for fetching the data from a first data store and the second execution branch comprise second operations for fetching at least a subset of the data from a second data store.

The present disclosure relates to systems and methods that perform hybrid query processing using a central processing unit and a graphics processing unit. The systems and methods produce an offloaded query plan for processing by a graphics processing unit. The systems and methods generate filtered data for the offloaded query plan and transfer the offloaded query plan and the filtered data to the graphics processing unit.

Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for optimizing a query plan to execute a query by pushing down top-k information in the query plan, which can be used within a data platform (e.g., database) environment. In particular, various example embodiments enable information from a top-k operator of a query plan to be pushed down the query plan and enable pushing the information through one or more outer join operators (e.g., through multiple levels of outer join operators) of the query plan to a select operator, such as an aggregation operator, positioned below the one or more outer join operators.