97 results about "Materialized view" patented technology

The invention discloses a high-performance AIOT real-time data processing method and device based on Flink, Kafka and Dores, and the method comprises the steps: constructing an end-to-end hierarchical architecture, capturing multi-source database change data in real time through Flink CDC, and analyzing semi-structured data in combination with Apache NiFi, and generating a standardized data stream; the method comprises the following steps: realizing message routing based on KafkaTopic division of a service domain, ensuring the consistency of a data structure by using Schema Registry, and optimizing the transmission efficiency through a dynamic compression algorithm; an Flink real-time computing engine is adopted to implement window aggregation, asynchronous dimension table association and incremental state persistence strategies, and the resource fluctuation and state management problems in streaming computing are solved; time pre-partitioning and intelligent cold and hot layering are achieved through a Dores storage layer, the query performance is improved in combination with materialized views and multi-level indexes, and finally a full-link closed loop of collection-transmission-calculation-storage-management is formed. According to the invention, the problems that the real-time performance and the consistency are difficult to balance and the system complexity is high in the prior art are solved.

The invention provides a materialized view updating method and device, a medium and electronic equipment. In the method, a temporary materialized view storage table can be introduced into a storage engine which only supports additional writing based on LSM-Tree and the like, so that after it is detected that a data table is changed, analysis and influence analysis can be carried out on change operation, and all materialized views stored in the temporary materialized view storage table are stored in the storage engine. And selecting the to-be-updated materialized view influenced by the change. Then, incremental updating can be performed on the to-be-updated materialized view according to the data change content, and an updating result of the to-be-updated materialized view is stored in the temporary materialized view storage table instead of directly modifying original materialized view data stored in the materialized view storage table. In the process, as the temporary materialized view storage table is not constrained by the additional write rule, efficient insertion, update and deletion operations can be supported, and incremental update of the materialized view can be realized on the premise of not damaging the performance of a bottom-layer storage structure.

The invention relates to an intermediate result storage and transparent replacement method and device for an asynchronous materialized view and electronic equipment, and the method comprises the steps: designing a storage structure of the materialized view, and obtaining an asynchronous incremental materialized view with a prepolymerization storage structure; when new data are inserted into the basic table, the asynchronous materialized incremental view is updated by using increment, and preset specific data are merged in the updating process to obtain a merged view; and rewriting query logic, and transparently rewriting the original query into a materialized view in combination with the asynchronous incremental materialized view. According to the method, through the design of a storage structure of the asynchronous incremental materialized view, a rewriting method for keeping query semantics and optimization of incremental data merging in the asynchronous incremental materialized view, the real-time query performance is effectively improved, and meanwhile the expandability and flexibility of the system are kept.

The invention provides a design method and system for efficiently querying a bank data warehouse based on an MPP architecture, and the method comprises the steps: S1, collecting and cleaning data from a plurality of bank business systems, and loading the data to a Greenplum distributed data warehouse; s2, performing hierarchical modeling in the warehouse, constructing a customer center wide table, and storing data in each node in a balanced manner by adopting partition and hash distribution strategies; s3, designing a batch processing task scheduling process, and improving the batch processing efficiency by utilizing parallel execution; s4, optimizing configuration indexes and materialized views according to the queries, and accelerating query response by utilizing parallel computing of the MPP architecture; and S5, monitoring a system load and dynamically and elastically scaling cluster resources to deal with service peak and valley demands. By means of the steps, high-performance batch processing and real-time query support of EB-level bank data are achieved on the GreenplumMPP cluster, the data processing efficiency and the query response speed are remarkably improved, and good expansibility is achieved so as to meet the service development requirement.

The invention discloses a materialized view processing method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: counting a historical slow query log of a service system in a preset time period, and determining a query statement which does not hit a cache according to the historical slow query log; counting query information of each query statement in the preset time period to screen out query statements of which the query information meets a preset requirement to obtain a query statement set; and determining a public query statement of the query statement set, obtaining offline data corresponding to the public query statement from the data lake, and performing materialization processing on the offline data to obtain a materialized view. According to the embodiment, the materialized view can be automatically created, and the problem that the traditional materialized view needs to be created strongly depending on manual analysis is solved, so that access to a data lake is remarkably reduced, and the query performance and user experience of a service system are greatly improved.

The invention discloses a materialized view construction method, a data processing system, a medium and a program product, and relates to the field of electrical digital data processing.The method comprises the steps that text query information is received, the execution sequence and the execution type of query operation in the text query information are determined, and a logic plan is generated; extracting a scanning operation, a projection operation, a connection operation and an aggregation operation from the logic plan to obtain a query sub-plan representing query semantics; according to the operation type of the query sub-plan, converting the query sub-plan into an initial plan slice; normalizing the initial plan film to generate a standard plan film in a preset format; according to a preset aggregation strategy, performing deduplication merging processing on the standard plan slices to obtain a target plan slice; and generating a table establishment statement of the materialized view based on the target plan film to construct the materialized view. By implementing the method, generation of the materialized view can be optimized, and the redundancy of the materialized view is reduced.

The invention discloses a data knitting cross-library view efficient query method and system, and relates to the technical field of data processing. The system comprises a unified query interface module, a metadata management module, an intelligent distributed query optimizer, a multi-level cache module, a materialized view management module, an execution engine module and a performance monitoring and feedback module. According to the efficient query method and system for the data knitting cross-library view, a dynamic cost model is constructed through an intelligent distributed query optimizer, and a push-down strategy is decided and queried in combination with source capability metadata; a multi-level cache and materialized view collaboration mechanism is adopted to reduce source data access; maintenance overhead of materialized views is reduced with incremental refresh driven by CDC.

The invention discloses an index data query method, device and equipment, and relates to the technical field of computers, the method comprises the following steps: receiving a query index request initiated by a user, and obtaining a query dimension associated with a query index; judging whether query dimensions associated with the query indexes are materialized and stored in a materialized view of a fast area or not; if the query dimensions associated with the query indexes are materialized and stored in the materialized view of the fast area, reading data of the query indexes from the materialized view of the fast area; otherwise, reading the data of the query index from the common area in a full-table scanning mode; and integrating the data of the plurality of query indexes read from the fast area and the common area, and generating a unified query result based on preset service logic. According to the method, the existence of all dimensions associated with each query index in the fast zone materialization view is dynamically judged, the optimal data reading path is intelligently selected for different indexes, and the overall query response time is shortened.

The invention discloses a dialogue type data analysis method and system based on a large model application technology in the field of rail transit, and the method comprises the steps: extracting data from a rail transit multi-source system through a streaming or batch processing mode, constructing an original data layer (ODS), and carrying out the automatic processing based on a materialized view technology to generate a detail layer (DWD), a summary layer (DWS) and an application data layer (ADS). And efficient hierarchical management and nearly real-time updating of the data are realized. The system receives a data query demand input by a user in a natural language form, performs intention recognition and semantic understanding by using a large model technology, matches a related data table structure in combination with vector retrieval, automatically generates a safe and accurate SQL query statement, and executes a query operation. And the query result is returned in a JSON format, and the front end dynamically generates a chart through visual tools such as ECharts and the like, so that a dialogue type data analysis result which is analyzed as required is realized.

This invention discloses a low-latency materialized view method, comprising the following steps: S1, after the application server initiates a database transaction to the source database and executes data operation instructions, before the database transaction is committed, a state intent event is constructed. This relates to the field of database technology. This application uses heavy data change information as the state intent event and a lightweight commit confirmation signal as the commit state event. By separating the state intent event and the commit state event and processing them asynchronously in parallel, the update latency of the materialized view no longer depends on the entire CDC link, but only on the transmission latency of the commit confirmation signal. This reduces transmission latency, achieves lower-latency data synchronization, and alleviates the high latency problem inherent in traditional materialized view updates that rely on change data capture technology.

The invention discloses a data query method and device and electronic equipment. The method comprises the steps of obtaining an original query statement; matching the original query statement with a plurality of materialized views to obtain a matching result, the materialized views being materialized views constructed by performing clustering analysis on historical query statements of a database; under the condition that the matching result is successful matching, determining a target query statement corresponding to the original query statement, and executing the target query statement in a target materialized view matched with the original query statement to obtain a first query result; and under the condition that the matching result is matching failure, determining a second query result corresponding to the original query statement from a base table of the database. According to the method and the device, the technical problems of poor generalization ability and storage redundancy of the materialized view caused by judging which queries are suitable for constructing the materialized view based on experience in related technologies are solved.

The invention discloses a decision-making system and method based on a Dores database, and the system comprises a data access layer which is used for accessing multi-source heterogeneous financial data in real time, a Dores storage calculation layer, and achieves the data storage, column-type bucket separation and materialized view pre-calculation based on the Dores database; the visual application layer is used for providing bidirectional interactive financial data visualization and decision support; wherein the data access layer acquires data in real time through a streaming computing engine and transmits the data to the Dores storage computing layer, and the visual application layer performs real-time data interaction with the Dores storage computing layer through a two-way communication protocol. According to the method, the synchronous delay of financial data is shortened from 6 hours to 1 second, second-level real-time integration is realized, the aggregation query response time of billion-level data is reduced to be within 1 second based on a Dores column type storage and materialized view technology, the user interaction response delay is lower than 500 milliseconds through WebSocket two-way communication, and real-time adjustment of parameters and dynamic refreshing of a billboard are supported.

The invention discloses a data processing method and device for e-commerce big data, equipment and a medium, and relates to the technical field of big data. The method comprises the following steps: extracting e-commerce detail data from a target database based on a preset configuration rule, and forwarding the e-commerce detail data to message-oriented middleware for storage; the e-commerce detail data comprises business data, entity data, rule data or configuration data; processing the e-commerce detail data stored in the message-oriented middleware to obtain target processing results, and synchronizing the target processing results to a first database; establishing a materialized view structure comprising at least two layers based on each target processing result stored in the first database; the materialized view structure layers matched with the target processing results are determined respectively, the target processing results are displayed on the materialized view structure layers respectively, the e-commerce big data can be rapidly and accurately processed, complexity is reduced, and the processing results are effectively displayed.

The embodiment of the invention relates to the field of data management, and discloses a semiconductor process formula multi-version parameter compression storage method and related products, and the method comprises the steps: dividing a semiconductor process formula into fixed parameters and variable parameters, and constructing corresponding formula specifications; storing the process formula in a column storage database to create a process formula parameter table, and managing a parameter version by utilizing a versioned merge tree engine; in response to an instruction for modifying the technological formula, inserting a target record into the corresponding technological formula parameter table; based on the materialized view of the target version parameter, in combination with a preset index condition and a set partition, responding to the query instruction; and separating a target field in the parameter so as to carry out fine-grained permission control and access management. According to the scheme, the storage space can be fully utilized, and the data processing efficiency is improved.

The application relates to a log analysis method and device, an electronic device and a storage medium, wherein the log analysis method comprises the following steps: obtaining original log data; processing a preset analysis requirement into an analysis rule for the original log data; based on a preset database syntax of a ClickHouse database, converting the analysis rule into a materialized view, and using the materialized view to perform real-time analysis on the original log data to obtain an analysis result of the original log data. The application uses the ClickHouse database to realize analysis on the original log data, so that real-time analysis on the original log data can be completed without relying on an additional analysis system, and the utilization rate of the performance of the ClickHouse database can be improved, thereby reducing the development cost and maintenance cost of a current log audit system.

A materialized view construction method, data processing system, medium, and program product relate to the field of electronic digital data processing. The method comprises: receiving text query information, determining the execution order and execution type of query operations in the text query information, and generating a logical plan; extracting scan operations, projection operations, join operations, and aggregation operations from the logical plan to obtain a query sub-plan representing the query semantics; converting the query sub-plan into an initial plan slice based on the operation type of the query sub-plan; normalizing the initial plan slice to generate a standard plan slice in a preset format; performing redundancy elimination and merging on the standard plan slice according to a preset aggregation strategy to obtain a target plan slice; and generating a materialized view table creation statement based on the target plan slice to construct the materialized view. Implementation of this application can optimize materialized view generation and reduce redundancy in materialized views.

The invention discloses a configurable large table grouping query optimization method based on a relational database, and relates to the technical field of databases. According to the method, grouping query performance is improved by establishing an aggregation result table, grouping aggregation query logic is asynchronously executed by utilizing a configurable timed task, and meanwhile, an original data scanning range is narrowed by adopting a flag bit mechanism. The method specifically comprises the steps that a scanning timed task is configured to read aggregation query configuration customized by a user regularly; setting a statistical flag bit for the original detail table; when incremental statistics is executed, uncounted records are locked; updating an aggregation result table according to the minimum grouping field; and a data verification mechanism is provided to ensure result accuracy. According to the method, the problem of data delay of a traditional materialized view scheme is solved, database pressure is reduced through a flag bit mechanism, user-defined aggregation logic is supported, and the grouping query efficiency under the large data volume is remarkably improved. The method is particularly suitable for a service scene in which complex grouping aggregation query needs to be frequently executed.

The invention provides a data query method, a data query device, electronic equipment and a storage medium. The data query method comprises the steps of obtaining a query statement; determining whether an original data table to be queried in the query statement has a materialized view or not; and if the materialized view has the query statement, determining a query partition corresponding to the query statement, and obtaining a query result corresponding to the query statement in the query partition of the materialized view. By adopting the technical scheme provided by the invention, the query result corresponding to the query statement can be quickly obtained without a third-party tool, and the data query efficiency is improved under the condition of ensuring the cost.

A system (100) and method (200) for managing queries (134) include: receiving a new query (210) including a first plurality of joins; accessing a filtered view of a database from memory, the filtered view being filtered by a previously received query according to a filter represented by a second plurality of joins (220), at least one of the first plurality of joins or the second plurality of joins including at least one NULL value; determining that the filter of the new query implies a filter of the previously received query (230); and executing the new query (240) using the filtered view of the previously received query based on the determination that the filter of the new query implies a filter of the previously received query.

The multi-tenant system includes one or more hardware processors that obtain a query that includes statements defining one or more operations to be performed on a subset of constituent records stored in a source database. The hardware processors parse the query to decipher a type of the one or more operations, the subset of the constituent records referred to, and one or more other conditions of the operations. The hardware processors obtain the subset of the constituent records from the parsed query from the source database. The hardware processors execute the one or more operations, to generate a materialized view that denormalizes the data. The hardware processors detect any changes in the subset of the constituent records, synchronize the materialized view with the any changes, store the synchronized materialized view within a data platform, and transmit the synchronized materialized view to a target database.

The invention relates to a lightweight network flow storage and management method and system based on flow clustering, and belongs to the field of computer network security, and the method comprises the steps: deploying a Zeek monitoring component and P4 equipment at a network boundary and a core node, and carrying out the structural analysis of a multi-source weblog; based on data source and flow feature clustering and grading processing, dividing into a key flow, a common flow and a secondary flow; hierarchical forwarding and priority scheduling of the logs are realized through Filebeat and Kafka; a hierarchical storage architecture of a core flow layer, a main service flow layer and an archiving flow layer is constructed based on ClickHouse, and efficient retrieval and traceability analysis are realized through a materialized view and a multi-dimensional index. According to the invention, flexible scheduling and grading processing of traffic can be realized in a high-concurrency network environment, the overall throughput efficiency and security situation awareness capability of the system are improved, real-time performance and storage cost control are both considered, and the method is suitable for threat monitoring and event tracing scenes in a large-scale network environment.

The present invention discloses a materialized view design method for database query optimization, comprising the following steps: for a PostgreSQL database, collecting historical query loads and corresponding patterns; constructing a Cosette pattern of the query load; using Cosette to perform equivalent identification, merging, and frequency statistics on subqueries; recommending the most frequently appearing candidate subqueries for actual materialization based on frequency and a greedy strategy; creating a materialized view in PostgreSQL, scoring and replacing the materialized view using a Newton's cooling law calculation formula; and rewriting the user's original query statement in the created materialized view environment for query optimization. The present invention conducts an in-depth analysis and research on view selection and view elimination strategies in materialized view design under a big data environment, proposes a view selection strategy based on a Cosette query statement equivalence prover, and proposes a materialized view scoring and elimination strategy based on Newton's cooling law. The efficiency and effectiveness of view design solutions under a big data environment are improved.

This application relates to the field of computer technology and discloses a method and apparatus for modeling communication topology information based on collaborative traffic monitoring. The method includes: caching collected network traffic data in memory and preprocessing the network traffic data to obtain network traffic attributes corresponding to the network traffic data; merging network traffic data with the same network traffic attributes collected within a preset time period in real time to obtain merged network traffic data, and parsing the merged network traffic data to obtain initial network traffic data; modeling the initial network traffic data to generate data objects that meet database requirements, and storing the data objects in the database to generate a communication link table; and performing database materialized view processing on the communication link table to generate a database physical view. Applying this method can effectively improve retrieval efficiency, meet the requirements of high-efficiency queries, and satisfy multi-dimensional graph query needs.

The invention discloses a database index and materialized view self-adaptive management system and method based on continuous SQL (Structured Query Language) analysis, and belongs to the technical field of database management systems. According to the method, a full-automatic life cycle management framework of'monitoring-analysis-decision-execution-feedback 'is constructed, a query log is analyzed in real time and key features are extracted by utilizing an SQL auditing analysis engine, an operation execution engine automatically executes corresponding DDL operation, and a model is iteratively optimized through a closed-loop feedback module. According to the method, the high-frequency query delay can be reduced by 60%-85%, the write operation throughput is improved by 30% or above, the storage space is saved by 25%-40%, meanwhile, DBA manual intervention is reduced by 90%, and dynamic self-adaptive management of indexes and materialized views is effectively achieved.

The invention discloses a method and device for dynamically reducing retrieval data and computer equipment, and the method comprises the steps: receiving a screening condition sent by a front end, and extracting a column name array and a condition array in the screening condition; performing hash operation on the column name array according to an MD5 encryption algorithm, and generating a table name of the dynamic materialized view; the table name, the condition array and the column name array are converted into SQL condition fragments according to a preset rule, the SQL condition fragments are spliced into a pre-written SQL template, and therefore a complete SQL table building statement is generated; generating a materialized view according to the SQL table establishment statement; and returning data meeting conditions to the front end according to the materialized view. According to the method, the columns needing to be queried can be determined through the screening conditions, the columns are extracted to automatically generate the materialized view, scanning of a large amount of data and pre-defining of the materialized view are reduced, the flexibility when the materialized view is used for querying data is improved, and storage resources and computing resources are utilized to the maximum extent.

The application provides a real-time materialized view system, the system comprising: an entry module receiving a batch query request and associating a Flink SQL engine based on the batch query request; a Flink execution module for: parsing the batch query request to obtain an SQL command; and in the case where the SQL command meets a preset rule, determining to perform a materialized view for the SQL command; and a materialized view management module performing stream calculation on the SQL command and creating a materialized view based on the result of the stream calculation. The application provides a real-time materialized view method, device, computer equipment and computer readable storage medium. The technical scheme of the application can bring great performance improvement and resource saving in the scheme of stream materialized batch, and solves the problem of low batch calculation efficiency.

Disclosed are a database query method, device, and storage medium based on a large language model. The method includes: obtaining a query set and a materialized view set; training a query load trend prediction model based on the query set and a constructed event knowledge base; training a Bayesian network based on the materialized view set; receiving a target query, retrieving related events, and inputting the query load trend prediction model to obtain a first materialized view statement, which is converted into a first variable set and then input into a Bayesian network and configured with evidence to obtain a second variable set; executing a second materialized view statement converted from the second variable set; and rewriting subsequent queries that match the second materialized view statement into queries that read data from a materialized view table and executing them. The present disclosure introduces event factors to predict query load changes, breaking through the limitations of historical queries, using a large language model and a Bayesian network to generate high-quality materialized views, and improving database query efficiency.

The application discloses a redefinition method, device and equipment and a storage medium. The method comprises the following steps: creating a materialized view log according to an original table; creating an intermediate table according to target attribute information corresponding to the original table; creating a first materialized view according to the original table and the intermediate table, wherein the intermediate table is a materialized view auxiliary table of the first materialized view; refreshing the first materialized view, and exchanging the name of the refreshed intermediate table and the name of the original table. Through the technical scheme, the time during which external services cannot be provided during definition modification is reduced, and special modification operations such as modifying a normal table into a partition table can be supported.

The embodiment of the invention provides a data warehouse model construction method and device, equipment, a medium and a product. The method comprises the following steps: acquiring multi-modal data, converting the multi-modal data into an original data set in a target format, storing the original data set in an original data layer, and constructing a data warehouse model layer by layer from the original data layer by adopting a materialized view strategy, the data warehouse model comprises an original data layer, an application data layer, a plurality of detail data layers and a summary data layer, and each data layer is composed of a materialized view storing a pre-calculation result, so that a complex operation chain does not need to be compiled and operated by an external scheduling system; the method is used for achieving the effects of reducing the construction complexity of the data warehouse, reducing the development and operation and maintenance cost and improving the development efficiency of the data warehouse.