130 results about "Materialized view" patented technology

The invention discloses a high-performance AIOT real-time data processing method and device based on Flink, Kafka and Dores, and the method comprises the steps: constructing an end-to-end hierarchical architecture, capturing multi-source database change data in real time through Flink CDC, and analyzing semi-structured data in combination with Apache NiFi, and generating a standardized data stream; the method comprises the following steps: realizing message routing based on KafkaTopic division of a service domain, ensuring the consistency of a data structure by using Schema Registry, and optimizing the transmission efficiency through a dynamic compression algorithm; an Flink real-time computing engine is adopted to implement window aggregation, asynchronous dimension table association and incremental state persistence strategies, and the resource fluctuation and state management problems in streaming computing are solved; time pre-partitioning and intelligent cold and hot layering are achieved through a Dores storage layer, the query performance is improved in combination with materialized views and multi-level indexes, and finally a full-link closed loop of collection-transmission-calculation-storage-management is formed. According to the invention, the problems that the real-time performance and the consistency are difficult to balance and the system complexity is high in the prior art are solved.

A materialized view created from one or more shared producer tables in separate producer clusters may be stored in a consumer cluster. The materialized view May undergo faster a refresh using an incremental refresh by obtaining snapshots of the producer table and incrementally refreshing based on deltas obtained by the snapshot. The materialized view may undergo various checks to determine whether the materialized view in the consumer resource is refreshable or unrefreshable based on schema compatibility. The materialized view may undergo further checks to determine whether the materialized view in the consumer resource is available or unavailable for incremental refresh based on detection of schema / data rearrangement operations on the producer table.

The invention discloses an Oracle removing method based on Clickhouse, which comprises the steps of data model design, real-time and off-line data complete storage and Clickhouse complete monitoring, and is characterized by comprising the following steps of: 1, designing a data model based on a materialized view of the Clickhouse; 2, developing offline and real-time tasks based on Spark and Flink, and respectively writing results into Hive and Topic; step 3, completely synchronizing the off-line and real-time data bins to the Clickhouse; and step 4, on the basis of the Grafana and the Prometheus, the Clickhome is monitored, and the Clickhome is monitored on the basis of the Grafana and the Prometheus. And a set of online query analysis model construction method for replacing Oracle from a data model to data storage, data analysis and Clickhome monitoring is formed.

The present disclosure relates to a database-based data processing method, device, medium, and apparatus, the method including: receiving a target query statement; determining a corresponding target materialized view for executing the target query statement according to a data query time corresponding to the target query statement and a time granularity of the materialized view in a database, wherein the database is constructed with the materialized view at different time granularities; rewriting the target query statement according to the time granularity of the target materialized view to obtain a plurality of target sub-query statements, wherein the target sub-query statements include sub-query statements which are queried based on the target materialized view; and merging the target sub-query statements, and performing a data query according to a query statement obtained by merging.

The invention discloses a materialized view generation method and system, and the method comprises the following steps: carrying out the iterative optimization of query result data based on configured task items until a corresponding materialized view is generated, the task items are divided into a conversion task and a plurality of optimization tasks, and the method specifically comprises the steps: determining a current task item; if the current task item is an optimization task, entering an optimization iteration step, and executing the following steps in the optimization iteration step: acquiring current to-be-optimized data; acquiring corresponding task configuration data; and expanding or merging vertexes in the current to-be-optimized data based on the task configuration data, or expanding or merging edges in the current to-be-optimized data so as to optimize and update the current to-be-optimized data. According to the method, the query result data is iteratively optimized on the basis of the query requirement, so that the generated materialized view is suitable for a graph database scene.

The present disclosure relates to a data processing method, a medium and an electronic device for a database. The method includes: obtaining a first execution statement for performing a target operation; reading, when a data source corresponding to the first execution statement includes first data corresponding to a committed transaction and second data corresponding to an uncommitted transaction, a materialized view obtained after performing the target operation on the first data as a target materialized view from a pre-created materialized view set; reading the second data and performing the target operation on the second data to obtain operation result data; performing data merging processing on the target materialized view and the operation result data to obtain merged data; and performing the target operation on the merged data to obtain an execution result corresponding to the first execution statement.

The invention provides a materialized view updating method and device, a medium and electronic equipment. In the method, a temporary materialized view storage table can be introduced into a storage engine which only supports additional writing based on LSM-Tree and the like, so that after it is detected that a data table is changed, analysis and influence analysis can be carried out on change operation, and all materialized views stored in the temporary materialized view storage table are stored in the storage engine. And selecting the to-be-updated materialized view influenced by the change. Then, incremental updating can be performed on the to-be-updated materialized view according to the data change content, and an updating result of the to-be-updated materialized view is stored in the temporary materialized view storage table instead of directly modifying original materialized view data stored in the materialized view storage table. In the process, as the temporary materialized view storage table is not constrained by the additional write rule, efficient insertion, update and deletion operations can be supported, and incremental update of the materialized view can be realized on the premise of not damaging the performance of a bottom-layer storage structure.

The invention relates to an intermediate result storage and transparent replacement method and device for an asynchronous materialized view and electronic equipment, and the method comprises the steps: designing a storage structure of the materialized view, and obtaining an asynchronous incremental materialized view with a prepolymerization storage structure; when new data are inserted into the basic table, the asynchronous materialized incremental view is updated by using increment, and preset specific data are merged in the updating process to obtain a merged view; and rewriting query logic, and transparently rewriting the original query into a materialized view in combination with the asynchronous incremental materialized view. According to the method, through the design of a storage structure of the asynchronous incremental materialized view, a rewriting method for keeping query semantics and optimization of incremental data merging in the asynchronous incremental materialized view, the real-time query performance is effectively improved, and meanwhile the expandability and flexibility of the system are kept.

The invention provides a design method and system for efficiently querying a bank data warehouse based on an MPP architecture, and the method comprises the steps: S1, collecting and cleaning data from a plurality of bank business systems, and loading the data to a Greenplum distributed data warehouse; s2, performing hierarchical modeling in the warehouse, constructing a customer center wide table, and storing data in each node in a balanced manner by adopting partition and hash distribution strategies; s3, designing a batch processing task scheduling process, and improving the batch processing efficiency by utilizing parallel execution; s4, optimizing configuration indexes and materialized views according to the queries, and accelerating query response by utilizing parallel computing of the MPP architecture; and S5, monitoring a system load and dynamically and elastically scaling cluster resources to deal with service peak and valley demands. By means of the steps, high-performance batch processing and real-time query support of EB-level bank data are achieved on the GreenplumMPP cluster, the data processing efficiency and the query response speed are remarkably improved, and good expansibility is achieved so as to meet the service development requirement.

Systems, methods, and devices for sharing materialized views in multiple tenant database systems. A method includes defining a materialized view over a source table that is associated with a first account of a multiple tenant database. The method includes defining cross-account access rights to the materialized view to a second account such that that second account can read the materialized view without copying the materialized view. The method includes modifying the source table for the materialized view. The method includes identifying whether the materialized view is stale with respect to the source table by merging the materialized view and the source table.

The invention discloses a materialized view processing method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: counting a historical slow query log of a service system in a preset time period, and determining a query statement which does not hit a cache according to the historical slow query log; counting query information of each query statement in the preset time period to screen out query statements of which the query information meets a preset requirement to obtain a query statement set; and determining a public query statement of the query statement set, obtaining offline data corresponding to the public query statement from the data lake, and performing materialization processing on the offline data to obtain a materialized view. According to the embodiment, the materialized view can be automatically created, and the problem that the traditional materialized view needs to be created strongly depending on manual analysis is solved, so that access to a data lake is remarkably reduced, and the query performance and user experience of a service system are greatly improved.

The invention discloses a materialized view construction method, a data processing system, a medium and a program product, and relates to the field of electrical digital data processing.The method comprises the steps that text query information is received, the execution sequence and the execution type of query operation in the text query information are determined, and a logic plan is generated; extracting a scanning operation, a projection operation, a connection operation and an aggregation operation from the logic plan to obtain a query sub-plan representing query semantics; according to the operation type of the query sub-plan, converting the query sub-plan into an initial plan slice; normalizing the initial plan film to generate a standard plan film in a preset format; according to a preset aggregation strategy, performing deduplication merging processing on the standard plan slices to obtain a target plan slice; and generating a table establishment statement of the materialized view based on the target plan film to construct the materialized view. By implementing the method, generation of the materialized view can be optimized, and the redundancy of the materialized view is reduced.

The invention discloses a database query method and device based on a large language model and a storage medium. The method comprises the steps of obtaining a query set and a materialized view set; training a query load trend prediction model based on the query set and the constructed event knowledge base; training a Bayesian network based on the materialized view set; receiving a target query, retrieving a related event, inputting a query load trend prediction model to obtain a first materialized view statement, converting the first materialized view statement into a first variable set, inputting the first variable set into a Bayesian network, and configuring evidence to obtain a second variable set; executing a second materialized view statement converted from the second set of variables; and rewriting a subsequent query matched with the second materialized view statement into a query for reading data from the materialized view table and executing the query. According to the method, event factors are introduced to predict the query load change, the limitation of historical query is broken through, the high-quality materialized view is generated by using the large language model and the Bayesian network, and the database query efficiency is improved.

The invention discloses a data knitting cross-library view efficient query method and system, and relates to the technical field of data processing. The system comprises a unified query interface module, a metadata management module, an intelligent distributed query optimizer, a multi-level cache module, a materialized view management module, an execution engine module and a performance monitoring and feedback module. According to the efficient query method and system for the data knitting cross-library view, a dynamic cost model is constructed through an intelligent distributed query optimizer, and a push-down strategy is decided and queried in combination with source capability metadata; a multi-level cache and materialized view collaboration mechanism is adopted to reduce source data access; maintenance overhead of materialized views is reduced with incremental refresh driven by CDC.

The invention discloses an index data query method, device and equipment, and relates to the technical field of computers, the method comprises the following steps: receiving a query index request initiated by a user, and obtaining a query dimension associated with a query index; judging whether query dimensions associated with the query indexes are materialized and stored in a materialized view of a fast area or not; if the query dimensions associated with the query indexes are materialized and stored in the materialized view of the fast area, reading data of the query indexes from the materialized view of the fast area; otherwise, reading the data of the query index from the common area in a full-table scanning mode; and integrating the data of the plurality of query indexes read from the fast area and the common area, and generating a unified query result based on preset service logic. According to the method, the existence of all dimensions associated with each query index in the fast zone materialization view is dynamically judged, the optimal data reading path is intelligently selected for different indexes, and the overall query response time is shortened.

The invention discloses a generator set data warehouse construction method and system based on an operation state, and relates to the field of data warehouses, and the method comprises the steps: collecting the operation parameters of a generator set in real time through a sensor and a monitoring device; preprocessing operation including cleaning, de-noising and standardizing is carried out on the collected operation parameters so as to improve the data quality; effective characteristic quantities are extracted from the preprocessed data to be used for describing the operation state of the generator set; converting the extracted characteristic quantity into a format conforming to a data warehouse mode; and loading the converted data into a data warehouse, and establishing a target index and a materialized view to optimize query performance. By means of the scheme, high-performance storage and management of the power generation data can be achieved, and the flexible data retrieval requirement is met.

The invention provides a real-time data warehouse construction method and system and a real-time data warehouse query method and system based on an asynchronous materialized view. The real-time data warehouse construction method comprises the steps that a database log is obtained, a data importing technology is used, the database log is imported into a preset message queue and then imported into a Dores database, the database log is stored in an original data layer, and an original data table is generated; creating a first materialized view by using a structured query language, storing a calculation result obtained by querying the original data table in the first target structured query language to the first materialized view to obtain a primary materialized view as a detail data layer, and generating a detail data table; and creating a second materialized view by using the structured query language, storing a calculation result obtained by querying the detail data table in the second target structured query language to the second materialized view to obtain a second-level materialized view as a data service layer, and generating a data service table. According to the real-time data warehouse construction method based on the asynchronous materialized view provided by the invention, the development cost and the maintenance cost are reduced.

The invention discloses a dialogue type data analysis method and system based on a large model application technology in the field of rail transit, and the method comprises the steps: extracting data from a rail transit multi-source system through a streaming or batch processing mode, constructing an original data layer (ODS), and carrying out the automatic processing based on a materialized view technology to generate a detail layer (DWD), a summary layer (DWS) and an application data layer (ADS). And efficient hierarchical management and nearly real-time updating of the data are realized. The system receives a data query demand input by a user in a natural language form, performs intention recognition and semantic understanding by using a large model technology, matches a related data table structure in combination with vector retrieval, automatically generates a safe and accurate SQL query statement, and executes a query operation. And the query result is returned in a JSON format, and the front end dynamically generates a chart through visual tools such as ECharts and the like, so that a dialogue type data analysis result which is analyzed as required is realized.

A method and one or more non-transitory storage media for materialized view-based query rewrite are provided. A plurality of materialized views is created based on a fact table and a dimension table. Each materialized view is based on a join between the dimension table and the fact table based on a respective foreign key column of the fact table. A database management system executes a query against the fact table and the dimension table, the query requiring one or more joins between the dimension table and the fact table based on one or more foreign key columns. For each given join of the one or more joins, responsive to the given join satisfying one or more rewrite criteria, the query is rewritten to replace the join between the dimension table and the fact table with a join between a respective materialized view for the given join and the fact table.

The invention provides a report query method and device based on a materialized view, and the method comprises the steps: obtaining a to-be-queried report which comprises multiple pieces of dimension information; obtaining a pre-query request, extracting a query dimension and aggregation logic in the pre-query request, and determining dimension information corresponding to the query dimension from the to-be-queried report based on the extracted query dimension, determining to-be-aggregated data information from the to-be-queried report based on the extracted aggregation logic and the determined dimension information corresponding to the query dimension, and obtaining a pre-calculation result through the aggregation logic based on the to-be-aggregated data information; constructing a materialized view based on the pre-calculation result, wherein the materialized view comprises an aggregation result corresponding to the aggregation logic; and obtaining a to-be-queried request, extracting an aggregation request in the to-be-queried request, and extracting an aggregation result in the materialized view based on the aggregation request as a query result corresponding to the to-be-queried request. According to the method, the data query efficiency is improved, and the consumption of system resources and the data query cost are reduced.

Embodiments of the present specification provide a creation method and apparatus for a materialized view of a database. In the creation method, relationship trees corresponding to respective SQL statements in a group of SQL statements for a target database are acquired, nodes in each relationship tree representing operators needing to be executed to complete the corresponding SQL statement. An initial relationship forest is constructed on the basis of the relationship trees corresponding to the respective SQL statements. The initial relationship forest is traversed, and a plurality of node pairs are formed on the basis of a current node and brother nodes thereof and child nodes of the current node and the brother nodes thereof. The structure of the initial relationship forest is modified on the basis of description information of operators represented by the two nodes in each node pair, to obtain a target relationship forest. A target relationship path is determined from the target relationship forest, and a materialized view of the target database is created on the basis of a target SQL statement corresponding to the target relationship path, wherein private data can be stored in the target database.

This invention discloses a low-latency materialized view method, comprising the following steps: S1, after the application server initiates a database transaction to the source database and executes data operation instructions, before the database transaction is committed, a state intent event is constructed. This relates to the field of database technology. This application uses heavy data change information as the state intent event and a lightweight commit confirmation signal as the commit state event. By separating the state intent event and the commit state event and processing them asynchronously in parallel, the update latency of the materialized view no longer depends on the entire CDC link, but only on the transmission latency of the commit confirmation signal. This reduces transmission latency, achieves lower-latency data synchronization, and alleviates the high latency problem inherent in traditional materialized view updates that rely on change data capture technology.

The invention discloses a data query method and device and electronic equipment. The method comprises the steps of obtaining an original query statement; matching the original query statement with a plurality of materialized views to obtain a matching result, the materialized views being materialized views constructed by performing clustering analysis on historical query statements of a database; under the condition that the matching result is successful matching, determining a target query statement corresponding to the original query statement, and executing the target query statement in a target materialized view matched with the original query statement to obtain a first query result; and under the condition that the matching result is matching failure, determining a second query result corresponding to the original query statement from a base table of the database. According to the method and the device, the technical problems of poor generalization ability and storage redundancy of the materialized view caused by judging which queries are suitable for constructing the materialized view based on experience in related technologies are solved.

The invention discloses a decision-making system and method based on a Dores database, and the system comprises a data access layer which is used for accessing multi-source heterogeneous financial data in real time, a Dores storage calculation layer, and achieves the data storage, column-type bucket separation and materialized view pre-calculation based on the Dores database; the visual application layer is used for providing bidirectional interactive financial data visualization and decision support; wherein the data access layer acquires data in real time through a streaming computing engine and transmits the data to the Dores storage computing layer, and the visual application layer performs real-time data interaction with the Dores storage computing layer through a two-way communication protocol. According to the method, the synchronous delay of financial data is shortened from 6 hours to 1 second, second-level real-time integration is realized, the aggregation query response time of billion-level data is reduced to be within 1 second based on a Dores column type storage and materialized view technology, the user interaction response delay is lower than 500 milliseconds through WebSocket two-way communication, and real-time adjustment of parameters and dynamic refreshing of a billboard are supported.

The invention discloses a data processing method and device for e-commerce big data, equipment and a medium, and relates to the technical field of big data. The method comprises the following steps: extracting e-commerce detail data from a target database based on a preset configuration rule, and forwarding the e-commerce detail data to message-oriented middleware for storage; the e-commerce detail data comprises business data, entity data, rule data or configuration data; processing the e-commerce detail data stored in the message-oriented middleware to obtain target processing results, and synchronizing the target processing results to a first database; establishing a materialized view structure comprising at least two layers based on each target processing result stored in the first database; the materialized view structure layers matched with the target processing results are determined respectively, the target processing results are displayed on the materialized view structure layers respectively, the e-commerce big data can be rapidly and accurately processed, complexity is reduced, and the processing results are effectively displayed.

The embodiment of the invention relates to the field of data management, and discloses a semiconductor process formula multi-version parameter compression storage method and related products, and the method comprises the steps: dividing a semiconductor process formula into fixed parameters and variable parameters, and constructing corresponding formula specifications; storing the process formula in a column storage database to create a process formula parameter table, and managing a parameter version by utilizing a versioned merge tree engine; in response to an instruction for modifying the technological formula, inserting a target record into the corresponding technological formula parameter table; based on the materialized view of the target version parameter, in combination with a preset index condition and a set partition, responding to the query instruction; and separating a target field in the parameter so as to carry out fine-grained permission control and access management. According to the scheme, the storage space can be fully utilized, and the data processing efficiency is improved.

The application relates to a log analysis method and device, an electronic device and a storage medium, wherein the log analysis method comprises the following steps: obtaining original log data; processing a preset analysis requirement into an analysis rule for the original log data; based on a preset database syntax of a ClickHouse database, converting the analysis rule into a materialized view, and using the materialized view to perform real-time analysis on the original log data to obtain an analysis result of the original log data. The application uses the ClickHouse database to realize analysis on the original log data, so that real-time analysis on the original log data can be completed without relying on an additional analysis system, and the utilization rate of the performance of the ClickHouse database can be improved, thereby reducing the development cost and maintenance cost of a current log audit system.

A materialized view construction method, data processing system, medium, and program product relate to the field of electronic digital data processing. The method comprises: receiving text query information, determining the execution order and execution type of query operations in the text query information, and generating a logical plan; extracting scan operations, projection operations, join operations, and aggregation operations from the logical plan to obtain a query sub-plan representing the query semantics; converting the query sub-plan into an initial plan slice based on the operation type of the query sub-plan; normalizing the initial plan slice to generate a standard plan slice in a preset format; performing redundancy elimination and merging on the standard plan slice according to a preset aggregation strategy to obtain a target plan slice; and generating a materialized view table creation statement based on the target plan slice to construct the materialized view. Implementation of this application can optimize materialized view generation and reduce redundancy in materialized views.

The invention provides a massive session data storage and query method which comprises the following steps: creating an index bucket in a database, the index bucket consisting of multiple layers of index spaces, and distributing different time precision labels for each layer of index space; monitoring a target data interface, storing session data obtained from the target data interface into a preset minute precision table in a database, and storing index information of the minute precision table into a first layer of index space of an index bucket; materialized views are configured for the index spaces from the first layer to the last but one layer to be used for monitoring the data change condition in the index space of the previous layer, and the materialized view in the index space of the first layer is used for monitoring the data updating condition of the minute precision table; and in response to monitoring that the data of the minute precision table is changed, sequentially updating the index information in the multi-layer index space. According to the method, the index bucket with time as precision is created in the data table generation process, so that the data query efficiency is improved, and the occupancy rate of a CPU (Central Processing Unit) is reduced.

The invention discloses a configurable large table grouping query optimization method based on a relational database, and relates to the technical field of databases. According to the method, grouping query performance is improved by establishing an aggregation result table, grouping aggregation query logic is asynchronously executed by utilizing a configurable timed task, and meanwhile, an original data scanning range is narrowed by adopting a flag bit mechanism. The method specifically comprises the steps that a scanning timed task is configured to read aggregation query configuration customized by a user regularly; setting a statistical flag bit for the original detail table; when incremental statistics is executed, uncounted records are locked; updating an aggregation result table according to the minimum grouping field; and a data verification mechanism is provided to ensure result accuracy. According to the method, the problem of data delay of a traditional materialized view scheme is solved, database pressure is reduced through a flag bit mechanism, user-defined aggregation logic is supported, and the grouping query efficiency under the large data volume is remarkably improved. The method is particularly suitable for a service scene in which complex grouping aggregation query needs to be frequently executed.