72 results about "Materialized view" patented technology

The invention discloses a high-performance AIOT real-time data processing method and device based on Flink, Kafka and Dores, and the method comprises the steps: constructing an end-to-end hierarchical architecture, capturing multi-source database change data in real time through Flink CDC, and analyzing semi-structured data in combination with Apache NiFi, and generating a standardized data stream; the method comprises the following steps: realizing message routing based on KafkaTopic division of a service domain, ensuring the consistency of a data structure by using Schema Registry, and optimizing the transmission efficiency through a dynamic compression algorithm; an Flink real-time computing engine is adopted to implement window aggregation, asynchronous dimension table association and incremental state persistence strategies, and the resource fluctuation and state management problems in streaming computing are solved; time pre-partitioning and intelligent cold and hot layering are achieved through a Dores storage layer, the query performance is improved in combination with materialized views and multi-level indexes, and finally a full-link closed loop of collection-transmission-calculation-storage-management is formed. According to the invention, the problems that the real-time performance and the consistency are difficult to balance and the system complexity is high in the prior art are solved.

The invention provides a materialized view updating method and device, a medium and electronic equipment. In the method, a temporary materialized view storage table can be introduced into a storage engine which only supports additional writing based on LSM-Tree and the like, so that after it is detected that a data table is changed, analysis and influence analysis can be carried out on change operation, and all materialized views stored in the temporary materialized view storage table are stored in the storage engine. And selecting the to-be-updated materialized view influenced by the change. Then, incremental updating can be performed on the to-be-updated materialized view according to the data change content, and an updating result of the to-be-updated materialized view is stored in the temporary materialized view storage table instead of directly modifying original materialized view data stored in the materialized view storage table. In the process, as the temporary materialized view storage table is not constrained by the additional write rule, efficient insertion, update and deletion operations can be supported, and incremental update of the materialized view can be realized on the premise of not damaging the performance of a bottom-layer storage structure.

The invention discloses a data knitting cross-library view efficient query method and system, and relates to the technical field of data processing. The system comprises a unified query interface module, a metadata management module, an intelligent distributed query optimizer, a multi-level cache module, a materialized view management module, an execution engine module and a performance monitoring and feedback module. According to the efficient query method and system for the data knitting cross-library view, a dynamic cost model is constructed through an intelligent distributed query optimizer, and a push-down strategy is decided and queried in combination with source capability metadata; a multi-level cache and materialized view collaboration mechanism is adopted to reduce source data access; maintenance overhead of materialized views is reduced with incremental refresh driven by CDC.

The invention discloses an index data query method, device and equipment, and relates to the technical field of computers, the method comprises the following steps: receiving a query index request initiated by a user, and obtaining a query dimension associated with a query index; judging whether query dimensions associated with the query indexes are materialized and stored in a materialized view of a fast area or not; if the query dimensions associated with the query indexes are materialized and stored in the materialized view of the fast area, reading data of the query indexes from the materialized view of the fast area; otherwise, reading the data of the query index from the common area in a full-table scanning mode; and integrating the data of the plurality of query indexes read from the fast area and the common area, and generating a unified query result based on preset service logic. According to the method, the existence of all dimensions associated with each query index in the fast zone materialization view is dynamically judged, the optimal data reading path is intelligently selected for different indexes, and the overall query response time is shortened.

The invention discloses a dialogue type data analysis method and system based on a large model application technology in the field of rail transit, and the method comprises the steps: extracting data from a rail transit multi-source system through a streaming or batch processing mode, constructing an original data layer (ODS), and carrying out the automatic processing based on a materialized view technology to generate a detail layer (DWD), a summary layer (DWS) and an application data layer (ADS). And efficient hierarchical management and nearly real-time updating of the data are realized. The system receives a data query demand input by a user in a natural language form, performs intention recognition and semantic understanding by using a large model technology, matches a related data table structure in combination with vector retrieval, automatically generates a safe and accurate SQL query statement, and executes a query operation. And the query result is returned in a JSON format, and the front end dynamically generates a chart through visual tools such as ECharts and the like, so that a dialogue type data analysis result which is analyzed as required is realized.

This invention discloses a low-latency materialized view method, comprising the following steps: S1, after the application server initiates a database transaction to the source database and executes data operation instructions, before the database transaction is committed, a state intent event is constructed. This relates to the field of database technology. This application uses heavy data change information as the state intent event and a lightweight commit confirmation signal as the commit state event. By separating the state intent event and the commit state event and processing them asynchronously in parallel, the update latency of the materialized view no longer depends on the entire CDC link, but only on the transmission latency of the commit confirmation signal. This reduces transmission latency, achieves lower-latency data synchronization, and alleviates the high latency problem inherent in traditional materialized view updates that rely on change data capture technology.

The invention discloses a data query method and device and electronic equipment. The method comprises the steps of obtaining an original query statement; matching the original query statement with a plurality of materialized views to obtain a matching result, the materialized views being materialized views constructed by performing clustering analysis on historical query statements of a database; under the condition that the matching result is successful matching, determining a target query statement corresponding to the original query statement, and executing the target query statement in a target materialized view matched with the original query statement to obtain a first query result; and under the condition that the matching result is matching failure, determining a second query result corresponding to the original query statement from a base table of the database. According to the method and the device, the technical problems of poor generalization ability and storage redundancy of the materialized view caused by judging which queries are suitable for constructing the materialized view based on experience in related technologies are solved.

The invention discloses a decision-making system and method based on a Dores database, and the system comprises a data access layer which is used for accessing multi-source heterogeneous financial data in real time, a Dores storage calculation layer, and achieves the data storage, column-type bucket separation and materialized view pre-calculation based on the Dores database; the visual application layer is used for providing bidirectional interactive financial data visualization and decision support; wherein the data access layer acquires data in real time through a streaming computing engine and transmits the data to the Dores storage computing layer, and the visual application layer performs real-time data interaction with the Dores storage computing layer through a two-way communication protocol. According to the method, the synchronous delay of financial data is shortened from 6 hours to 1 second, second-level real-time integration is realized, the aggregation query response time of billion-level data is reduced to be within 1 second based on a Dores column type storage and materialized view technology, the user interaction response delay is lower than 500 milliseconds through WebSocket two-way communication, and real-time adjustment of parameters and dynamic refreshing of a billboard are supported.

The application relates to a log analysis method and device, an electronic device and a storage medium, wherein the log analysis method comprises the following steps: obtaining original log data; processing a preset analysis requirement into an analysis rule for the original log data; based on a preset database syntax of a ClickHouse database, converting the analysis rule into a materialized view, and using the materialized view to perform real-time analysis on the original log data to obtain an analysis result of the original log data. The application uses the ClickHouse database to realize analysis on the original log data, so that real-time analysis on the original log data can be completed without relying on an additional analysis system, and the utilization rate of the performance of the ClickHouse database can be improved, thereby reducing the development cost and maintenance cost of a current log audit system.

A materialized view construction method, data processing system, medium, and program product relate to the field of electronic digital data processing. The method comprises: receiving text query information, determining the execution order and execution type of query operations in the text query information, and generating a logical plan; extracting scan operations, projection operations, join operations, and aggregation operations from the logical plan to obtain a query sub-plan representing the query semantics; converting the query sub-plan into an initial plan slice based on the operation type of the query sub-plan; normalizing the initial plan slice to generate a standard plan slice in a preset format; performing redundancy elimination and merging on the standard plan slice according to a preset aggregation strategy to obtain a target plan slice; and generating a materialized view table creation statement based on the target plan slice to construct the materialized view. Implementation of this application can optimize materialized view generation and reduce redundancy in materialized views.

The invention discloses a configurable large table grouping query optimization method based on a relational database, and relates to the technical field of databases. According to the method, grouping query performance is improved by establishing an aggregation result table, grouping aggregation query logic is asynchronously executed by utilizing a configurable timed task, and meanwhile, an original data scanning range is narrowed by adopting a flag bit mechanism. The method specifically comprises the steps that a scanning timed task is configured to read aggregation query configuration customized by a user regularly; setting a statistical flag bit for the original detail table; when incremental statistics is executed, uncounted records are locked; updating an aggregation result table according to the minimum grouping field; and a data verification mechanism is provided to ensure result accuracy. According to the method, the problem of data delay of a traditional materialized view scheme is solved, database pressure is reduced through a flag bit mechanism, user-defined aggregation logic is supported, and the grouping query efficiency under the large data volume is remarkably improved. The method is particularly suitable for a service scene in which complex grouping aggregation query needs to be frequently executed.

A system (100) and method (200) for managing queries (134) include: receiving a new query (210) including a first plurality of joins; accessing a filtered view of a database from memory, the filtered view being filtered by a previously received query according to a filter represented by a second plurality of joins (220), at least one of the first plurality of joins or the second plurality of joins including at least one NULL value; determining that the filter of the new query implies a filter of the previously received query (230); and executing the new query (240) using the filtered view of the previously received query based on the determination that the filter of the new query implies a filter of the previously received query.

The multi-tenant system includes one or more hardware processors that obtain a query that includes statements defining one or more operations to be performed on a subset of constituent records stored in a source database. The hardware processors parse the query to decipher a type of the one or more operations, the subset of the constituent records referred to, and one or more other conditions of the operations. The hardware processors obtain the subset of the constituent records from the parsed query from the source database. The hardware processors execute the one or more operations, to generate a materialized view that denormalizes the data. The hardware processors detect any changes in the subset of the constituent records, synchronize the materialized view with the any changes, store the synchronized materialized view within a data platform, and transmit the synchronized materialized view to a target database.

The invention relates to a lightweight network flow storage and management method and system based on flow clustering, and belongs to the field of computer network security, and the method comprises the steps: deploying a Zeek monitoring component and P4 equipment at a network boundary and a core node, and carrying out the structural analysis of a multi-source weblog; based on data source and flow feature clustering and grading processing, dividing into a key flow, a common flow and a secondary flow; hierarchical forwarding and priority scheduling of the logs are realized through Filebeat and Kafka; a hierarchical storage architecture of a core flow layer, a main service flow layer and an archiving flow layer is constructed based on ClickHouse, and efficient retrieval and traceability analysis are realized through a materialized view and a multi-dimensional index. According to the invention, flexible scheduling and grading processing of traffic can be realized in a high-concurrency network environment, the overall throughput efficiency and security situation awareness capability of the system are improved, real-time performance and storage cost control are both considered, and the method is suitable for threat monitoring and event tracing scenes in a large-scale network environment.

This application relates to the field of computer technology and discloses a method and apparatus for modeling communication topology information based on collaborative traffic monitoring. The method includes: caching collected network traffic data in memory and preprocessing the network traffic data to obtain network traffic attributes corresponding to the network traffic data; merging network traffic data with the same network traffic attributes collected within a preset time period in real time to obtain merged network traffic data, and parsing the merged network traffic data to obtain initial network traffic data; modeling the initial network traffic data to generate data objects that meet database requirements, and storing the data objects in the database to generate a communication link table; and performing database materialized view processing on the communication link table to generate a database physical view. Applying this method can effectively improve retrieval efficiency, meet the requirements of high-efficiency queries, and satisfy multi-dimensional graph query needs.

The invention discloses a database index and materialized view self-adaptive management system and method based on continuous SQL (Structured Query Language) analysis, and belongs to the technical field of database management systems. According to the method, a full-automatic life cycle management framework of'monitoring-analysis-decision-execution-feedback 'is constructed, a query log is analyzed in real time and key features are extracted by utilizing an SQL auditing analysis engine, an operation execution engine automatically executes corresponding DDL operation, and a model is iteratively optimized through a closed-loop feedback module. According to the method, the high-frequency query delay can be reduced by 60%-85%, the write operation throughput is improved by 30% or above, the storage space is saved by 25%-40%, meanwhile, DBA manual intervention is reduced by 90%, and dynamic self-adaptive management of indexes and materialized views is effectively achieved.

The application provides a real-time materialized view system, the system comprising: an entry module receiving a batch query request and associating a Flink SQL engine based on the batch query request; a Flink execution module for: parsing the batch query request to obtain an SQL command; and in the case where the SQL command meets a preset rule, determining to perform a materialized view for the SQL command; and a materialized view management module performing stream calculation on the SQL command and creating a materialized view based on the result of the stream calculation. The application provides a real-time materialized view method, device, computer equipment and computer readable storage medium. The technical scheme of the application can bring great performance improvement and resource saving in the scheme of stream materialized batch, and solves the problem of low batch calculation efficiency.

The application discloses a redefinition method, device and equipment and a storage medium. The method comprises the following steps: creating a materialized view log according to an original table; creating an intermediate table according to target attribute information corresponding to the original table; creating a first materialized view according to the original table and the intermediate table, wherein the intermediate table is a materialized view auxiliary table of the first materialized view; refreshing the first materialized view, and exchanging the name of the refreshed intermediate table and the name of the original table. Through the technical scheme, the time during which external services cannot be provided during definition modification is reduced, and special modification operations such as modifying a normal table into a partition table can be supported.

The embodiment of the invention provides a data warehouse model construction method and device, equipment, a medium and a product. The method comprises the following steps: acquiring multi-modal data, converting the multi-modal data into an original data set in a target format, storing the original data set in an original data layer, and constructing a data warehouse model layer by layer from the original data layer by adopting a materialized view strategy, the data warehouse model comprises an original data layer, an application data layer, a plurality of detail data layers and a summary data layer, and each data layer is composed of a materialized view storing a pre-calculation result, so that a complex operation chain does not need to be compiled and operated by an external scheduling system; the method is used for achieving the effects of reducing the construction complexity of the data warehouse, reducing the development and operation and maintenance cost and improving the development efficiency of the data warehouse.

The application relates to a parallel incremental synchronization method, a system, a storage medium and an electronic device. The method comprises the following steps: establishing a materialized view log table for each data table to be synchronized; establishing a plurality of extraction threads, and extracting data in a plurality of materialized view log tables in batches and in parallel through the plurality of extraction threads, storing the data extracted in batches and in parallel in a plurality of data structures, and recording a starting sequence number and an ending sequence number of corresponding batch data in each data structure; caching the plurality of data structures in a plurality of buffer queues; establishing a plurality of warehousing threads, and warehousing the data structures cached in the plurality of buffer queues in parallel through the plurality of warehousing threads, confirming the incremental data to be synchronized according to the starting sequence number and the ending sequence number recorded in each data structure after warehousing, and deleting corresponding records in the materialized view log table. The application can synchronize data in a plurality of materialized view log tables in parallel.

The invention relates to an active-active arbitration cross-domain data synchronization method, system and device and a medium. The method comprises the following steps: acquiring a write-in request, and generating an operation log record of a source site according to the write-in request; fragmenting the operation log record according to the entity identifier through a duplicator, and pushing the fragmented operation log record to a remote site to obtain an operation log record copy of the remote site; performing conflict candidate detection on the compression causal abstract in the operation log record copy and the local concurrent pending operation log record corresponding to the timestamp to obtain a conflict candidate set; determining an arbitration strategy of each conflict pair according to the operation semantic tag in the conflict candidate set, and performing merging operation on each conflict pair according to the arbitration strategy to obtain an arbitration certificate; and writing the final state of the write request indicated by the arbitration certificate into local materialized views of the source site and the remote site, and carrying out persistence processing to obtain a data visible consistent result. By adopting the method, self-arbitration synchronization under the dynamic conflict condition can be realized.

A materialized view construction method, a device, and a medium in a database system are provided. The method includes: receiving a materialized view construction request for a target base table; determining whether the materialized view construction request satisfies a preset construction condition after determining that the target base table belongs to a target type; and constructing a materialized view of the target base table based on a target query statement when determining that the materialized view construction request is executable.

The invention discloses a data processing method and device, a storage medium and a computer program product. The method comprises the following steps: acquiring a data query request; a target materialized view of a target relation corresponding to the data query request is determined, a segment-level fragmentation abstract corresponding to the target materialized view is obtained, the target materialized view comprises at least one data identifier under the target relation, the at least one data identifier is stored in the target materialized view in a segmented mode, and a segment-level fragmentation abstract corresponding to the segment-level fragmentation abstract is obtained; each segment corresponds to one segment-level fragmentation abstract; determining a candidate fragment set corresponding to the data query request according to the segment-level fragment abstract; and querying each fragment in the candidate fragment set to obtain a data content query result corresponding to the data query request. According to the method and the device, the technical problem of high data query delay caused by undifferentiated access of a query strategy to fragments after a bottom-layer main table is horizontally fragmented in related technologies is solved.

This invention relates to the field of database technology and discloses a dynamic data management method oriented towards business complexity. This method parses business logs to construct a real-time business interaction topology graph, calculates the structural entanglement degree representing logical coupling and the access discrete entropy representing the degree of hotspot dispersion, and generates a business state vector. It uses a time series model to predict future states, employs a recursive cost function considering lock blocking diffusion and resource contention effects to calculate execution costs, and generates policy instructions when the net utility meets a lag threshold. Responding to the instructions, it executes dynamic materialized views or transient vertical splitting operations including hot / cold separation, and performs nonlinear adaptive synchronous flow control based on real-time load. This invention quantifies business complexity through graph theory features and combines a predictive decision-making mechanism to achieve proactive data architecture optimization, effectively solving the performance degradation problem in highly coupled and hotspot scenarios.

The application provides a database with a materialized nested loop join optimization method, a storage medium and equipment. The method comprises the following steps: obtaining a nested loop join; judging whether the nested loop join meets an optimization condition; if yes, calling an optimizer to create a first path containing a materialized index node and a second path containing a materialized node, judging whether the cost of executing the first path is less than the cost of executing the second path, and in the case that the cost of executing the first path is less than the cost of executing the second path, calling the optimizer to generate a first execution plan according to the first path; and calling an executor to execute the first execution plan. Through the method, the nested loop join meeting the optimization condition can be screened out, and a materialized index node is created. The node expands the capacity of the original materialized node, so that the node has the capacity of fast indexing, thereby pushing down the join condition to the materialized index node, accelerating the scanning speed of the materialized view, and improving the performance of the database.

The application relates to a blockchain cryptocurrency fund flow searching method and a control system. The method comprises the following steps: acquiring transaction detail data information, preprocessing the transaction detail data information, and generating a materialized view; presetting a search condition of a query address, and acquiring the query address from the materialized view according to the search condition; traversing the query address, obtaining a first result graph, saving a target address in the first result graph, and searching from the target address in sequence through breadth-first search and depth-first search to obtain a target path list. The breadth-first search and the depth-first search are combined, the searching efficiency is effectively improved, the requirement of fast searching a feature address is met, the data searching speed and precision can reach an optimal level, and the calculation pressure of a server is reduced.

The invention provides a data synchronization method and device, equipment and a storage medium, and belongs to the field of data storage, and the method comprises the steps: responding to a data updating log of a data lake table corresponding to a data lake, and obtaining a data change item in the data lake, the data change item at least comprising one of a data addition item, a data deletion item and a data modification item; performing data segmentation on data in the data updating item to obtain a plurality of data packets; performing statement format conversion on each data packet to obtain a plurality of target data packets in a target statement format, the target statement format being materialized view data in a data format supported by the OLAP engine; and sending each target data packet to an OLAP engine, so that the OLAP engine performs materialized view data updating based on each target data packet. The efficiency of updating the materialized view data is improved. The method can be applied to the technical fields of medical treatment and finance, and the efficiency of updating materialized view data in financial business and medical business is improved.

The invention discloses a method and a system for automatically generating a database materialized view based on deep learning. According to the method, semantic representation of sub-queries can be learned from historical query statements, and potential common sub-structures are identified to reduce redundancy; the query acceleration income and the storage cost of the materialized view are quickly estimated by using a multi-task prediction model, and one-by-one real creation and evaluation are avoided; and reinforcement learning is introduced, and the materialized view combination is globally optimized under the constraint of storage budget, so that efficient and automatic view selection and generation are realized in a complex query scene, and the query performance and the resource utilization rate are remarkably improved.

A materialized view management service (MVMS) is capable of monitoring resource allocation changes of a source data object at a source data store and responsively generating resource change alerts to the owner of a target data object (the materialized view) in the target data store. Resource allocation changes may include autoscaling changes to the source data object's partition scheme, throughput limit, storage limit, and the like. The MVMS generates resource change alerts in response to these detected events and pushes the alerts to interested subscribers. Depending on the embodiment, the alerts may be pushed to human administrators, or the target data store itself, which may be configured to automatically adjust the resource allocation of the target data object to match the source data object. Advantageously, the disclosed alerts allow view owners to gain real time visibility of resource auto-scaling at the data source and appropriately react to such changes.