34 results about "Full table scan" patented technology

The data query optimization method, system, database and electronic device based on machine learning provided by the present invention include: obtaining an index automatic recommendation model; wherein the index automatic recommendation model is obtained based on feature training of each historical query in the historical query log; obtaining a query statement, and when the query mode of the query statement is a full table scan, predicting the optimal index corresponding to the query statement through the index automatic recommendation model; creating an optimal index, and optimizing the optimal index through a preset enhanced learning mechanism combined with the query performance after using the optimal index. The present invention dynamically recommends the optimal index for the query statement through the index automatic recommendation model, thereby ensuring the accuracy and pertinence of the index. The present invention can also automatically optimize the index through the enhanced learning mechanism, greatly reducing the complexity and labor cost of index management.

The invention provides a database operation statement processing method and a related product. The method comprises the following steps: acquiring a database operation statement; judging whether the database operation statement is a target counting statement or not; if yes, a counting record corresponding to the target data table is inquired in a preset counting table according to the database operation statement, the total record number of the target data table is calculated according to the counting record, and the preset counting table is used for recording data row addition and deletion operation corresponding to the data table with the rapid counting function started. Through the method, full table scanning of one table can be converted into index scanning of the specified table, the query time is greatly shortened, and the efficiency improvement effect is more remarkable especially for a scene with large-scale data.

The invention discloses an index data query method, device and equipment, and relates to the technical field of computers, the method comprises the following steps: receiving a query index request initiated by a user, and obtaining a query dimension associated with a query index; judging whether query dimensions associated with the query indexes are materialized and stored in a materialized view of a fast area or not; if the query dimensions associated with the query indexes are materialized and stored in the materialized view of the fast area, reading data of the query indexes from the materialized view of the fast area; otherwise, reading the data of the query index from the common area in a full-table scanning mode; and integrating the data of the plurality of query indexes read from the fast area and the common area, and generating a unified query result based on preset service logic. According to the method, the existence of all dimensions associated with each query index in the fast zone materialization view is dynamically judged, the optimal data reading path is intelligently selected for different indexes, and the overall query response time is shortened.

The data migration method comprises the following steps: acquiring storage information from a source database; sorting data blocks corresponding to the plurality of data block numbers based on the plurality of data file numbers and the plurality of data block numbers to obtain a data block queue; dividing the data block queue to obtain a plurality of fragments; based on the data block serial number, the data file serial number and the data object serial number corresponding to one or more data blocks of each fragment, constructing row identifier RowID range information of each fragment; according to the RowID range information of each fragment, reading data of a plurality of fragments from a source database; and writing the data of the plurality of fragments into a target database in parallel. According to the method, in the process that the source database is moved to the target database, full table scanning on the data table in the source database can be avoided, the influence on the source database is greatly reduced, and the table slicing efficiency is improved.

The present application relates to a graph database attribute management method and system. The beneficial effects of the present application are: by setting a dual default value mechanism, separating the logic of reading and writing default values ​​of attributes, flexible modification of default values ​​and compatibility with historical data constraints are achieved. Data backfilling is avoided, and default values ​​are managed through metadata, avoiding full table scans and batch updates; dynamic flexibility is provided, and iterative updates of business logic are supported by modifying and writing default values; it has good compatibility, is transparent to the application layer, and does not require modification of existing query statements. The present invention successfully solves the problem of conflicts between default value modifications and historical data constraints, avoids inconsistencies between index data and original data, prevents the problem of inconsistent data queried with and without indexes, and prevents the problem of metadata modifications affecting original data values.

The invention discloses a query statement processing method and device, equipment, a storage medium and a product. The invention relates to the technical field of databases. The method comprises the steps that a slow query type corresponding to a target slow query statement is determined according to an execution plan of the target slow query statement, the first type corresponds to full-table scanning, full-index scanning and the situation that the number of scanning lines is larger than a preset numerical value, the second type corresponds to the full-partition scanning situation, and a lexical analysis mode determined according to the slow query type is utilized, performing lexical analysis on the target slow query statement to obtain a first analysis result, and displaying first processing information in response to the condition that the first analysis result comprises a preset slow query reason of the target slow query statement, the first processing information comprises at least one of a preset slow query reason, a recommended optimization writing method corresponding to the preset slow query reason and optimization suggestion information corresponding to the preset slow query reason. By adopting the technical scheme, the optimization efficiency of the slow query statement can be improved.

The application provides a SQL statement query optimization method, device, medium and product. The SQL statement query optimization method comprises the following steps: obtaining a SQL statement; if it is found that there is an inequality predicate in the SQL statement, extracting the inequality predicate to obtain an original predicate structure; rewriting the original predicate structure into an equivalent predicate structure to obtain a candidate predicate structure; and determining an execution path by comparing the query costs of the original predicate structure and the candidate predicate structure. In this way, the inequality predicate structure which cannot be applied to partition pruning and index can be rewritten into a predicate structure which can be applied to partition pruning and index, thereby helping to reduce the number of areas that need to be scanned, or helping to avoid full table scanning and reducing disk I / O.

The present invention provides a data processing method, electronic device, and storage medium. In the present invention, an index tree scan method is used in conjunction with a mechanism for randomly sampling data pages of a database table. This method eliminates the need for a full table scan of the target database table when obtaining randomly sampled data from the target database table. This effectively reduces the number of disk I / O accesses, conserves disk I / O resources, and enhances the query performance of the database system, particularly in distributed database systems with massive amounts of data.

The invention discloses a data and performance verification method and device of a database, equipment, a medium and a product, and relates to the technical field of big data. The method comprises the steps of obtaining a table list; controlling production platform equipment according to the table list, and generating first metadata of the production environment database and a first execution plan statement of the first query statement; obtaining and checking whether the first metadata is consistent with second metadata corresponding to the test environment database to obtain a checking result; the production platform equipment is controlled, and the first execution plan statement is operated to screen out a first target query statement for full-table scanning; generating statistical information of tables involved in the first target query statement; displaying the verification result and the statistical information; wherein the production environment database and the test environment database are in network isolation. Under the condition that the plaintext information in the production environment database cannot be obtained, the data and performance of the production environment database and the test environment database which are in mutual network isolation are verified.

The invention discloses a progressive visibility mapping upgrading and freezing optimization method and device for a database. The method comprises the following steps of: additionally arranging a PDVMFROZEN flag bit at the head of a visibility mapping VM block to mark a new version (including all-frozen bit); setting a vmversion field in a table memory structure Relation, and dynamically detecting new and old VM versions; modifying a VM interface and adding a VMALLFROZEN macro compatible new version and an old version; automatically marking a new version for the new VM block, and gradually upgrading the old VM block; in the VACUUM process, a frozen block is skipped through VMALLFROZEN, and an all-open bit is automatically marked. According to the method, full-table scanning overhead is reduced, I / O and CPU occupation is reduced, online smooth upgrading is supported, long-term performance and operation and maintenance convenience of the database are improved, and compatibility of new and old versions and system stability are guaranteed.

The invention relates to the technical field of computers, in particular to a data comparison method and device based on an index clustering table, according to the data comparison method and device based on the index clustering table, the offset of a clustering primary key column value in a cache file serves as a virtual ROWID to replace a real ROWID, comparison is carried out, the virtual ROWIDs with different summary information are output after comparison is completed, and when a comparison report is generated, the comparison report is generated, and the data comparison efficiency is improved. Finding a corresponding clustering primary key column value in the cache file by using the virtual ROWID, and querying complete data from the to-be-compared clustering table through the clustering primary key column value to obtain a comparison result; when the data size of a clustering table is large and difference data exists, due to the fact that no first-level index exists on an ROWID column of the clustering table, full-table scanning is needed when complete difference data are queried in the clustering table through ROWID of a database, and efficiency is very low, however, according to the method, the complete data are queried by obtaining a clustering main key column through virtual ROWID mapping, and the efficiency is very high. And the performance of the whole comparison process is improved.

The application provides a database operation statement optimization method, medium, product and equipment. The method comprises the following steps: obtaining a database operation statement to be optimized, wherein the database operation statement comprises a target null judgment function, and the target null judgment function is an expression comprising a null value processing logic; analyzing the target null judgment function to determine the parameter type and operation logic of the target null judgment function; rewriting the target null judgment function into an equivalent conditional judgment expression according to the parameter type and operation logic; and replacing the target null judgment function in the database operation statement with the conditional judgment expression, thereby obtaining an optimized database operation statement. Through the method, the problems of large cardinality estimation error and inaccurate execution cost estimation caused by the use of the null judgment function in the existing database can be effectively solved, the optimizer can correctly identify the query condition and normally enable the index path to perform index scanning, and resource waste caused by full table scanning can be avoided.

The invention relates to the technical field of information processing, and discloses an unread message counting system and method based on read message recording and caching technology optimization, and the system comprises a message detail table which only stores the related information of read messages; the Redis cache is used for updating the unread message count of each user in real time; each user has an independent key-value pair, a key is a user ID, and a value is the number of current unread messages; the method further comprises a counting process of new message arrival, message reading and query of the number of unread messages. Only read messages are recorded, so that the storage space is greatly saved, the data storage requirement is greatly reduced, and the method is suitable for application scenes with large message quantity and relatively low reading rate; the Redis cache provides high-speed read-write operation, can complete data read-write within microsecond-level time, is used for updating and querying unread message counts in real time, and directly obtains the unread message counts by using the Redis cache, so that full-table scanning is avoided, and the query efficiency is greatly improved.

The invention discloses a statistical acceleration method based on a data template, and relates to the field of big data and real-time computing, when data is stored in a database, bitmap templates are generated according to historical data in a table, each template has an ID as a data virtual column, and the virtual columns are persistently stored in an FRC; when query is carried out, corresponding coding values constructed according to conditions are read firstly, intersection and union calculation is carried out in a template to find a conforming template ID set, whether a conforming template ID exists in each FRC file or not is judged, and then filtering optimization is achieved; the query performance is remarkably improved, efficient balance of storage and query is achieved, dynamic expansion and high-dimensional data processing are supported, automatic and intelligent template generation is achieved, an existing system architecture is compatible, and the problems that when an existing database system processes query under the multi-column null value combination condition, an efficient data filtering mechanism is lacked, the query performance is low, and the query efficiency is high are solved. In a mass data scene, the problems of frequent full-table scanning, large I / O overhead, long response time and the like occur.

The invention discloses a semantic retrieval method and system for structured data, and relates to the technical field of data analysis, and the method comprises the following steps: setting a label of field information in a relational database, expanding a field initial code according to input data, generating a field expansion code, and querying a jump order, and updating the Markdown document in combination with the field extension code and the query jump order, obtaining jump field information, and constructing a corresponding relationship between the query jump order and the second timestamp. According to the method, the retrieval range is dynamically adjusted by querying the jump order, full-table scanning is avoided, redundant calculation is remarkably reduced, field codes are automatically expanded according to input data, service rapid iteration is adapted, document changes are marked through the first timestamp, the latest effective fields are positioned in combination with the jump order, and historical data are prevented from being repeatedly processed; full-link automation from label setting to document generation, such as Markdown document automatic update field description.

The invention provides a database query optimization method and a related product. The query optimization method comprises the following steps: analyzing an input query instruction to generate a dynamic sampling instruction; executing the dynamic sampling instruction to dynamically sample the base table corresponding to the query instruction and the index of the base table to obtain a statistical result; on the basis of the statistical result, calculating the expected operation cost of the query instruction for performing query according to the access path based on the full-table scanning and the access path based on the index scanning; and performing data query by taking the access path corresponding to the minimum one of the expected operation costs as a target access path of the query instruction. According to the method, the sample coverage degree of dynamic sampling can be improved, the optimal access path can be determined in more access paths, and the effects of improving the database query performance and reducing the query overhead are achieved.

Embodiments of the present disclosure provide a data processing method, an electronic device, and a storage medium. In the embodiments of the present disclosure, in a process of obtaining random sampling data from the target database table, through an index tree scanning manner in combine with a mechanism for random sampling a data page of the database table, full-table scanning is not performed on the target database table, thereby effectively reducing the number of times that I / O access is performed on the disk, saving disk I / O resources, and enhancing the query performance of the database system, especially in the case that the distributed database system has mass data.

The invention discloses a data splitting method and device based on customer number prefixes, and the method comprises the steps: splitting a customer number interval according to the length of the customer number prefixes, and obtaining a plurality of prefixes; replacing the content in the LIKE condition of the SQL query statement with a prefix to obtain a query statement of the prefix; querying the total number of clients corresponding to all prefixes from a client information table by using the query statements of all prefixes; determining the number of target number segments according to the total number of clients and the node processing capability; and distributing the total number of customers according to the number of the target number segments and the prefix set to obtain a plurality of customer number segments. Data splitting is carried out based on customer number prefixes, and loads are reasonably distributed by comprehensively analyzing the prefix data size, the node processing capacity and the number of target number segments. In the query level, the LIKE condition is replaced by a specific prefix by means of a dynamic SQL to realize targeted query, so that full-table scanning is avoided fundamentally, I / O overhead is reduced greatly, and normal execution of a query task is guaranteed.

The invention discloses a cross-library updating optimization method and device and electronic equipment. The method comprises the steps that in the cross-database association updating process, execution plans of SQL in a source database and a target database are detected respectively, execution plan detection results are obtained, and the execution plan detection results comprise a remote interaction detection result corresponding to the source database and a full-table scanning and index detection result corresponding to the target database; under the condition that the execution plan detection result indicates that the driven table in the target database has index missing, the scanning cost of the driven table is determined according to the return row number of the driving table in the target database and the corresponding single scanning time; and adding an index in the driven table according to the scanning cost. According to the method and the device, the technical problems that when performance degradation occurs in large-batch cross-library updating, cross-library updating optimization in related technologies generally depends on manual intervention, an automatic optimization means is lacked, and high consistency of data cannot be ensured are solved.

Embodiments of the present disclosure provide a data processing method, an electronic device, and a storage medium. In the embodiments of the present disclosure, in a process of obtaining random sampling data from the target database table, through an index tree scanning manner in combine with a mechanism for random sampling a data page of the database table, full-table scanning is not performed on the target database table, thereby effectively reducing the number of times that I / O access is performed on the disk, saving disk I / O resources, and enhancing the query performance of the database system, especially in the case that the distributed database system has mass data.

The invention provides a block chain verifiable analysis type query processing method and device, and relates to the technical field of distributed databases, and the method comprises the steps: in an archiving node, verifying the credibility of each block, merging transactions, updating a state database and a read-write set hash, and constructing an MBT index structure to obtain an initialized system; the initialized system receives a query request initiated by a user, generates a query plan tree, processes operators in Enclave and signs a result, the user verifies the hash matching between the signature and the result, the archiving node performs full-table scanning and maintains read-write set hash, the consistency of the read set hash and the write set hash is compared, and a query result is obtained. The method is used for solving the performance bottleneck of a traditional zero-knowledge certification technology in high-dimensional aggregation operation in the prior art, and also solving the memory space bottleneck of a pure hardware scheme.

The invention relates to the field of computers, can be applied to business system platforms of financial science and technology, medical treatment and health and the like, and discloses an SQL logic detection and optimization method, device, equipment and medium. The method comprises the steps that in response to a script detection request, a to-be-detected SQL script code is obtained; table connection scene recognition is conducted on the SQL script code, and whether a table connection scene exists in the SQL script code or not is determined; if a table connection scene exists in the SQL script code, obtaining connection key information behind a specified keyword position; performing global retrieval on the SQL script code according to the connection key information, and confirming whether a screening condition for the partition field exists or not; and if the screening condition for the partition field does not exist, outputting risk prompt information of full-table scanning so as to optimize the SQL script code. The resource consumption risk of the current script is automatically identified to optimize the code standardization degree by detecting the mechanical energy full-table scanning risk of the SQL script code in the development process, so that the timeliness of the data processing task is improved.

Embodiments of the present application disclose a voucher generation method and related equipment, which are used for solving the problems of large database pressure and batch processing efficiency bottleneck caused by using database exclusive lock and full table scanning voucher serial number in the prior art. The method of the embodiments of the present application comprises: in response to a voucher generation request, obtaining voucher serial number cache information under a target accounting period and a target voucher serial number corresponding to the voucher generation request; based on the voucher serial number cache information, determining an occupied interval and a broken serial number interval of a historical voucher serial number; based on the broken serial number interval and the occupied interval, allocating at least one target voucher serial number for each voucher generation request; and generating a voucher corresponding to each target voucher serial number for the target voucher serial number allocated for each voucher generation request.

The invention relates to the technical field of data warehouses and multi-dimensional analysis, in particular to an automatic cubic table construction method based on metric dependency analysis, which comprises the following steps of: classifying metric fields, identifying and calculating a dependency relationship, and constructing a directed acyclic graph; based on the directed acyclic graph, generating a measurement construction path which depends on order; constructing a dimension level by adopting recursive aggregation from bottom to top, and caching an intermediate result to reduce full-table scanning; metric calculation is executed according to paths, parallel calculation of non-dependency / low-dependency metrics and increment construction are supported, and data consistency is guaranteed by combining updating-inserting double-stage writing. The method improves cubic table construction efficiency and data consistency, reduces resource consumption, adapts to multi-dimensional analysis requirements in the fields of education, finance, retail and the like, and is suitable for data warehouses and business intelligent systems.

The invention provides a report query method and device, equipment, a storage medium and a program product, relates to the technical field of data processing, and can reduce the error rate of manually configuring SQL query statements and improve the query efficiency. According to the specific technical scheme, when a user inputs a query request, a system firstly understands the intention of the user and generates a corresponding SQL statement; secondly, a preliminary query scheme is formulated according to the actual situation of the database, index scanning or full-table scanning is determined, and an SQL statement structure is optimized; and then predicting the calculation complexity, the data scanning amount and the parallel feasibility of the query, and finally determining and executing a query execution plan in combination with the load condition of the current database. In the execution process, the database load can be checked in real time, and the execution strategy is dynamically adjusted. Finally, a visual report is generated according to a query result, and chart display and export functions are supported.

The invention discloses a statistical method for residence duration of Internet of Things equipment and a terminal. The statistical method comprises the following steps: acquiring a station identifier, an equipment identifier and a state event of each piece of Internet of Things equipment; taking the station identifier and the equipment identifier of each piece of Internet of Things equipment as index identifiers, and storing the latest timestamp of the state event of each piece of Internet of Things equipment; in response to a statistical requirement for the target Internet of Things equipment, reading the latest timestamps corresponding to the state events of the target Internet of Things equipment according to the index identifiers, and sorting the state events according to the sequence of the timestamps; and determining the state event with the top timestamp in the sequence as the current state of the target Internet of Things equipment, and taking the difference value between the current time and the timestamp of the current state as the residence duration of the current state of the target Internet of Things equipment. According to the method, the dependence on CPU and memory resources is greatly reduced, and the performance risk caused by full-table scanning is reduced.

The invention relates to a graph database attribute management method and system, and the method and system have the beneficial effects that: by setting a double-default-value mechanism, the reading and writing-in default value logic of attributes are separated, and the compatibility of flexible modification of default values and historical data constraints is realized. Data backfilling is avoided, default values are managed through metadata, and full-table scanning and batch updating are avoided; dynamic flexibility is provided, and iterative updating of service logic is supported by modifying a write default value; the method has good compatibility and is transparent to an application layer, and existing query statements do not need to be modified. According to the method, the problem of conflict between default value modification and historical data constraint is successfully solved, the condition that index data and original data are inconsistent is avoided, the problem that data inquired by using indexes and data inquired by not using the indexes are inconsistent is prevented, and the problem that original data values are influenced by metadata modification is solved.

The invention relates to the technical field of intelligent factory network security, in particular to a dynamic micro-isolation strategy processing system and method based on machine learning, and the system comprises a data processing module, a model training module, a strategy decision module and a strategy execution module. Extracting a flow characteristic, a time sequence characteristic and an access mode characteristic; the model training module is used for training a strategy prediction model based on the bidirectional gating recurrent neural network; the strategy decision module inputs real-time feature data, and triggers full-table scanning when the confidence coefficient is low; and the strategy execution module implements an isolation strategy and feeds back effect data to optimize the model. The method comprises the steps of data processing, model training, strategy decision making and strategy execution, a historical access mode is automatically learned through machine learning, and a micro-isolation strategy is dynamically generated. According to the method, the problems of low efficiency and poor flexibility of a traditional static strategy are solved, and the processing efficiency and the safety protection capability are remarkably improved.

The invention relates to the technical field of big data and the field of financial science and technology, and discloses a histogram-based query optimization method, which comprises the following steps: determining database tables and associated fields thereof, generating histograms of the associated fields, comparing data distribution densities of the associated fields, and selecting the database table with the most sparse data distribution density as a propulsion reference table. Traversing associated field data of the reference table item by item, generating a matching range in combination with the distribution information of the non-reference table, executing a matching operation in the matching range, and if the matching fails, generating a new matching range according to the next data item of the reference table and executing the matching operation, and finally, ending the operation and generating a query result set when the traversal of the data items is completed or a matching termination condition is met. According to the method, the data distribution characteristics of the associated fields are accurately described through the histogram, dynamic optimization of the matching range is achieved, full-table scanning and unnecessary matching operation are effectively avoided, the execution efficiency of connection query is remarkably improved, and meanwhile the resource occupancy rate is reduced.

The invention provides a data restoration method and device based on data comparison and a medium, and relates to the technical field of data processing. The data packets are distributed to a destination library repair component through message middleware, the repair component starts multi-thread parallel processing, and each thread receives a data packet task from the middleware in sequence; according to the identifier of the data packet, executing interval query by using a primary key, obtaining a corresponding data block in a target library, and forming an ordered data set consistent with a source end structure; performing linear traversal comparison on the same primary key interval data blocks of the source end and the target library to generate a difference data list; and repairing and synchronizing the difference data. According to the method, the data interval is divided through the primary key or the hidden column, and multi-thread parallel processing is combined, so that full-table scanning is avoided, the data volume of single operation is reduced, and the repair efficiency is improved. The multi-thread parallel processing can greatly improve the processing speed, improve the data recovery efficiency and reduce the processing time.