42 results about "Full table scan" patented technology

The invention relates to a data storage optimization method for a hash joint. The method includes the steps of selecting a foreign key of a radix hash storage on a fact table, determining a dimension table, setting n bits of radix base numbers, determining 2n hash partition storages, setting 2n radix hash partition queues, storing the fact table and the dimension table by means of a page chain table, carrying out PAX column storage in a page, carrying out radix hash on recorded hash partition attribute values according to low n bits when records are inserted, storing the hash partition attribute values in corresponding pages, dynamically applying a new page after a certain radix hash partition page is filled with the records, directly having access to a specific radix hash partition queue according to a hash value of a connection key, having access to all records of the radix hash partition according to page addresses stored in the queue, having access to the records according to an original physical page link sequence of the tables when full-table scan is carried out, storing a small table R and a large table S in a partition manner, adopting a column type connection method to the fact table during hash joint in a database, and increasing or reducing the bits of the radix hash partitions to achieve dynamic increasing and reducing of the hash partitions.

The invention discloses a data encryption and retrieval method for a database. The method includes the steps: a) encrypting raw data by means of a one-way encryption algorithm to generate a non-decipherable indexable head under the condition of keeping sequence of the raw data unchanged; b) continuing to encrypt the raw data to generate decipherable ciphertext data; c) combining the indexable head and the decipherable ciphertext data to generate encryption combined data, and storing the encryption combined data in the database, wherein the one-way encryption algorithm in the step a) refers to MD5 (message-digest algorithm 5), SHA (secure hash algorithm) or HMAC (hash message authentication code); the encryption algorithm in the step b) refers to the symmetric key encryption algorithm. The data encryption and retrieval method for the database has the advantages that since the indexable head and the decipherable ciphertext data are combined for encrypted storage of all of the raw data by means of cipertext, and the indexable head are used for quick retrieval of the encrypted data, full table scanning is avoided while accuracy in data decryption and security in data retrieval are guaranteed.

The invention discloses a system and a method for integrated searching of structured data and unstructured data. After original data are stored into a data base, a B+ tree index is established for the structured data, and a reverse index is established for the unstructured data; during searching, the B+ tree index is queried in case of structured searching, the reverse index is queried in case of unstructured searching, the B+ tree index and the reverse index are queried respectively in case of mixed structured and unstructured searching, query results of the indexes are merged, and finally, the original data are obtained according to the query results of the indexes. According to the method, the problem of low efficiency caused by full table scanning performed during the unstructured searching of the current data base is solved, the performance of the structured searching is guaranteed, and meanwhile, the performance of performing of the unstructured searching in the data base is greatly improved.

The invention discloses an indexing method supporting time series data aggregation function, which supports fast ad hoc query of a simple aggregation operation. The basic thought of the method is that a summary table and segment trees (Segment Tree) are combined, and a segment forest model formed by multiple segment trees is established on the summary table, so that the full table scan operation of the summary table is avoided. Meanwhile, through dynamically constructing a segment forest in a bottom-up method, the defect that the conventional segment tree does not support increase is avoided. In addition, a query algorithm directly positions index data through calculation through calculation, the recursive traversal operation of the segment forest is avoided, and the frequency for disk IO is reduced. An experimental result shows that through adoption of a calculation query way of the summary table and the segment forest, the frequency for disk IO is effectively reduced and the query performance is remarkably improved.

The invention provides a stratified sampling method and device and computer equipment. The stratified sampling method includes: storing to-be-sampled data in a database into a data table; acquiring to-be-sampled layers, partitioning the data table according to the to-be-sampled layers, and sorting data in each partition corresponding to each to-be-sampled layer; acquiring the sampling proportion corresponding to each to-be-sampled layer, and sampling the sorted data according to the sampling proportion. The stratified sampling method has the advantages that direct stratified sampling can be achieved on the basis of SQL, additional data process scripts for reading and processing the data do not need to be compiled, the data sources are stored in the same table, and data redundancy caused byindependent table building for each layer of data; in addition, independent sampling is performed on the partition corresponding to each to-be-sampled layer, and high resource occupation caused by one-step whole-table scanning can be avoided.

The invention discloses an index system and an index method for retrieving time sequences of ultra-large-scale data. The index system comprises a vertical hierarchical index module and a corresponding time axis index module. The vertical hierarchical index module comprises a plurality of hierarchies of indexes, each hierarchy of indexes comprises a hash function and a plurality of data sets, and data in the data sets are mapped into a corresponding next hierarchy of data set by the aid of the hash function of the corresponding next hierarchy of indexes; the time axis index module is used for creating event lists and time lists. The index system and the index method have the advantages that original big data sets can be ultimately distributed into a plurality of small data sets by means of hierarchy -by- hierarchy hash mapping by the aid of the hierarchical index module, operation such as query processing, data loading and storage optimization can be independently executed on each small data set, accordingly, the vertical hierarchical index module can be combined with the time axis index module, risks of full-table scanning operation in time sequence retrieval operation procedures can be prevented, and the time sequence retrieval speeds can be greatly increased; the index system and the index method for retrieving the time sequences of the ultra-large-scale data can be widely applied to the field of data processing.

The invention relates to a database index search method and apparatus, and a user terminal. The method comprises the steps of receiving a query instruction, wherein the query instruction comprises to-be-searched character segments and a query character; according to the to-be-searched character segments, performing matching in a data segment storage table to obtain one or more corresponding character segments, and correspondingly obtaining one or more main keys according to the character segments; and according to the one or more main keys, performing matching in a source table to obtain records of one or more corresponding source field contents. According to the method and the apparatus, data needed to be searched for is divided into data segments, then the data segments are associated with the source table, and a back wildcard character mode is adopted during query, so that the problem of full table scanning caused by use of Like query in a front wildcard character mode is solved and the query performance is improved.

The invention relates to the technical field of data processing, in particular to an HBase secondary-index storage and query system and a query method thereof. The HBase secondary-index storage and query system comprises a client-side, a query processing module, a query execution engine module and an HBase storage module. The HBase secondary-index storage and query system stores a data table in an SSD and is higher in data read-write speed compared with a traditional SATA hard disk, and the query efficiency is greatly improved. In addition, the HBase secondary-index storage and query system optimizes matching logic, can use a filter column condition included in an index table as a filter to conduct scan on the index table so as to obtain a rowkey of the data table, then uses a query condition as a filter to conduct accurate get on the data table by using the rowkey so as to obtain a query result, omits whole-table scanning of the data table and greatly improves the query efficiency.

The invention discloses a data processing method and device for a distributed database and a storage medium. The method comprises the steps of generating a service feature bit sequence based on a service field value corresponding to a service field, generating an identification feature bit sequence corresponding to to-be-processed data according to current time, and splicing the service feature bit sequence and the identification feature bit sequence to generate a row key Rowkey of the to-be-processed data. According to the method, the device and the storage medium, the service and the query performance are integrated, the Rowkey can be quickly generated, the Rowkey can be used for query, full-table scanning is avoided, and the query speed is increased. By using the Rowkey generated by theself-defined 64-system mask, it can be guaranteed that the binary sequence of Rowkey generation is from small to large, the sorting rule of the HBase insertion sequence is met, and the Rowkey generation range can be rapidly determined.

The invention discloses an inquiry method, device and apparatus for an electroinc device, and a storage medium. The method comprises the following steps of: acquiring a query request of an electronicinvoice, wherein the query request comprises data to be queried, data attributes and data attribute relationships; obtaining an invoice identification list, wherein the invoice identification is obtained from an invoice index table according to the data to be queried, and the attributes of the invoice identification list are the same as the data attributes; obtaining an electronic invoice identification set, wherein the electronic invoice identification set is a screening result of an invoice identification list according to a data attribute relationship; and obtaining the electronic invoice data corresponding to the electronic invoice identifier in the set of electronic invoice identifiers. The invention solves the problem that the query efficiency of the electronic invoice data is low due to the fact that the relational database is adopted in the prior art for storing and managing the billions-level invoice data, achieves that the electronic invoice data can be obtained without scanning the whole table of the electronic invoice data table, thereby improving the technical effect of the query efficiency of the electronic invoice data.

The invention discloses a real-time data synchronization method and device, equipment and a storage medium. The method comprises the steps of creating a plurality of ES indexes based on a synchronization system; the synchronization system creates heterogeneous data based on the updated data; the synchronization system writes the heterogeneous data into the corresponding ES indexes; the synchronization system sends each piece of operation information of the ES to a compensation system; the compensation system determines a compensation target based on the operation information. According to the technical scheme, real-time synchronization of Mysql and ES data is achieved, the data can be directly queried from the ES index without being queried through Mysql full-library full-table scanning, the data query efficiency is improved, the mass data query requirement is met, and service processing and data synchronization are completely decoupled; in addition, data loss caused by various accidents is automatically compensated through timing comparison between heterogeneous data sources, and the data integrity is ensured under the condition that manual intervention is not needed.

The invention discloses a method for constructing an Hbase secondary index table, and relates to the field of databases. Hbase only provides query based on row keys and full-table scanning, full-tablescanning is needed when non-row keys are queried, and the query efficiency is low. The method comprises the following steps: establishing an index table for an original table on an hbase table; for each record of the original table, recording; generating corresponding index records on the index table, row keys of the index records comprise partition keys, index values, connectors and original table main keys, wherein the partition keys are start key fields of regions where the index records are located, the partition keys serve as the index table main keys, and pre-partitioning the partitionkeys in a random hash mode in combination with the index table; pre-defining the partition value; determining an index partition value in advance when an index table is established; and pre-partitioning the index table according to the index partition value when the index table is established. Parallel performance can be effectively improved, and query efficiency is improved.

The embodiment of the invention provides a data processing method and device, electronic equipment and a computer storage medium. According to the scheme provided by the embodiment of the invention, during linked table query, a filtering condition operator of linked table query is converted into a conjunctive normal form composed of N expressions capable of being pushed down, and the expressions capable of being pushed down represent expressions executed before a data table is connected with a join operator. According to the N sub-query tasks corresponding to the N expressions capable of being pushed down, the N sub-query tasks are processed to obtain N query sub-tables; and the N query sub-tables are combined to generate a query result. Through equivalent conversion of the expression of the filtering condition operator, filtering is executed before data table connection, full table scanning is avoided, data scanning overhead is greatly reduced, and query time is shortened.

The invention provides a multi-condition paging query optimization method and system, a terminal and a storage medium. The multi-condition paging query optimization method comprises the steps of acquiring related tables according to query conditions, and counting the number of related data of the related tables; acquiring the sequence of each correlation table and the data quantity of each page atthe front end; calculating the number of target data according to the number of target query pages at the front end and the number of data of each page; and determining the correlation table to whichthe target query page number belongs according to the target data number, the data number of each page, the correlation data number of each correlation table and the sequence. Full-table scanning isavoided, a temporary table does not need to be generated in an associated mode, and the data query efficiency is improved.

The invention provides a method and device for eliminating differences between hollow character strings and invalid character strings in a database. The method comprises the steps of obtaining all character type columns in a service table from a data dictionary of the database; verifying whether all character type columns in the service table are provided with credible marks or not according to the credible mark record table, If yes, ending the step; if not, performing full-table scanning on the character type column data without the credible mark, and detecting whether an empty character string exists or not; if the empty character string exists, updating the empty character string into an invalid character string, otherwise, setting credible marks of the character type columns in the service table and the related views, and recording the credible marks into a credible mark recording table; and obtaining a credible mark of the character type column of the service table from the credible mark record table, and recompiling the standard database statement according to the credible mark of the character type column to obtain a database native structured query language.