Patents
Literature
Hiro is an intelligent assistant for R&D personnel, combined with Patent DNA, to facilitate innovative research.
Hiro

42 results about "Full table scan" patented technology

A full table scan (also known as a sequential scan) is a scan made on a database where each row of the table is read in a sequential (serial) order and the columns encountered are checked for the validity of a condition. Full table scans are usually the slowest method of scanning a table due to the heavy amount of I/O reads required from the disk which consists of multiple seeks as well as costly disk to memory transfers.

Self discovering adaptive security system and method

A system and method for providing security to a data warehousing application. A system is described having: a dimension analysis system that identifies dimensions defined for viewing data in a data warehouse; a security table generation system for automatically generating security tables for identified dimensions; a security tables scanning system for analyzing inputs within the security tables; and an end-user view definitions update system for updating the data warehouse based on inputs in the security tables.
Owner:TWITTER INC

OLAP star connection query optimizing method under CPU and GPU mixing framework

The invention discloses an OLAP (On-Line Analytical Processing) star connection query optimizing method under a CPU (Central Processing Unit) and GPU (Graphics Processing Unit) mixing framework. The method comprises the following steps: first filtering and optimizing an OLAP star connection operation through bitmap connection indexing, buffering high-frequently accessed connecting bitmap in a GPU buffer memory, next, loading fact table external key attribute group satisfying the connecting bitmap filtering condition into the GPU buffer memory to carry out star connection filtering, and finally switching the whole table scanning of the big fact table of the internal memory into accessing randomly according to the position through filtering bitmap generated by the GPU, thereby improving the querying and treating performances of the OLAP star connection. The method improves the storage efficiency of the GPU buffer memory and the parallel processing efficiency of the GPU, and improves the OLAP querying and treating performance of the hybrid processor platform on the whole.
Owner:RENMIN UNIVERSITY OF CHINA

Data storage optimization method for hash joint

The invention relates to a data storage optimization method for a hash joint. The method includes the steps of selecting a foreign key of a radix hash storage on a fact table, determining a dimension table, setting n bits of radix base numbers, determining 2n hash partition storages, setting 2n radix hash partition queues, storing the fact table and the dimension table by means of a page chain table, carrying out PAX column storage in a page, carrying out radix hash on recorded hash partition attribute values according to low n bits when records are inserted, storing the hash partition attribute values in corresponding pages, dynamically applying a new page after a certain radix hash partition page is filled with the records, directly having access to a specific radix hash partition queue according to a hash value of a connection key, having access to all records of the radix hash partition according to page addresses stored in the queue, having access to the records according to an original physical page link sequence of the tables when full-table scan is carried out, storing a small table R and a large table S in a partition manner, adopting a column type connection method to the fact table during hash joint in a database, and increasing or reducing the bits of the radix hash partitions to achieve dynamic increasing and reducing of the hash partitions.
Owner:RENMIN UNIVERSITY OF CHINA

Database partition method and device

ActiveCN102063490AGuaranteed timeImprove the efficiency of full table scanSpecial data processing applicationsBusy hourFull table scan
The invention provides a database partition method used for partitioning a database. The method comprises the following steps of: acquiring multiple required report forms; determining database partition rules according to time nodes of the multiple required report forms; and partitioning the database according to the determined database partition rules. The database partition method is used for analyzing the time nodes of the required reports and customizing the partition rules according to the time nodes, so that the data of various reports such as full network daily reports, busy hour reports and the like best accord with the characteristics of full partition table scanning strategies, the required data can be acquired during full partition table scanning, the scanned records which do not meet the search conditions reach the minimum, the time required for searching and scanning is reduced, the full table scanning efficiency is improved, and many original conditions of needing index scanning are changed into conditions of full table scanning. The invention also provides a database partition device.
Owner:DATANG MOBILE COMM EQUIP CO LTD +1

Data encryption and retrieval method for database

The invention discloses a data encryption and retrieval method for a database. The method includes the steps: a) encrypting raw data by means of a one-way encryption algorithm to generate a non-decipherable indexable head under the condition of keeping sequence of the raw data unchanged; b) continuing to encrypt the raw data to generate decipherable ciphertext data; c) combining the indexable head and the decipherable ciphertext data to generate encryption combined data, and storing the encryption combined data in the database, wherein the one-way encryption algorithm in the step a) refers to MD5 (message-digest algorithm 5), SHA (secure hash algorithm) or HMAC (hash message authentication code); the encryption algorithm in the step b) refers to the symmetric key encryption algorithm. The data encryption and retrieval method for the database has the advantages that since the indexable head and the decipherable ciphertext data are combined for encrypted storage of all of the raw data by means of cipertext, and the indexable head are used for quick retrieval of the encrypted data, full table scanning is avoided while accuracy in data decryption and security in data retrieval are guaranteed.
Owner:上海新炬网络技术有限公司

System and method for integrated searching of structured data and unstructured data

The invention discloses a system and a method for integrated searching of structured data and unstructured data. After original data are stored into a data base, a B+ tree index is established for the structured data, and a reverse index is established for the unstructured data; during searching, the B+ tree index is queried in case of structured searching, the reverse index is queried in case of unstructured searching, the B+ tree index and the reverse index are queried respectively in case of mixed structured and unstructured searching, query results of the indexes are merged, and finally, the original data are obtained according to the query results of the indexes. According to the method, the problem of low efficiency caused by full table scanning performed during the unstructured searching of the current data base is solved, the performance of the structured searching is guaranteed, and meanwhile, the performance of performing of the unstructured searching in the data base is greatly improved.
Owner:南京烽火星空通信发展有限公司

Method and system by using temporary performance objects for enhanced query performance

The invention provides a method and a system by using temporary performance objects for enhanced query performance, and discloses a method and an apparatus, including computer program products, implementing and using techniques for optimizing query performance in a relational database management system. A query is received at the relational database management system. It is determined whether the query is expected to be a long-running query. In response to determining that the query is expected to be a long-running query, a full table scan is started in order to fetch records needed to satisfy the query from the relational database management system. In parallel with conducting the full table scan, a performance object is built, which is capable of satisfying the query in the relational database management system. In response to completing the construction of the performance object prior to completing the full table scan, the full table scan is stopped and the newly built performance object is used instead to satisfy the query.
Owner:IBM CORP

Indexing method supporting time series data aggregation function

The invention discloses an indexing method supporting time series data aggregation function, which supports fast ad hoc query of a simple aggregation operation. The basic thought of the method is that a summary table and segment trees (Segment Tree) are combined, and a segment forest model formed by multiple segment trees is established on the summary table, so that the full table scan operation of the summary table is avoided. Meanwhile, through dynamically constructing a segment forest in a bottom-up method, the defect that the conventional segment tree does not support increase is avoided. In addition, a query algorithm directly positions index data through calculation through calculation, the recursive traversal operation of the segment forest is avoided, and the frequency for disk IO is reduced. An experimental result shows that through adoption of a calculation query way of the summary table and the segment forest, the frequency for disk IO is effectively reduced and the query performance is remarkably improved.
Owner:TSINGHUA UNIV

A method and system for efficient performance prediction of structured query for big data

A system for predicting an Elapsed Response Time (ERT) of a query to be executed in a production database comprises a db emulator module for creating an emulated database based on empty schema derived by extrapolating statistical information from the development environment, an execution plan generation module for generating a query execution plan of the query, a query classifying module for classifying each sub-query based on the operational parameters into at least one of a full table scan, a primary index scan, a fast index scan and a non-unique index scan, a time determination module for executing each sub-query on the emulated database to determine an Input Output (IO) access time and an operator time, and an ERT prediction module for predicting the ERT of the query to be executed in the production database based on the I0 access time, the operator time, and a network time.
Owner:TATA CONSULTANCY SERVICES LTD

Data table query method and device, terminal equipment and storage medium

The embodiment of the invention discloses a data table query method and device, terminal equipment and a storage medium. The method comprises the steps that a query operation, aiming at a target datatable, in a current affair is received, the query start time of receiving the query operation is recorded, whether or not a modification operation aiming at the target data table exists in the influence time period from the affair start time of the current affair to the query start time is judged, the total number of records affecting the target data table is determined, and the total number of target records of the target data table is determined according to the total number of initial records of the target data table predetermined at the query start time and the total number of the affectedrecords. Accordingly, the total number of the target records of the target data table is determined through the total number of the initial records of the predetermined target data table and the total number of the affected records, the problem that full table scanning is conducted when data table query is conducted, and the query efficiency is low is solved, the data table query efficiency is improved, occupied memory resources are reduced, and the overall performance of a database is optimized.
Owner:SHANGHAI DAMENG DATABASE

Stratified sampling method and device and computer equipment

The invention provides a stratified sampling method and device and computer equipment. The stratified sampling method includes: storing to-be-sampled data in a database into a data table; acquiring to-be-sampled layers, partitioning the data table according to the to-be-sampled layers, and sorting data in each partition corresponding to each to-be-sampled layer; acquiring the sampling proportion corresponding to each to-be-sampled layer, and sampling the sorted data according to the sampling proportion. The stratified sampling method has the advantages that direct stratified sampling can be achieved on the basis of SQL, additional data process scripts for reading and processing the data do not need to be compiled, the data sources are stored in the same table, and data redundancy caused byindependent table building for each layer of data; in addition, independent sampling is performed on the partition corresponding to each to-be-sampled layer, and high resource occupation caused by one-step whole-table scanning can be avoided.
Owner:TENCENT TECH (SHENZHEN) CO LTD

Database substring filtering index system and method for constructing and inquiring database substring filtering index system

The invention relates to a database substring filtering index system and a method for constructing and inquiring the database substring filtering index system. The database substring filtering index system is characterized by comprising a fragmented data storage module and a filtering index storage module, wherein the fragmented data storage module is used for storing input data in a fragmented manner and extracting the characteristics of the substrings describing the fragmented data from each fragmented data to construct one or more substring characteristic filter bitmaps; and the filtering index storage module is used for storing substring characteristic filter bitmaps to be substring characteristic indexes; the invention further provides the method for constructing and inquiring the substring filtering index system. The database substring filtering index system has a rational design, solves the problem that the traditional index space is large, and the compressed data is difficult to index, achieves the function of efficiently inquiring substrings and keywords of massive text data, thereby greatly reducing the disk reading-writing times to data and improving full table scanning performance of the database.
Owner:天津神舟通用数据技术有限公司

Index system and index method for retrieving time sequences of ultra-large-scale data

ActiveCN106777111AHigh speedAvoid the risks of full table scan operationsSpecial data processing applicationsHash functionFull table scan
The invention discloses an index system and an index method for retrieving time sequences of ultra-large-scale data. The index system comprises a vertical hierarchical index module and a corresponding time axis index module. The vertical hierarchical index module comprises a plurality of hierarchies of indexes, each hierarchy of indexes comprises a hash function and a plurality of data sets, and data in the data sets are mapped into a corresponding next hierarchy of data set by the aid of the hash function of the corresponding next hierarchy of indexes; the time axis index module is used for creating event lists and time lists. The index system and the index method have the advantages that original big data sets can be ultimately distributed into a plurality of small data sets by means of hierarchy -by- hierarchy hash mapping by the aid of the hierarchical index module, operation such as query processing, data loading and storage optimization can be independently executed on each small data set, accordingly, the vertical hierarchical index module can be combined with the time axis index module, risks of full-table scanning operation in time sequence retrieval operation procedures can be prevented, and the time sequence retrieval speeds can be greatly increased; the index system and the index method for retrieving the time sequences of the ultra-large-scale data can be widely applied to the field of data processing.
Owner:SOUTH CHINA NORMAL UNIVERSITY

Database index search method and apparatus, and user terminal

The invention relates to a database index search method and apparatus, and a user terminal. The method comprises the steps of receiving a query instruction, wherein the query instruction comprises to-be-searched character segments and a query character; according to the to-be-searched character segments, performing matching in a data segment storage table to obtain one or more corresponding character segments, and correspondingly obtaining one or more main keys according to the character segments; and according to the one or more main keys, performing matching in a source table to obtain records of one or more corresponding source field contents. According to the method and the apparatus, data needed to be searched for is divided into data segments, then the data segments are associated with the source table, and a back wildcard character mode is adopted during query, so that the problem of full table scanning caused by use of Like query in a front wildcard character mode is solved and the query performance is improved.
Owner:GUANGDONG YOUMAI INFORMATION COMM TECH

HBase secondary-index storage and query system and query method thereof

The invention relates to the technical field of data processing, in particular to an HBase secondary-index storage and query system and a query method thereof. The HBase secondary-index storage and query system comprises a client-side, a query processing module, a query execution engine module and an HBase storage module. The HBase secondary-index storage and query system stores a data table in an SSD and is higher in data read-write speed compared with a traditional SATA hard disk, and the query efficiency is greatly improved. In addition, the HBase secondary-index storage and query system optimizes matching logic, can use a filter column condition included in an index table as a filter to conduct scan on the index table so as to obtain a rowkey of the data table, then uses a query condition as a filter to conduct accurate get on the data table by using the rowkey so as to obtain a query result, omits whole-table scanning of the data table and greatly improves the query efficiency.
Owner:FENGHUO COMM SCI & TECH CO LTD

Data processing method and device for distributed database and storage medium

The invention discloses a data processing method and device for a distributed database and a storage medium. The method comprises the steps of generating a service feature bit sequence based on a service field value corresponding to a service field, generating an identification feature bit sequence corresponding to to-be-processed data according to current time, and splicing the service feature bit sequence and the identification feature bit sequence to generate a row key Rowkey of the to-be-processed data. According to the method, the device and the storage medium, the service and the query performance are integrated, the Rowkey can be quickly generated, the Rowkey can be used for query, full-table scanning is avoided, and the query speed is increased. By using the Rowkey generated by theself-defined 64-system mask, it can be guaranteed that the binary sequence of Rowkey generation is from small to large, the sorting rule of the HBase insertion sequence is met, and the Rowkey generation range can be rapidly determined.
Owner:北京航天智造科技发展有限公司

Inquiry method, device and apparatus for electronic invoice and storage medium for electronic invoice

The invention discloses an inquiry method, device and apparatus for an electroinc device, and a storage medium. The method comprises the following steps of: acquiring a query request of an electronicinvoice, wherein the query request comprises data to be queried, data attributes and data attribute relationships; obtaining an invoice identification list, wherein the invoice identification is obtained from an invoice index table according to the data to be queried, and the attributes of the invoice identification list are the same as the data attributes; obtaining an electronic invoice identification set, wherein the electronic invoice identification set is a screening result of an invoice identification list according to a data attribute relationship; and obtaining the electronic invoice data corresponding to the electronic invoice identifier in the set of electronic invoice identifiers. The invention solves the problem that the query efficiency of the electronic invoice data is low due to the fact that the relational database is adopted in the prior art for storing and managing the billions-level invoice data, achieves that the electronic invoice data can be obtained without scanning the whole table of the electronic invoice data table, thereby improving the technical effect of the query efficiency of the electronic invoice data.
Owner:彩讯科技股份有限公司

Data real-time synchronization method and device, equipment and storage medium

The invention discloses a real-time data synchronization method and device, equipment and a storage medium. The method comprises the steps of creating a plurality of ES indexes based on a synchronization system; the synchronization system creates heterogeneous data based on the updated data; the synchronization system writes the heterogeneous data into the corresponding ES indexes; the synchronization system sends each piece of operation information of the ES to a compensation system; the compensation system determines a compensation target based on the operation information. According to the technical scheme, real-time synchronization of Mysql and ES data is achieved, the data can be directly queried from the ES index without being queried through Mysql full-library full-table scanning, the data query efficiency is improved, the mass data query requirement is met, and service processing and data synchronization are completely decoupled; in addition, data loss caused by various accidents is automatically compensated through timing comparison between heterogeneous data sources, and the data integrity is ensured under the condition that manual intervention is not needed.
Owner:ZHONGAN ONLINE P&C INSURANCE CO LTD

Method for constructing Hbase secondary index table

The invention discloses a method for constructing an Hbase secondary index table, and relates to the field of databases. Hbase only provides query based on row keys and full-table scanning, full-tablescanning is needed when non-row keys are queried, and the query efficiency is low. The method comprises the following steps: establishing an index table for an original table on an hbase table; for each record of the original table, recording; generating corresponding index records on the index table, row keys of the index records comprise partition keys, index values, connectors and original table main keys, wherein the partition keys are start key fields of regions where the index records are located, the partition keys serve as the index table main keys, and pre-partitioning the partitionkeys in a random hash mode in combination with the index table; pre-defining the partition value; determining an index partition value in advance when an index table is established; and pre-partitioning the index table according to the index partition value when the index table is established. Parallel performance can be effectively improved, and query efficiency is improved.
Owner:ZHEJIANG HUAYUN INFORMATION TECH CO LTD

Big data storage, analysis and query system

The big data storage, analysis and query system comprises a distributed data storage framework, a distributed data analysis framework, a distributed data search framework and a data visual display framework; the distributed data storage architecture is used for providing high throughput to access data of an application program with a super-large data set; the distributed data analysis architectureis a general engine and is used for completing processing operation of large-scale data; the distributed data search architecture is used for a distributed mass data search engine in various scenes;and the data visualization display framework adopts an MVC structure. Through distributed storage and query, a plurality of nodes perform storage and querying at the same time, so that the efficiencyis improved, and the time is saved. the query in the mass data, compared with a traditional database in which full-table scanning is adopted and the time is slow, in distributed query technologies, the same data can reach the second-level speed, the analysis types are rich, various data samples are supported, distributed analysis is supported, and the analysis speed is greatly increased.
Owner:写逸网络科技(上海)有限公司

Data processing method and device, electronic equipment and computer storage medium

The embodiment of the invention provides a data processing method and device, electronic equipment and a computer storage medium. According to the scheme provided by the embodiment of the invention, during linked table query, a filtering condition operator of linked table query is converted into a conjunctive normal form composed of N expressions capable of being pushed down, and the expressions capable of being pushed down represent expressions executed before a data table is connected with a join operator. According to the N sub-query tasks corresponding to the N expressions capable of being pushed down, the N sub-query tasks are processed to obtain N query sub-tables; and the N query sub-tables are combined to generate a query result. Through equivalent conversion of the expression of the filtering condition operator, filtering is executed before data table connection, full table scanning is avoided, data scanning overhead is greatly reduced, and query time is shortened.
Owner:ALIBABA GRP HLDG LTD

Multi-condition paging query optimization method and system, terminal and storage medium

The invention provides a multi-condition paging query optimization method and system, a terminal and a storage medium. The multi-condition paging query optimization method comprises the steps of acquiring related tables according to query conditions, and counting the number of related data of the related tables; acquiring the sequence of each correlation table and the data quantity of each page atthe front end; calculating the number of target data according to the number of target query pages at the front end and the number of data of each page; and determining the correlation table to whichthe target query page number belongs according to the target data number, the data number of each page, the correlation data number of each correlation table and the sequence. Full-table scanning isavoided, a temporary table does not need to be generated in an associated mode, and the data query efficiency is improved.
Owner:SUZHOU LANGCHAO INTELLIGENT TECH CO LTD

Method and device for eliminating differences between hollow character strings and invalid character strings in database

The invention provides a method and device for eliminating differences between hollow character strings and invalid character strings in a database. The method comprises the steps of obtaining all character type columns in a service table from a data dictionary of the database; verifying whether all character type columns in the service table are provided with credible marks or not according to the credible mark record table, If yes, ending the step; if not, performing full-table scanning on the character type column data without the credible mark, and detecting whether an empty character string exists or not; if the empty character string exists, updating the empty character string into an invalid character string, otherwise, setting credible marks of the character type columns in the service table and the related views, and recording the credible marks into a credible mark recording table; and obtaining a credible mark of the character type column of the service table from the credible mark record table, and recompiling the standard database statement according to the credible mark of the character type column to obtain a database native structured query language.
Owner:DAREWAY SOFTWARE

A time series retrieval index system and method for super large-scale data

ActiveCN106777111BHigh speedAvoid the risks of full table scan operationsSpecial data processing applicationsDatabase indexingHash functionFull table scan
The invention discloses an index system and an index method for retrieving time sequences of ultra-large-scale data. The index system comprises a vertical hierarchical index module and a corresponding time axis index module. The vertical hierarchical index module comprises a plurality of hierarchies of indexes, each hierarchy of indexes comprises a hash function and a plurality of data sets, and data in the data sets are mapped into a corresponding next hierarchy of data set by the aid of the hash function of the corresponding next hierarchy of indexes; the time axis index module is used for creating event lists and time lists. The index system and the index method have the advantages that original big data sets can be ultimately distributed into a plurality of small data sets by means of hierarchy -by- hierarchy hash mapping by the aid of the hierarchical index module, operation such as query processing, data loading and storage optimization can be independently executed on each small data set, accordingly, the vertical hierarchical index module can be combined with the time axis index module, risks of full-table scanning operation in time sequence retrieval operation procedures can be prevented, and the time sequence retrieval speeds can be greatly increased; the index system and the index method for retrieving the time sequences of the ultra-large-scale data can be widely applied to the field of data processing.
Owner:SOUTH CHINA NORMAL UNIVERSITY

A data table query method, device, terminal equipment and storage medium

The embodiment of the invention discloses a data table query method and device, terminal equipment and a storage medium. The method comprises the steps that a query operation, aiming at a target datatable, in a current affair is received, the query start time of receiving the query operation is recorded, whether or not a modification operation aiming at the target data table exists in the influence time period from the affair start time of the current affair to the query start time is judged, the total number of records affecting the target data table is determined, and the total number of target records of the target data table is determined according to the total number of initial records of the target data table predetermined at the query start time and the total number of the affectedrecords. Accordingly, the total number of the target records of the target data table is determined through the total number of the initial records of the predetermined target data table and the total number of the affected records, the problem that full table scanning is conducted when data table query is conducted, and the query efficiency is low is solved, the data table query efficiency is improved, occupied memory resources are reduced, and the overall performance of a database is optimized.
Owner:SHANGHAI DAMENG DATABASE

Method and device for determining data comparison result and equipment

The invention provides a method and device for determining a data comparison result, and equipment The method comprises the following steps that: data in a main table and an auxiliary table are loadedinto a data comparison table, wherein the table structure of the data comparison table comprises a main table field and a comparison field; the initial value of the comparison field corresponding toa row with the same data corresponding to the main table field in the data comparison table isupdated into a first numerical value, so that a first data comparison table is obtained; based on a main key field included in the main table field, rows of which comparison fields in the first data comparison table are not a first numerical value are grouped; the initial value of each row of comparison fields in each group is updated according to the number of rows included in each group, so that a second data comparison table is obtained; and a comparison result corresponding to each row of data inthe second data comparison table is determined according to a preset relationship between a comparison field value and the comparison result. By utilizing the method, device and equipment of the embodiment of the invention, full-table scanning can be avoided, and the comparison efficiency is improved.
Owner:CHINA CONSTRUCTION BANK
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products