38 results about "Bitmap index" patented technology

The invention discloses a fault root cause positioning method and system for a server cluster, and relates to the technical field of network fault diagnosis. According to the method, nanosecond-level synchronous acquisition of micro-service call chains, container indexes, physical nodes and network data is realized through a precise time protocol, and a consistent data set is constructed through entity association and standardized processing; a service-resource topological graph is dynamically constructed, and an inter-service calling edge weight model is innovatively designed: a real-time load factor and a historical fault index attenuation sum processed by a Sigmoid function are fused, and the weight is periodically updated to accurately quantify the inter-node influence intensity; converting the topological graph into a Bayesian network; when a fault occurs, a three-level assembly line compression alarm is adopted, frequent item sets are mined through bitmap indexes and parallel FP-Growth, and strong causal association item sets are screened in combination with topological edge weights and KL divergence; strong causal alarm is taken as evidence, probabilistic root cause sorting is output through reverse random walk sampling, and high-precision positioning of complex distributed system faults is achieved.

The invention relates to the technical field of big data management, in particular to a data tracing method based on big data mining, which comprises the following steps of: acquiring an unstructured operation log, analyzing the unstructured operation log into a structured event record, and associating to generate a heterogeneous operation session set; performing access mode association analysis on the session set to extract a read-write path mode, and combining association strength and a time decay factor to determine a dependency weight and construct a blood relationship map; eliminating a loop of the graph to generate a directed acyclic graph, constructing a spanning tree serving as a retrieval trunk and a node index coding interval, and mapping a cross-branch associated edge to a bitmap index to generate a probability skeleton tree index; and positioning a trunk path by using the interval, calling a bitmap to perform multi-path verification, and outputting a traceability path and confidence. A probability consanguinity map and a probability skeleton tree index are constructed through mining logs, and data consanguinity reconstruction and large-scale node low-delay retrieval in the heterogeneous black box environment are achieved.

The invention provides a bitmap index container optimization method and system based on a multi-dimensional dynamic decision, and relates to the technical field of databases, and the method comprises the steps: constructing a feature fusion vector by calculating a skewness coefficient value and fusing a data distribution feature vector, a data access rule vector and a change trend vector; a self-adaptive threshold value is obtained through iterative optimization of a double-strategy gradient algorithm, whether the container type is switched or not is judged, data deduplication, priority ranking and batch data migration are conducted through a temporary container, and a bidirectional index table is established to redirect an access request.

The invention discloses a heterogeneous time series data processing method and system, electronic equipment and a storage medium, and belongs to the technical field of industrial data processing. The method comprises the following steps: acquiring original working condition data of each device in a target area in real time, and converting the original working condition data into intermediate data in a unified format; performing standardization processing on the intermediate data in sequence, outputting standardized data, and performing quality verification on the standardized data; performing data fusion processing on the standardized data according to the quality verification result, and outputting and storing fused data; constructing a hierarchical bitmap index structure of fused data according to the basic index key; and obtaining and analyzing a query request, inputting an analysis result into the hierarchical bitmap index structure, performing accelerated query operation according to an index result, and outputting target data. According to the method, continuous and efficient data fusion and query processing in a large data volume state can be realized, and the expansibility is high.

The invention discloses a scientific and technological achievement supply and demand matching method and system based on index fingerprinting and dynamic bitmap indexing, and belongs to the technical field of information retrieval and text data processing. In order to solve the problems of low matching efficiency, insufficient accuracy and insufficiency in interpretability and the like caused by fuzzy expression of enterprise demands and fragmentation of achievement index description, the specific scheme of the invention comprises the following steps of: firstly, performing normalization processing on index terms of achievements and demand texts, generating binary index fingerprints and constructing an achievement dynamic bitmap index; secondly, analyzing the demand to generate a demand fingerprint and quantitative constraint information; performing dynamic gating pruning based on the coverage degree through multi-strategy mixed recall preliminary screening; and finally, fusing the semantic similarity and the index satisfaction degree to carry out score sorting, and outputting an interpretable matching result. According to the method, rapid preliminary screening is achieved through fingerprinting bitmap indexing, the docking efficiency and the result reliability are improved in combination with multi-strategy fusion, and the defects of an existing method in the aspects of matching efficiency, accuracy and interpretability are effectively overcome.

The application discloses a kind of methods and systems for fault root cause positioning of server cluster, it is related to network fault diagnosis technical field;The present application realizes microservice call chain, container index, nanosecond level synchronous collection of physical node and network data by precise time protocol, constructs consistent data set after entity association and standardization processing;Based on this, service-resource topology graph is dynamically constructed, and the inter-service call edge weight model is innovatively designed: the real-time load factor is fused, and the historical fault index attenuation and processed by Sigmoid function are superposed, and the weight is periodically updated to accurately quantify the influence strength between nodes;Topology graph is converted into Bayesian network;When fault occurs, three-stage pipeline is used to compress alarm, frequent item set is mined by bitmap index and parallel FP-Growth, and strong causal association item set is screened by combining topology edge weight and KL divergence;Strong causal alarm is taken as evidence, and probabilistic root cause ranking is output by reverse random walk sampling, to realize high-precision positioning of complex distributed system fault.

Embodiments of the present application disclose a data query method, an electronic device and a computer readable storage medium. The electronic device stores a first data table including a first tuple and a first bitmap index corresponding to the first tuple. The first tuple includes one or more queryable attribute values. The first bitmap index includes a plurality of bits arranged in sequence. The method includes receiving a query request for the first data table. The query request includes a query item including an attribute value to be queried. A second bitmap index corresponding to the query item is generated based on the attribute value to be queried. The second bitmap index corresponding to the query item and values of the plurality of bits at the same positions in the first bitmap index corresponding to the first tuple are compared simultaneously to determine whether the first tuple satisfies the query item. Embodiments of the present application can improve the efficiency of data query.

The application provides an intelligent selection method and device for industrial instruments based on RAG and expert system, and belongs to the technical field of cross between artificial intelligence and industrial automation. L instrument parameters with K selectable items for each parameter are divided into a core parameter set with M (M<=6) parameters and an attribute parameter set with N (N=L-M) parameters, the core parameter set defines a basic model, and dynamic binding of the basic model stored in a table structure and attribute parameters stored in a dictionary structure is realized; a user inquires about a parameter question in a natural language, RAG generates accurate parameter suggestions based on expert knowledge, an expert system verifies parameter accuracy and cross-parameter compatibility in real time, and provides correction suggestions when conflicts occur. The selection process is two-stage matching: a bitmap index locks the basic model corresponding to the core parameter, and a hash lookup verifies the attribute parameter. The instrument selection method provided by the application obtains not only the instrument itself, but also a solution meeting application requirements.

The application relates to the field of data analysis, and provides a data sampling method based on data modeling, a device and a storage medium.The method comprises the following steps: obtaining a data sampling requirement, the data sampling requirement comprising data attributes and a sampling quantity; comparing the data attributes with preset statistical attributes to obtain target attributes and non-statistical attributes; extracting target data corresponding to the sampling quantity from a preset bitmap index according to the target attributes; randomly generating virtual data of the sampling quantity according to the non-statistical attributes; and combining the target data and the virtual data to obtain sampling data, each piece of the sampling data comprising target data corresponding to the target attributes and virtual data corresponding to the non-statistical attributes. The target attribute data meeting the data sampling requirement is quickly generated in the form of the bitmap index, and the data of the non-statistical attributes is generated in a random manner, so that the data requirement and the quantity requirement of the data sampling are met, and the sampling efficiency is effectively improved, and the reliability of the sampling data is ensured.

According to the economic information automatic statistical analysis method and system, a unified data foundation is laid through multi-source data integration and structured processing, and the invalid I / O operation in the query process is greatly reduced in combination with the double-layer index design of physical block storage, dictionary coding and block-level metadata index and in-block bitmap index; the data positioning and statistical calculation efficiency is obviously improved; meanwhile, on the basis of an asynchronous running dynamic recombination optimization mechanism, self-adaptive adjustment of data storage layout on query requirements is achieved by collecting historical query loads, mining frequent query modes, simulating and verifying candidate strategies and dynamically updating physical partitioning strategies, and the problem that a traditional static strategy is difficult to adapt to the pain point of the dynamic query loads is effectively solved; according to the method, high performance, intelligence and self-evolution of economic information statistical analysis are realized, the result readability and the use convenience are improved, and the method is adaptive to precise analysis requirements of large-scale and multi-source heterogeneous economic data.

The invention provides an industrial instrument intelligent type selection method and device based on cooperation of an RAG and an expert system, and belongs to the technical field of crossing of artificial intelligence and industrial automation. According to the method, L instrument parameters with the option number of K of each parameter are divided into a core parameter set of M < = 6 parameters and an attribute parameter set of N = L-M parameters, the core parameter set defines a basic model, and dynamic binding of the basic model stored in a table structure and the attribute parameters stored in a dictionary structure is achieved; a user inquires a parameter problem in a natural language, the RAG generates an accurate parameter suggestion based on expert knowledge, and an expert system verifies parameter accuracy and cross-parameter compatibility in real time and provides a correction suggestion when conflicts occur. The model selection process is two-stage matching, wherein a basic model corresponding to a core parameter is locked through bitmap index, and a verification attribute parameter is searched through hash. According to the instrument type selection method provided by the invention, the obtained result is not only the instrument, but also a solution for meeting the application requirements of the instrument.

The invention relates to the technical field of data processing, in particular to a mass log data archiving method for a cross-border e-commerce platform, which comprises the following steps: receiving cross-border e-commerce transaction log streams, analyzing and extracting user session IDs and commodity SKU codes in logs, counting the frequency of sharing the user session IDs or the commodity SKU codes among different log entries in a buffer area, and storing the user session IDs or the commodity SKU codes in the buffer area. And establishing a log attribute association frequency table, and calculating log clustering coefficient indexes of different log entries on the clustering dimension according to the log attribute association frequency table. According to the method, the compressed sequence is generated through line number difference calculation, difference distribution density is counted to serve as a storage format decision basis, an array container format is selected to reduce storage space occupation when the distribution density is lower than a threshold value, and a bitmap container format is automatically switched to improve bit operation efficiency when the distribution density is higher than the threshold value. And the established adaptive sparse bitmap index can dynamically balance the storage cost and the retrieval speed in different data sparseness scenes.

The application discloses a cross-grain RAID strip local dynamic reconstruction method and device based on a logical-physical mapping bitmap index, comprising the following steps: first, a logical-physical mapping bitmap index for marking the validity of each sector data in a RAID strip is maintained in a flash translation layer (FTL) of a solid state disk; when a target grain failure is detected, the bitmap index is called by the system for analysis; then, according to the marking of the bitmap index, the effective data area on other healthy grains in the corresponding strip is accurately scheduled to perform local reading and XOR calculation, and all invalid data areas are actively skipped to reconstruct the effective data in the failed grain; finally, the recovered discrete effective data is dynamically aggregated into a new complete strip and then written into a standby physical space. Through the application, internal read amplification and write amplification in the reconstruction process can be significantly reduced, the failure recovery time is shortened, and the performance stability of the SSD in the degradation mode is ensured.

This invention belongs to the field of memory, specifically relating to a wear-aware space allocation method for persistent memory file systems based on multi-level bitmaps. The method includes constructing a fine-grained multi-granularity allocation heap based on the number of consecutive pages contained in the free block min-heap; constructing an index structure based on multi-level bitmaps based on the availability of the min-heap; and quickly locating the first available min-heap with a granularity greater than or equal to the requested granularity using the multi-level bitmap index structure, based on the selected granularity requested by the upper-layer application, and allocating the free block with the lowest average wear from this min-heap. This invention solves the problem of low wear leveling accuracy in existing persistent memory file systems, further improving wear accuracy; it also solves the problem of high performance overhead caused by large block partitioning and sub-block average wear, significantly improving the performance of persistent memory file systems.

The invention provides an electric power real-time library compressed bitmap index creating method, which belongs to the technical field of electric power real-time libraries, and comprises the following steps: reading database parameter information of a commercial database by utilizing a database editing tool, and initializing all configuration table information of an electric power real-time library; editing real-time library metadata of the power real-time library by using a database editing tool, and generating a power grid data model; according to remote signaling and telemetering data of the power monitoring system, editing a remote signaling and telemetering table by using a database editing tool, and determining an index column of a power real-time library; reading real-time library metadata by using a real-time library process, creating a data dictionary, and allocating a real-time library memory according to the data dictionary; according to the remote signaling and telemetering table and the index column, a compressed bitmap index is created, and the electric power real-time library containing the compressed bitmap index is obtained. According to the method, the retrieval efficiency of the real-time library of the power system can be improved under the condition that the memory space occupied by the index is not increased.

Techniques for document sorting with bitmap indexes are described. A request to sort a set of documents is received. A bitmap that identifies the set of documents to be sorted is obtained. An ordered set of sort bitmaps is obtained, each sort bitmap in the ordered set of sort bitmaps corresponding to a bit in an encoding of a data type to be sorted. A series of passes of bitwise operations is performed with the ordered set of sort bitmaps, each pass to identify a next document identifier to add to a sorted set of document identifiers by processing an initial bitmap with bitwise operations in order against at least a portion of the ordered set of sort bitmaps. The initial bitmap of the first pass in the series of passes is initialized with the obtained bitmap. The sorted set of document identifiers is stored.

The application relates to a data query method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: obtaining a data query request for a multidimensional data space; the data query request carries a target dimension combination; obtaining a space index corresponding to the multidimensional data space; the multidimensional data space comprises a plurality of data subspaces, the space index comprises index bitmaps corresponding to dimension members included in each dimension, each position in the index bitmap indicates a data subspace, and the element value of the same position in each index bitmap is used to indicate a subspace dimension combination corresponding to the corresponding data subspace; performing logical operation on the index bitmaps corresponding to each target member in the space index to obtain a result bitmap corresponding to the target dimension combination; and determining a query result corresponding to the data query request based on the result bitmap. The method can improve the efficiency of data query.

The invention discloses a vehicle consumable data matching method and device, and relates to the technical field of big data and automobile supply chains. A specific embodiment of the method comprises the following steps: extracting a matching field from to-be-matched vehicle model data, and carrying out numerical coding to obtain a field attribute code; determining coding types of the field attribute codes according to the number of the field attributes and the number of different numerical values in the field attribute codes, wherein the coding types comprise an enumeration type, an ordered enumeration type and a combined numerical value type; according to the field attribute code and the code type, constructing a multi-dimensional index to perform vehicle consumable data matching based on the multi-dimensional index, and when the code type is an enumeration type, constructing a bitmap index; when the coding type is an ordered enumeration type, constructing a tree structure index; and when the coding type is a combined numeric type, constructing a multi-layer tree structure index. According to the embodiment, the applicability of a data matching system is improved, the increase of system complexity and maintenance cost is avoided, and the data matching efficiency and the accuracy of a matching result are improved.

The present application belongs to the field of information retrieval and big data analysis, and particularly relates to a bitmap index compression method for genomic variation data, comprising: splitting original genomic variation data sequences according to field attributes, and storing in a columnar database; establishing original bitmap indexes for genomic variation data in the columnar database; dividing the original bitmap indexes into several segments; using a run-length algorithm to compress and store the public part of the segmented bitmap data, and using an integer list or a bit sequence to compress and store the non-public part. The present application saves more storage space by merging public affixes in the bitmap sequence, improves the bit operation speed, and makes the bitmap index load into the memory faster.

The invention discloses a commodity recommendation system and method based on dynamic index combination, and the system comprises a data storage layer, a configuration management layer, a data processing and calculation layer, and a recommendation service layer, and the data storage layer is used for building an MPP database, and storing commodity dimension data, sales index data, human-commodity algorithm indexes, and bitmap indexes; the configuration management layer is used for providing a user interface for configuration management and defining a recommendation task; the data processing and calculating layer is used for processing and calculating data and generating a commodity list; and the recommendation service layer is used for receiving the recommendation request, calling the configuration management platform to obtain corresponding recommendation configuration, driving the data processing and calculation layer to generate a recommendation table, and returning a commodity list. Through the dynamic template engine and the configuration platform, highly customized recommendation can be realized according to service requirements.

The present disclosure relates to a method, device, computer device and storage medium for storing data by using bitmap index. The method comprises: obtaining data identification information to be stored and data state information corresponding to the data identification information, wherein the data state information comprises system service information described by a Boolean variable; determining an integer variable to be stored corresponding to the data state information, wherein the integer variable to be stored is a positive number; and creating a bitmap index table according to the integer variable to be stored and the data identification information, wherein the data state information is described by the integer variable to be stored in the bitmap index table, and the bitmap index table is used to store the integer variable to be stored and the data identification information. By using the method, the storage of related information can be performed without occupying a large amount of storage space, and the consumption of storage resources can be reduced.

A method of writing data in a replay protected memory block (RPMB) area of a storage device in response to a request of a host device includes receiving a write request, including a message authentication code, data, and a bitmap index, from the host device and verifying the write request based on the message authentication code and the bitmap index. The verifying the write request may include calculating a message authentication code based on data and a bitmap index received from the host device, comparing a message authentication code, calculated in the storage device, with the message authentication code of the write request, and comparing the bitmap index of the write request with bitmap indexes, stored in the storage device, to check whether a replay attack has been made.

The present application relates to big data governance technical field, specifically to a kind of data traceability method based on big data mining, comprising: collecting unstructured operation log and parsing into structured event record, associated generation heterogeneous operation session set;Access mode association analysis is carried out to session set, and read-write path mode is extracted, and dependence weight is determined in combination with association strength and time decay factor and constructs blood relationship atlas;Loop is eliminated to atlas, and directed acyclic graph is generated, and generating tree and node index coding interval are constructed as retrieval backbone, and cross branch association edge is mapped to bitmap index to generate probability skeleton tree index;Interval positioning main path is used, and bitmap is called to carry out multi-path verification, and traceability path and confidence are output. By mining log, probability blood relationship atlas and probability skeleton tree index are constructed, data blood relationship reconstruction and large-scale node low-delay retrieval in heterogeneous black box environment are realized.

The invention provides a capacity-hidden joint query dynamic searchable encryption method, and belongs to the technical field of searchable encryption. According to the method, a new index structure is constructed based on an extensible bitmap, a homomorphic addition symmetric encryption algorithm is used for transforming the bitmap of each keyword into an'encryption and homomorphic accumulation 'index structure, and only one corresponding encrypted bitmap index is maintained for each keyword; a new index structure is a fixed-length encrypted bitmap, a server can only observe the length of the bitmap and the query round, and the number of real hit documents and the intersection scale are difficult to recover, so that the query result capacity hiding is realized while the joint query is supported; in the updating process, the client encrypts the increment bitmap through a one-time secret key, the server only executes modular addition operation on an existing ciphertext so that addition and deletion can be completed in a ciphertext domain, all rounds of secret keys are independent, the search state evolves along with counting, and therefore associated leakage of updating and querying is weakened.

The application relates to the field of data storage, and discloses a big data compression storage and fast retrieval method and system, which comprises the following steps: dynamically generating a dimension weight vector based on multi-dimensional information entropy, and generating a rotating Hilbert code by adjusting the direction of a space-filling curve; adaptively determining the number of segments and an error threshold by using a time series prediction model, and compressively storing coding residual errors by using a linear model; constructing a double-layer collaborative index structure comprising a space bitmap index and a value range B+ tree index, and realizing local index reconstruction when the weight changes through an incremental updating mechanism; mapping a query range to a coding interval, screening candidate data segments in combination with the index, and returning a result by fusing model parameters and residual error decoding; asynchronously processing new data coding correction and obsolete data elimination, and optimizing storage space consistency. The application realizes the unification of high compression rate, fast retrieval and adaptive update.

The application discloses a multi-modal data verifiable query method and system based on a blockchain, and the method comprises the following steps: acquiring multi-modal data, storing the multi-modal data in an off-chain database; extracting metadata from the multi-modal data to obtain a plurality of metadata; generating a unique identifier for each metadata; packing each metadata and the unique identifier corresponding to each metadata, uploading the packed data to the blockchain, and storing the data in a block of the blockchain; creating a double-layer index for the blockchain based on the metadata; acquiring data to be queried, determining a plurality of candidate block subsets according to the data to be queried and a multi-layer subset bitmap index; taking the intersection of all candidate block subsets to obtain a set of blocks to be scanned; filtering each block in the set of blocks to be scanned by using a cuckoo filter to obtain a target block; searching the target block by using a Merkle Patricia tree to obtain target metadata; and performing on-chain and off-chain collaborative verification on the target metadata.

The invention provides a data real-time compression and query method based on a prefix dictionary and a bitmap index, and relates to the field of data compression and query, and the method comprises the steps: obtaining a state list of an expressway within a preset time, and decompressing and storing the state list; updating the blacklist through the decompressed state list; the updated blacklist is read into a memory, compression is carried out in a prefix and bitmap mode, and a db file is generated from compressed data; whether state list data corresponding to the blacklist exists or not is inquired through the compressed db file, and real-time quick retrieval of the blacklist is completed. According to the technical scheme, quick retrieval of the highway vehicle blacklist is achieved, and the real-time response requirement of lane charging is met.

The application provides a bitmap index container optimization method and system based on multi-dimensional dynamic decision, relates to the technical field of databases, and comprises the following steps: a skewness coefficient value is calculated, a feature fusion vector is constructed by fusing a data distribution feature vector, a data access rule vector and a change trend vector, a double strategy gradient algorithm is used for iterative optimization to obtain an adaptive threshold value, it is judged whether to switch the container type, data deduplication, priority sorting are performed through a temporary container, batch migration data is performed, and a bidirectional index table is established to redirect an access request.

A system and method for executing a query in a data analytics storage engine is provided. The method includes receiving a query to locate target data in a data analytics storage engine, the data analytics storage engine including data rows divided into one or more partitions of data, the one or more partitions having data columns corresponding to the data rows, and bitmap data embedded in the one or more partitions, wherein the bitmap data is associated with the data columns; and locating the target data using the bitmap data in the one or more partitions.