42 results about "Concept drifting" patented technology

The invention discloses a data stream anomaly detection system based on empirical characteristics and convolution neural network. The system includes an empirical feature extraction module, which is used to identify statistical features and header features as features based on artificial experience, which play a more important role in data packet anomaly recognition; a bit stream conversion picture module used to convert the data stream into the form of two-dimensional gray-scale picture, and then through the convolutional neural network perception, the global high-level perception features are extracted; a fusion splicing module used for fusing the above modules as the data stream characteristics and identifying abnormal data streams by using the full connection layer of the neural network; a distillation model module that replaces complex networks in actual deployment; a concept drift fine-tuning module updates the detection model of concept drift; an update experience database module adding new network attacks or hidden attack instructions to the artificial experience database. The invention accurately and efficiently detects abnormal behaviors such as network failure, user misoperation, network attack and the like.

The invention relates to the technical field of intelligent information processing and discloses a method for classifying data streams in a dynamic data environment. The method comprises the following steps: partitioning the data streams; establishing different classifiers for different concept drift; storing in a characteristic data pool of the classifiers; when a new data block arrives, judging whether the concept drift occurs or not by Kullback-Leibler (KL) divergence; if the concept drift does not occur, classifying by using the classifier at the last moment; if the concept drift occurs, seeking the proper classifiers from the characteristic data pool of the classifiers by the KL divergence and classifying; and if no coincident classifier exists, training a new classifier, adding the new classifier into the characteristic data pool of the classifiers and deleting the outdated classifiers. By the method, stable and mutational concept drift can be detected simultaneously; when the concept drift occurs, classification is performed by selecting the proper classifier to guarantee the efficiency of a model; and the performance of the model is guaranteed by deleting the outdated classifiers.

The invention discloses a short-text data stream classification method based on topic models and concept drift detection. The method includes: 1, acquiring an external corpus from a knowledge libraryto construct the LDA topic model; 2, dividing a short-text data stream into data blocks according to a sliding window mechanism, and using the LDA topic model to expand short text in the data blocks to obtain an expanded data stream; 3, constructing the online BTM topic model for each data block in the expanded short-text data stream, and obtaining a topic representation of each piece of short text; 4, selecting data blocks of Q topic representations to construct a classifier to use the same to predict a class label of a newly arrived data block; 5, dividing the data blocks of the Q topic representations into category clusters according to class label distribution, and calculating semantic distances between the category clusters and the newly arrived data block to judge whether concept drift occurs; and 6, updating the classifier according to a concept drift situation. The method can be used for the short-text data stream classification problem of unceasingly changed class label distribution.

The invention relates to the technical field of intelligent information processing, and discloses a visualization method for concept drift of a data stream in a dynamic data environment. The visualization method comprises the following steps: achieving static treatment on the data stream; establishing different concept presentation modes according to different concept drift modes and saving the different concept presentation modes in concept pools; and when a new data block comes, utilizing the KL divergence algorithm to search similar concept presentations in the concept pools, if the similar concept presentations exist, counting the similar concept presentations, and if not exist, adding the new data block into the concept pools to serve as a new concept to be saved. The visualization method can be used for detecting the changed drift types of various data streams, can fully analyze the concept drift process in the data stream through counting, finally, utilizes the Bays method to draw a concept drift and transition graph according to the statistic result, and visualizes the concept drift and transition graph for assisting data mining in a concept level.

The present disclosure provides methods and systems for coping with outliers and perfect learning in a concept-drifting environment. Data in the concept-drifting environment are time series data, and embodiments of the present disclosure uses moving windows to solve the problem of concept-drifting.

The invention discloses an adaptive network flow concept drift detection method based on information entropy, and relates to the technical field of data processing. The concept drift problem caused byflow characteristic changes due to time lapse and different network environments is solved. By using sliding window techniques, aiming at new and old data streams, discretizing the flow characteristic attributes into a plurality of branches; comparing the plurality of flow characteristics together; comparing differences of new and old data stream sliding windows by counting information entropiesof each characteristic attribute and each branch; and when specific detection is carried out, the method comprises the following steps: firstly, solving a threshold calculation formula of the size ofa sliding window by utilizing a Hoeffding boundary theory; then, determining the size of a sliding window by utilizing a Tie-buffering method; according to the method, firstly, obtaining a new data flow sliding window and an old data flow sliding window, then calculating entropy values of the new data flow sliding window and the old data flow sliding window through information entropy, finally, jusging whether concept drifting happens to data or not according to threshold values and information entropy difference values of the new window and the old window, thus concept drifting can be effectively detected, classifiers are updated, and good classification performance and generalization capacity are shown.

The invention provides a method and a system for acquiring concept drift amount of data distribution. The method comprises the steps of obtaining all data including training data and test data; clustering all the data by using a Gaussian mixture model to obtain a plurality of clustering clusters of all the data; respectively obtaining concept drift amounts of data distribution of all data contained in each cluster of the plurality of clusters; and obtaining the concept drift amount of the data distribution of all the data by utilizing the concept drift amount of the data distribution of all the data contained in each cluster of the plurality of clusters. Therefore, the concept drift amount of the data distribution can be accurately obtained, the change condition of the data distribution can be accurately judged, and the reliability of the system is greatly improved.

An embodiment of the invention discloses a financial fraud identification method which is used for solving a problem that an existing FDS reduces a Concept drift influence and simultaneously is difficult to satisfy an instant detecting requirement. The method comprises the following steps of acquiring current transaction behavior data of a target user; and inputting the current transaction behavior data into a ground floor classifier and acquiring a determination result output by the ground floor classifier. The embodiment of the invention also provides a financial fraud identification system.

The invention discloses an online short text data stream classification method based on feature extension, which comprises the following steps of: 1, constructing a Word2vec model according to an external corpus, and obtaining a word vector set Vec; 2, vectorizing the short text data stream by using Vec, and performing text vectorization extension based on a CNN model; 3, constructing an online deep learning network for the extended text vectors; 4, introducing concept drift semaphore into neurons in the LSTM network and detecting the distribution change of the short text stream; and 5, completing the model updating of the online deep learning network and the prediction of the short text data stream. According to the method, the classification accuracy of the short text data streams can beeffectively improved, concept drift is correctly detected, the model is adjusted, and therefore the purpose of rapidly adapting to the short text data stream environment is achieved.

The invention relates to a network flow analysis method and device based on multi-source network flow data. The method comprises the following steps: deploying a pre-training classifier pool at network flow data sources, for each data source, receiving multi-source network flow data at the current moment, carrying out flow classification on the multi-source network flow data through an online classifier, carrying out feature processing and transformation on the data collected by each network flow data source every preset time interval, and carrying out feature extraction on the data collected by each network flow data source; transmitting the processed flow data features and the feature transformation matrix to a flow drift detection module; and the flow drift detection module comprises historical concept data so as to detect concept drift according to the flow data features, the feature transformation matrix and the historical concept data, and if the concept drift is detected, a multi-source deployed online classifier is reset. By adopting the method, the multi-source network flow data can be continuously and accurately analyzed in real time.

The invention discloses a concept drift detection method based on classifier diversity and a Mcdiarmid inequality. The objective of the invention is to detect whether conceptual drift occurs in a datastream by combining the inconsistency of a plurality of classifiers and the Mcdiarmid inequality. The method comprises the following steps: 1, incrementally training two individual classifiers with relatively large divergence, monitoring the diversity of the pair of classifiers for a newly coming data stream, and calculating the difference measurement of prediction results between the classifiers; 2, setting the size of the sliding window h to be n, and if the sliding window h is not full, automatically adding the difference measurement result of the latest data stream into the sliding windowh; and if the sliding window h is full, moving the initial difference measurement result out of the sliding window, and adding the latest result; 3, giving a confidence coefficient, and solving a threshold value for judging drifting through the confidence coefficient and an Mcdiarmid inequality theory. 4, associating each element in the sliding window with a weight, calculating the difference value between the weighted average value of the sliding window at the current moment and the maximum weighted average value observed at present, and comparing the difference value with the previously obtained threshold value to judge whether drifting occurs or not, so that concept drifting can be effectively detected, a classifier is updated, and better classification performance and generalization ability are shown.

The invention relates to an abnormal data flow online calibration system heurized by a brain-like hierarchical memory mechanism. The system comprises a missing data filling module (I), a dimension reduction module (II), a multi-dimensional counting Bloom filter module (III), a hierarchical memory library module (IV), an experience knowledge base module (V) and a calibrated data block module (VI).The I is used for filling up missing data in data blocks in a data stream batch processing link. The II is used for performing low-dimensional representation on the high-dimensional data in the data block. The III is used for judging whether the new data sample is abnormal . The IV is used for storing the historical data sample processed in the step II. The V is used for storing historical data samples which are processed by I but not processed by II. The III and theIV are matched with each other to be replaced, and the IV and the V are matched with each other to be updated; and the VI is usedstoring the replaced and updated new data sample. According to the online calibration system, under the condition that original data distribution is not changed, outliers, noise, missing values and the self-adaptive concept drift phenomenon are corrected in real time.

According to the data stream classification method and device, the concept drift state is accurately recognized according to the classification error rate of the integrated classifier at the current moment and the concept drift detection threshold value, the data window at the next moment is flexibly set according to the concept drift state, and if the data window is rapidly increased when concept drift does not occur, the data window can be rapidly increased. The data stream classification efficiency is improved; when it is not determined whether concept drift occurs or not, the increasing amplitude of a data window is reduced, whether concept drift occurs or not is further determined, and the influence of noise on classification accuracy is reduced; when concept drift occurs, a data window is quickly reduced, the sensitivity to sudden change type concept drift is improved, and the integrated classifier is quickly adjusted, so that the accuracy of data stream classification by the integrated classifier is integrally improved.

The invention discloses a data flow concept drift detection method based on historical model diversity, which comprises the following steps of: extracting data flow on line by adopting an on-line bagging method, constructing a basic tree in a random forest based on a diversity standard, and detecting the change of a data flow characteristic space, pre-warning possible concept drifting, identifyinga concept drifting area, identifying and processing noise by integrating drifting pre-warning and the drifting area, and finally detecting the concept drifting by an integration method of a random forest. According to the method, the concept drift detection problem is solved by using a random forest, and various new historical model storage strategies are provided, so that the problems of instance storage and how to promote future concept drift detection by using the models are solved.

The invention discloses a credit card fraud detection method based on unbalanced flow data classification. The credit card stream data classification model comprises a bagged credit card stream data oversampling mechanism, a credit card stream data multi-type concept drift processing mechanism, a credit card stream data basic classifier weighting mechanism, a credit card stream data integration pruning mechanism and a classification prediction mechanism. According to the method, credit card flow data small samples at any past moment do not need to be reserved, and credit card dynamic flow datawith unbalanced category distribution can be effectively classified and predicted so as to detect credit card fraudulent transaction data.

The invention discloses an unbalanced concept drift data stream classification method based on G-mean dynamic weighting, and belongs to the field of data stream classification. According to the method, through an online weighting mechanism, the weights of all base classifiers are updated once and are not influenced by class distribution when a new instance rather than a complete data block arrives, the creation time of the classifiers and the Gmean performance of the classifiers on the latest p data are considered when the base classifiers are updated, and therefore the classification accuracy of the base classifiers is improved. The G-mean performance is insensitive to the class distribution of the data, the importance of multiple classes and few classes can be balanced, the classification accuracy of the minority classes is improved, meanwhile, the TPR and the TNP are incrementally calculated through the time attenuation factor, and any historical data does not need to be stored; meanwhile, two elimination mechanisms are used for controlling the scale of the integrated model, and the high efficiency and accuracy of decision making are kept.

The invention provides a model adaptive training method and device, equipment, a medium and a program product, and the method comprises the steps: obtaining all operation data of an original model during actual operation, and detecting a first concept drift value of the original model during actual operation according to the operation data; and then, according to the value model and the first concept drift value, distributing each operation data to a tagged data set and/or a non-tagged data set, and when the data size of the tagged data set is greater than or equal to a preset threshold value, utilizing an adaptive training model, and according to the tagged data set and the non-tagged data set, performing adaptive training on the original model to determine a new trained model. The technical problem of how to enable the AI model to carry out adaptive training under the condition that human intervention is as little as possible is solved. The technical effects that the manual annotation amount needed when the developers update the models is reduced, and the performance stability of the models is improved in the mode of integrating the multiple models are achieved.

The invention discloses a data enhancement method, system and device and a computer readable storage medium. The method comprises the steps: determining whether concept drift occurs in a current data enhancement method or not; if the current data enhancement method generates concept drift, generating a target data enhancement method; judging whether the target data enhancement method meets a data enhancement performance index or not; and if the target data enhancement method meets the data enhancement performance index, performing data enhancement on the online data based on the target data enhancement method to obtain target data. According to the data enhancement method, the target data enhancement method can be generated after the current data enhancement method generates concept drift, and the target data enhancement method is applied to perform data enhancement on the online data after the target data enhancement method meets the data enhancement performance index, so that the data enhancement method for updating the online data in real time is realized; therefore, the concept drift phenomenon can be avoided, and the accuracy of data enhancement is ensured.

The invention relates to a dynamic production environment anomaly monitoring system oriented to real-time data streams. The dynamic production environment anomaly monitoring system comprises a real-time data collection module (I), a single-node sparse discrete encoder generation module (II), a single-node real-time anomaly monitoring module (III), a full-process all-node parallel real-time monitoring module (IV) and a production mode updating module (V), wherein the I is used for collecting numerical values of all nodes and integrating the numerical values into a multi-dimensional sample; theII is used for generating a plurality of sparse discrete encoders, wherein each sparse discrete encoder is used for performing sparse discrete encoding on the numerical value of each node; the III isused for judging whether the value is abnormal; the IV is used for monitoring all nodes in parallel in real time, and identifying noise interference and concept drift of each node according to the decision matrix; and the V is used for changing the production mode when it is determined that concept drift occurs. The dynamic production environment abnormity monitoring system oriented to the real-time data flow can monitor system abnormity in real time and distinguish whether noise interference or production mode change exists from the system abnormity.