39 results about "Web data mining" patented technology

An enterprise-wide web data mining system, computer program product, and method of operation thereof, that uses Internet based data sources, and which operates in an automated and cost effective manner. The enterprise web mining system comprises: a database coupled to a plurality of data sources, the database operable to store data collected from the data sources; a data mining engine coupled to the web server and the database, the data mining engine operable to generate a plurality of data mining models using the collected data; a server coupled to a network, the server operable to: receive a request for a prediction or recommendation over the network, generate a prediction or recommendation using the data mining models, and transmit the generated prediction or recommendation.

A method, system, and computer program product provides automated determination of the size of the sample that is to be used in training a neural network data mining model that is large enough to properly train the neural network data mining model, yet is no larger than is necessary. A method of performing training of a neural network data mining model comprises the steps of: a) providing a training dataset for training an untrained neural network data mining model, the first training dataset comprising a plurality of rows of data, b) selecting a row of data from the training dataset for performing training processing on the neural network data mining model, c) computing an estimate of a gradient or cost function of the neural network data mining model, d) determining whether the gradient or cost function of the neural network data mining model has converged, based on the computed estimate of the gradient or cost function of the neural network data mining model, e) repeating steps b)-d), if the gradient or cost function of the neural network data mining model has not converged, and f) updating weights of the neural network data mining model, if the gradient or cost function of the neural network data mining model has converged.

The invention relates to a network video classification method based on historical access records and belongs to the technical field of computer network data mining. The method comprises, firstly, automatically analyzing historical access record datasets of videos, extracting meaningful characteristics, generating standby data files for the historical access record datasets of the videos, converting historical access records into structurized documents applicable to training through the data files and then performing machine learning on the structurized documents through logistic regression to obtain prediction models; utilizing the prediction models, according to the integrity of the historical access record information of videos to be predicted, to select corresponding methods to perform classification prediction on the videos to be predicted. Compared with the prior art, the network video classification method based on the historical access records can reduce labor costs and simplify parameters involved in computation and is more accurate in prediction effects and lower in time consumption. Meanwhile, due to the fact of being capable of being clustered or not according to the integrity of the historical access record information of the videos to be predicted, the models has a wide application range.

The invention discloses a method for identifying and positioning an unknown mobile station in a special position area, which performs identification and mobile positioning on the mobile station in the special position area. The method is implemented by identifying the mobile station in the special position area through data mining and combining the identification result and a mobile positioning network. The method overcomes the limitation that the conventional mobile network positioning method serving as a one-dimensional active positioning method must determine a mobile station number IMSI and solves the problem that a network data mining mode positions an idle mobile station.

The present invention discloses a network data mining method based on a vertical search. Firstly, the method of the vertical search is adopted to search data from a network; the information gained is preprocessed, the structured data after purging through the data is saved in a data base; an analysis is performed to the data in the data base to find a rule, therefore to construct a model, and a matching is performed to the feature vector of a collection and the feature vector of a target sample, herefrom, the degree of association of the relevant collection information is gained; a prediction is performed to an unknown data, and an evaluation is delivered by being compared with a actual result, therefore to perform a revision to an original model parameter, and the authoritative information is supplied to a user. The present invention adopts a network data mining method of a vertical search to gain the relevant information, the relevant specialized information can be effectively gained, the repeated information and the spam information are little, and the enquiry of a user specialization can be met.

The invention belongs to the field of network data mining field, and specifically relates to a community detection method based on a dynamic synchronous model. The method comprises the steps of firstly reading social network data, and performing network vectorization according to a social network graph to obtain a vectorized one-dimensional coordinate sequence; setting synchronization parameters and calculating a synchronization range; performing synchronization clustering, wherein each node is synchronized in the synchronization range according to the extensional synchronous model until a local synchronization status is available; dividing communities according to the coordinate position of each node; calculating the modularity of the division; adding the synchronization parameters constantly; executing a new round of synchronization clustering process until the synchronization range covers all the nodes. Nodes in the network are clustered through a kuramoto model, so that a link density can be accurately described, the difference of the network link density is effectively reflected, the automatic detection of a social network community structure is realized, and the community detection results are selected and optimized.

The invention belongs to the technical field of network data mining and particularly relates to a detection method for an overlapping community based on multi-label propagation. The detection method comprises the following steps: A, constructing a social network diagram; B, analyzing a network rough core; C, initializing a label set; D, executing label propagation; E, decomposing a discontinuous community. According to the detection method disclosed by the invention, the link density between every two nodes is fully considered; the detection method has higher accuracy and effectiveness; in addition, manual input of data in the label propagation process is avoided.

The invention discloses a post-loan risk early warning system based on semantic sentiment analysis. The post-loan risk early warning system is characterized by comprising a network data mining module, a semantic sentiment analysis module, a total analysis module and a user interaction module. The network data mining module is used for collecting relevant information of customer enterprises from the network, wherein the relevant information comprises one or more of news, reviews, Microblogs, reports and complaints relevant to the client enterprises. The semantic sentiment analysis module is used for receiving the relevant information, analyzing the sentiment components of the relevant information and generating sentiment polarity K and sentiment intensity M. The total analysis module is used for obtaining the sentiment polarity K and the sentiment intensity M, generating the value of the sentiment polarity K and the value of the sentiment intensity M according to the source of the relevant information, and then obtaining a reliable coefficient P and an overall reliable coefficient W through calculation in sequence according to a predetermined formula. The user interaction module is used for giving a warning when the overall reliable coefficient W is smaller than a warning value. The post-loan risk early warning system based on semantic sentiment analysis can give an early warning for great changes of the client enterprises in time, help a bank to manage the client enterprises better, and effectively reduce post-loan risks.

The invention discloses a Web data mining system on the basis of a Hadoop platform and relates to a data mining system. The system comprises a user interaction layer, a service application layer, a Web data mining platform layer and a distributed storage calculation layer; the user interaction layer is used for interaction between a user and the system and comprises a user management module, a service module and a display module; the service application layer comprises a service response module and a workflow module; the Web data mining platform layer comprises a data loading module, a result storage module, a mode evaluation module, a parallel ETL (Extract Transform and Load) module and a parallel data mining algorithm module; and the distributed storage calculation layer uses Hadoop to implement file distributed storage and parallel calculation functions and comprises an HDFS (Hadoop Distributed File System) module, a MapReduce module and a distributed management module. According to the invention, the calculation and storage requirements of each module with the requirement on huge calculation capacity are expanded onto each node in an HADOOP cluster and related data mining work is carried out by utilizing the parallel calculation and storage capacity of the cluster.

The invention discloses a mass web data mining method based on Hadoop, and belongs to the field of computer data processing. A genetic algorithm is fused with the MapReduce of Hadoop, and mass Web data in a Hadoop-based distributed file storage system (HDFS) is mined to further verify the high efficiency of a platform, and a preferred access route of a user in a Web log is mined with a fused algorithm on the platform. As proved by an experiment result, the efficiency of Web data mining can be remarkably increased by processing of a large amount of Web data with a distributed algorithm in Hadoop.

The invention provides a network data excavation method, which is used for performing text classification and text clustering on acquired webpage information so as to extract topics. The network data excavation method specifically comprises the following steps of S1, catching the webpage information by a preset network probe according to an industrial body; S2, performing text extraction on the acquired network information; S3, performing text classification on extracted texts by a preset classifier to generate a plurality of text type systems; S4, clustering texts under each text type system to generate a plurality of text sub types, wherein each text sub type corresponds to each topic; S5, storing webpage links, and constructing an index according to the text type systems and the text sub types. The network data excavation method provided by the invention can combine repeated information.

The invention discloses a Web community dividing method based on importance degrees and separation degrees of nodes, and belongs to the technical field of Web data mining. The method includes the following steps of firstly, representing a Web network in the form of a figure, representing Web pages through the nodes in the figure, and representing the links between the Web pages through sides between the nodes; secondly, calculating the degree of each node in the figure and the similarity among the nodes; thirdly, calculating the separation degree of each node through the importance degree of the node and the similarity among the nodes; fourthly, calculating the representation degree of each node through the importance degree and the separation degree of the node; fifthly, sequencing all the nodes in the network according to the importance degrees, and selecting a central node of the network community according to the representation degrees of the nodes; sixthly, determining a communitylabel of each network node based on the importance degree and the similarity of the nodes; seventhly, putting the Web pages represented by the nodes with the same community labels in the same community to complete community division.

The invention provides a Hadoop-based mass Web data mining genetic method and belongs to the field of data mining and analysis. According to the Hadoop-based mass Web data mining genetic method, a genetic algorithm is fused with a MapReduce and is used for performing Web data analysis in a Hadoop cluster environment. Experimental results show that a platform can obtain implicit information with apractical value and is high in execution efficiency, and can not only improve the mining efficiency but also overcome the disadvantage of a network environment.