1370 results about "Large scale data" patented technology

The present invention relates to methods, systems and apparatus for capturing, integrating, organizing, navigating and querying large-scale data from high-throughput biological and chemical assay platforms. It provides a highly efficient meta-analysis infrastructure for performing research queries across a large number of studies and experiments from different biological and chemical assays, data types and organisms, as well as systems to build and add to such an infrastructure. According to various embodiments, methods, systems and interfaces for associating experimental data, features and groups of data related by structure and/or function with chemical, medical and/or biological terms in an ontology or taxonomy are provided. According to various embodiments, methods, systems and interfaces for filtering data by data source information are provided, allowing dynamic navigation through large amounts of data to find the most relevant results for a particular query.

The present invention relates to the field of deep learning techniques, in particular to a deep convolution neural network training method and a device. The deep convolution neural network training method and the device comprise the steps of a, pretraining the DCNN on a large scale data set, and pruning the DCNN; b, performing the migration learning on the pruned DCNN; c, performing the model compression and the pruning on the migrated DCNN with the small-scale target data set, In the process of migrating learning of large-scale source data set to small-scale target data set, the model compression and the pruning are performed on the DCNN by the migration learning method and the advantages of model compression technology, so as to improve the migration learning ability to reduce the risk of overfitting and the deployment difficulty on the small-scale target data set and improve the prediction ability of the model on the target data set.

The present invention discloses a visual target detection and labeling method. The method includes: an image inputting step, to input an image to be detected; a candidate region extracting step, to extract a candidate window as the candidate region from the image to be detected using selectively search algorithm; a feature description extracting step, to perform feature description on the candidate region using a pre-trained large-scale convolutional neural network and output the feature description of the candidate region; a visual target predicting step, to predict the candidate region based on the feature description of the candidate region using a pre-trained object detection module, to estimate regions having the visual target; and a position labeling step, to labeling the position of the visual target according to the estimated result. Experiments show that, compared with the mainstream week supervision visual target detection and labeling method, the present invention has a stronger ability to excavate positive samples and a more general application prospect, and is suitable for visual target detection and automatic labeling tasks on the large-scale data set.

The invention discloses a personal image designing and product recommendation method based on online shopping. The method comprises the steps of personal data collecting, personal figure simulation, product simulation, simulated fitting, personal image designing, specific circumstance image designing, related matching suggestion and commodity recommendation, early purchase warning and social sharing. The method has universality and transportability and aims at different terminals and devices, the terminals and the devices are convenient to carry, and users are served everywhere. The image designing can be carried out according to different occasions and different circumstances of the users. The strong individualized costume matching function and the intelligent commodity recommendation function are achieved. The large-scale data analyzing and mining function is achieved. The interests and hobbies of the users are learned continuously, the analysis of the behavior of the users is carried out, and intimate service is continuously provided for the users.

The present invention is directed to a new visualization platform for the interactive exploration of large datasets. The present invention integrates a collection of relevant visualization techniques to provide a new visual metaphor for viewing large datasets. It is capable of providing comprehensive support for data exploration, integrating large-scale data visualization with querying, browsing, and statistical evaluation. A variety of techniques are utilized to minimize processing delays and the use of system resources, including processing pipelines, direct IO, memory mapping, and dynamic linking of “on-the-fly” generated code.

The invention relates to a graph-based semi-supervised high-spectral remote sensing image classification method. The method comprises the following steps: extracting the features of an input image; randomly sampling M points from an unlabeled sample, constructing a set S with L marked points, constructing a set R with the rest of the points; calculating K adjacent points of the points in the sets S and R in the set S by use of a class probability distance; constructing two sparse matrixes WSS and WSR by a linear representation method; using label propagation to obtain a label function F<*><S>, and calculating the label prediction function F<*><R> of the sample points in the set R to determine the labels of all the pixel points of the input image. According to the method, the adjacent points of the sample points can be calculated by use of the class probability distance, and the accurate classification of high-spectral images can be achieved by utilizing semi-supervised conduction, thus the calculation complexity is greatly reduced; in addition, the problem that the graph-based semi-supervised learning algorithm can not be used for large-scale data processing is solved, and the calculation efficiency can be improved by at least 20-50 times within the per unit time when the method provided by the invention is used, and the visual effects of the classified result graphs are good.

The invention discloses a network bandwidth self-adaptive QOS (quality of service) transmission method and system and a terminal device. The network bandwidth self-adaptive QOS transmission method comprises the following steps of: presetting first network state information and a code stream grade in a streaming media session process; obtaining second network state information in real time by utilizing a network period feedback protocol; and comparing the second network state information with the first network state information to determine whether the current network is jammed, if so, down-regulating the code stream grade or sending speed, and if not, keeping or up-regulating the code stream grade or the speed to realize dynamic regulation to real-time communication flows of mobile streaming media. According to the network bandwidth self-adaptive QOS transmission method and system and the terminal device disclosed by the invention, the size of code stream and the sending speed of a streaming media business source can be automatically regulated according to fluctuation of mobile internet bandwidth through a large-scale data acquisition and model test to ensure the stability of real-time transmission of streaming media audio-video data in a mobile internet environment, therefore, different network conditions can be adapted, and user experience can be enhanced.

The invention discloses a high-availability distribution type full-text index method. The method comprises the following steps of: firstly, starting a basic service system, and then starting an index cluster service and an inquiry cluster service on each node; establishing, updating and deleting an index on full-text data; and finally, inquiring the index. According to the high-availability distribution type full-text index method, an inquiring and indexing process can be separated and the increment or batch type establishment of existing indexes can be simultaneously supported; the batch type indexes can be used for establishing an index for large-scale data in short time; and the increment type indexes avoid reestablishing the index. Index files can be divided into three layers of structures comprising an index file, an index fragment and an index sub-fragment, thus enhancing the expandability and the availability of the index file. According to the high-availability distribution type full-text index method, a dynamic index task configuration task is provided; and parameters in the index task are arranged to dynamically meet requirements of establishing the index by a user through different types of data.

The invention relates to a data compensation method for real time traffic information, which comprises the following steps: optimizing a road net according to the traffic information generated by real-time processing; conducting the abnormal data rejection to the traffic information generated by real-time processing, so as to obtain data which complies with current traffic tendency; choosing historical data which complies with the change of the traffic tendency from a historical database according to the obtained data of real-time traffic tendency to be taken as an auxiliary information source for compensating vacant information; taking a vacant road chain road as a center to construct a compensatory area according to the auxiliary information source and the road net after being optimized, taking the area as a unit to conduct compensatory calculation to road chain which is not covered with traveling track information; determining a filling mode according to the road chain role of a vacant road chain and the number of the vacant road chain in the compensatory area, thereby accomplishing the compensation. The data compensation method fully utilizes the characteristic of the change of traffic flow tendency to develop the information in the historical database to realize the real-time calculation of large-scale data, and the data compensation method has the advantages of high computational efficiency and versatility which is not restricted by areas.

The invention provides a Hadoop-based commodity recommendation system comprising five layers: a data access layer, a data model layer, a recommendation algorithm layer, a recommendation engine layer, and a recommendation application layer. The insides of all layers employ modularization designs and all layers interact with each other based on interfaces, thereby realizing high cohesion and low coupling of the system architecture and high extensibility. The database of the system employs Hive; convenient on-line shopping functions like commodity browsing, purchasing payment, logistics allocation, after-sales service and the like can be provided for users. On the basis of the high-efficiency, reliable, and convenient-extension distributed system infrastructure, the system is suitable for large-scale data storage and computing processing; the system based on a novel design and a good design plan has the broad market promotion prospects.

The invention relates to an intelligent face tracking system and method based on depth learning and large-scale clustering. The system comprises a video input unit, a distribution server, a face identification server cluster, a streaming media server, a distributed file server, a message center server, a web front server and a client of a common operation system. According to the invention, by use of large-scale cluster servers, based on depth learning based face identification technology, a quite high identification rate can still be maintained under the condition of degraded image quality, and more important, a quite low false alarm rate and a quite low missed examination rate are maintained in a large-scale database, such that the reliability and the robustness of the intelligent tracking system can be ensured, and the intelligent tracking system based on face identification can be really applied to the field of safety protection.

The invention discloses a method and system for classifying data by adopting a decision tree. The method comprises the following steps of: parallel computing the information gain of each attribute in training data based on a MapReduce mechanism, and selecting optimum division decision attributes as nodes to construct the decision tree; based on the decision tree, classifying input data records. Aparallel decision tree ID3 algorithm is realized based on the MapReduce. A large data set can be processed, and the parallel efficiency is high. The parallel computing is realized for the nodes in the decision tree and the nodes in the same layer.

A large-scale data processing system, a large-scale data processing method, and a non-transitory tangible machine-readable medium are provided. The large-scale data processing system comprises an interface and a processor. The interface accesses a multi-dimensional data model, wherein the multi-dimensional data model comprises a plurality of dimensions, the dimensions form a multi-dimensional space of measures, each dimension is a single space comprising a plurality of members with a common set of attributes, and each measure is a data element organized and accessible through the multi-dimensional space of the cross-product of all dimensions. The processor builds at least one Tree Object (TO), wherein the TO is derived by converting the multi-dimensional data model into an N-level tree data structure according to a level order of N attributes, each tree node in the TO meets all conditions of attributes for all ancestor nodes, and N is a positive integer.

It is a purpose of this invention to achieve Scale-Out of the distributed database system that assumes a real-time update to be a requirement and which is achieved by dividing the database system into two or more database domains. This is to achieve handling of even larger scale databases while providing even higher performance. Assuming that the large-scale database system has been distributed to two or more of above-mentioned data base domains, in multi transaction processing with real-time update of the database object across two or more of above-mentioned database domain, this invention is achieved by executing the above-mentioned multi transaction processing to the database meta information storage management part in the database meta information management repository device by applying partition topology technology or replication topology technology for exchange and synchronization of meta information such as status information etc. at even higher speeds.

The invention discloses a face recognition method and system based on a large-scale face database. The recognition method comprises the following steps: obtaining a recognition image; image preprocessing: carrying out illumination compensation, graying, filtering denoising and normalization processing on an image to be detected to obtain a high-quality grey level image; face detection: detecting and positioning a face in a scene image, and separating the face from a background; characteristic extraction and expression: describing and modeling a face mode, and expressing the face by a characteristic vector; face matching: utilizing a similarity between the characteristic vector used for calculating the face image to be detected and a training sample to obtain discrimination information; and outputting a result. The face recognition method and system based on the large-scale face database not only improves the speed and the performance of face recognition under a large-scale database. Meanwhile, the face recognition method and system can be conveniently applied to the recognition field of other modes including facial expression recognition, face tracking and the like.

A data infrastructure for graph-based computing that combines the natural language expressiveness of the Semantic Web and the mathematical rigor of graph theory to discover meaningful associations across multiple sources towards computer-assisted serendipitous insight discovery. The process automatically integrates massive size datasets accessed using Semantic Web standards and technologies and normalizes data in graphs. The process generates a plurality of conditional probability distributions based on type-triple meta-data and triple statistics to model saliency and automatically construct and evaluate a plurality of sub-graphs based on the plurality of conditional probabilities for contextual-saliency. The process then renders a plurality of paths (i.e. sequence of associations) that model meaningful pairwise relations between objects of the normalized integrated data. The pluralities of conditional probabilities reveal and rank previously unknown associations between entities of user-interest in the knowledge graph.

A data distribution network includes a distribution-coordinating server and a plurality of client nodes. Each of the client nodes is configured to pull or push data from other clients or servers. The distribution-coordinating server monitors the locations of data sources and the statuses of data transfers between clients. By tracking the locations of data on the network and client data requests, the distribution-coordinating server can instruct the clients to perform data pushes and/or pulls to coordinate large-scale distributions of data among a multitude of clients. Metadata describing the data and clients can be exchanged between the clients and the distribution-coordinating server to coordinate the data distributions. Also, the distribution-coordinating server can be configured to coordinate data distributions through network security firewalls.

The monitoring device provides value to parents by assisting with monitoring their infants via a convenient sensor package, straightforward interface, and informative data. With movement, orientation, and temperature data, some strong indicators of general well-being can be monitored and conclusions extracted without the parent needing to be constantly involved. The low-power transceiver technology also means that the device can integrate with smart devices for even more convenience. Such a smart device can in turn communicate with modem large-scale data storage and analysis centers for data logging and analytics, which allows useful analytics to be passed back to the user. The advanced sensing, displaying, and analyzing of data makes the invention stands out in the field of infant monitoring devices.

The invention provides a data parallel processing method. The data parallel processing method comprises the following steps that 1, a main management node receives data and acquires the incidence relation of the data; 2, the main management node calculates allocatable GPUs and GPU work loads of work computing nodes; 3, the main management node partitions the data and distributes the partitioned data to all the work computing nodes; 4, the work computing nodes perform parallel processing on the received data and transmit processing results back to the main management node; 5, the main management node merges the results and then outputs the results. The data parallel processing method has the following advantages that a master-slave architectural pattern is adopted to be used for high-performance large-scale data parallel processing, operation stage partition is performed on specific operations converted by application programs according to DNA feature modeling, node granularity grade operation deployment is performed according to a partition result, and the execution efficiency of a parallel task of data flow in a single node is improved by adopting a thread parallel optimization mechanism and fully utilizing multiple computing kernels.

An individual identity identification method based on deep learning comprises the steps of: utilizing individual data with marked identities to carry out pairing, establishing positive and negative training samples, utilizing the training samples to train an established dual-channel convolution neural network model until the neural network model is convergent, and obtaining neural network model parameters after the training; and utilizing the trained neural network model and related parameters to carry out matching between individual images to be identified and registered individual images, determining the identities of the individual images to be identified according to the magnitude ordering of matching similarities, and outputting a result. According to the invention, characteristic extraction and distance measuring are unified in one one-to-end network, and the global optimization of the whole body is realized; a deep network is utilized to learn characteristic and measuring matrixes, the generalization is higher, and the individual identity identification problem under complex large-scale data is overcome; in addition, the real-time speed is reached, and the practical value is higher.