1375 results about "Large scale data" patented technology

The present invention relates to methods, systems and apparatus for capturing, integrating, organizing, navigating and querying large-scale data from high-throughput biological and chemical assay platforms. It provides a highly efficient meta-analysis infrastructure for performing research queries across a large number of studies and experiments from different biological and chemical assays, data types and organisms, as well as systems to build and add to such an infrastructure. According to various embodiments, methods, systems and interfaces for associating experimental data, features and groups of data related by structure and / or function with chemical, medical and / or biological terms in an ontology or taxonomy are provided. According to various embodiments, methods, systems and interfaces for filtering data by data source information are provided, allowing dynamic navigation through large amounts of data to find the most relevant results for a particular query.

The present disclosure relates generally to a data enrichment service that extracts, repairs, and enriches datasets, resulting in more precise entity resolution and correlation for purposes of subsequent indexing and clustering. As the data enrichment service can include a visual recommendation engine and language for performing large-scale data preparation, repair, and enrichment of heterogeneous datasets. This enables the user to select and see how the recommended enrichments (e.g., transformations and repairs) will affect the user's data and make adjustments as needed. The data enrichment service can receive feedback from users through a user interface and can filter recommendations based on the user feedback.

The present invention relates to the field of deep learning techniques, in particular to a deep convolution neural network training method and a device. The deep convolution neural network training method and the device comprise the steps of a, pretraining the DCNN on a large scale data set, and pruning the DCNN; b, performing the migration learning on the pruned DCNN; c, performing the model compression and the pruning on the migrated DCNN with the small-scale target data set, In the process of migrating learning of large-scale source data set to small-scale target data set, the model compression and the pruning are performed on the DCNN by the migration learning method and the advantages of model compression technology, so as to improve the migration learning ability to reduce the risk of overfitting and the deployment difficulty on the small-scale target data set and improve the prediction ability of the model on the target data set.

The present invention discloses a visual target detection and labeling method. The method includes: an image inputting step, to input an image to be detected; a candidate region extracting step, to extract a candidate window as the candidate region from the image to be detected using selectively search algorithm; a feature description extracting step, to perform feature description on the candidate region using a pre-trained large-scale convolutional neural network and output the feature description of the candidate region; a visual target predicting step, to predict the candidate region based on the feature description of the candidate region using a pre-trained object detection module, to estimate regions having the visual target; and a position labeling step, to labeling the position of the visual target according to the estimated result. Experiments show that, compared with the mainstream week supervision visual target detection and labeling method, the present invention has a stronger ability to excavate positive samples and a more general application prospect, and is suitable for visual target detection and automatic labeling tasks on the large-scale data set.

In a communication network comprising nodes and links between the nodes, a controller node disseminates link state information. A nodal routing table exists at each node comprising routes between pairs of nodes. The nodal routing table is either populated by the given node based on network information received from the controlling node or populated at the controlling node and received by the given node. Each node receives heartbeat signals from its neighbouring nodes. An unexpected delay between heartbeat signals may be perceived as a failure of a link. The perceived failure of that link is reported by the perceiving node to the controlling node. Upon receiving link failure information from a node, the controlling node may determine a subset of nodes in the network influenced by the link failure and indicate the link failure to the determined subset of influenced nodes.

The invention discloses a personal image designing and product recommendation method based on online shopping. The method comprises the steps of personal data collecting, personal figure simulation, product simulation, simulated fitting, personal image designing, specific circumstance image designing, related matching suggestion and commodity recommendation, early purchase warning and social sharing. The method has universality and transportability and aims at different terminals and devices, the terminals and the devices are convenient to carry, and users are served everywhere. The image designing can be carried out according to different occasions and different circumstances of the users. The strong individualized costume matching function and the intelligent commodity recommendation function are achieved. The large-scale data analyzing and mining function is achieved. The interests and hobbies of the users are learned continuously, the analysis of the behavior of the users is carried out, and intimate service is continuously provided for the users.

The present invention is directed to a new visualization platform for the interactive exploration of large datasets. The present invention integrates a collection of relevant visualization techniques to provide a new visual metaphor for viewing large datasets. It is capable of providing comprehensive support for data exploration, integrating large-scale data visualization with querying, browsing, and statistical evaluation. A variety of techniques are utilized to minimize processing delays and the use of system resources, including processing pipelines, direct IO, memory mapping, and dynamic linking of “on-the-fly” generated code.

The invention relates to a graph-based semi-supervised high-spectral remote sensing image classification method. The method comprises the following steps: extracting the features of an input image; randomly sampling M points from an unlabeled sample, constructing a set S with L marked points, constructing a set R with the rest of the points; calculating K adjacent points of the points in the sets S and R in the set S by use of a class probability distance; constructing two sparse matrixes WSS and WSR by a linear representation method; using label propagation to obtain a label function F<*><S>, and calculating the label prediction function F<*><R> of the sample points in the set R to determine the labels of all the pixel points of the input image. According to the method, the adjacent points of the sample points can be calculated by use of the class probability distance, and the accurate classification of high-spectral images can be achieved by utilizing semi-supervised conduction, thus the calculation complexity is greatly reduced; in addition, the problem that the graph-based semi-supervised learning algorithm can not be used for large-scale data processing is solved, and the calculation efficiency can be improved by at least 20-50 times within the per unit time when the method provided by the invention is used, and the visual effects of the classified result graphs are good.

A method and system for creating hypercomplex representations of data includes, in one exemplary embodiment, at least one set of training data with associated labels or desired response values, transforming the data and labels into hypercomplex values, methods for defining hypercomplex graphs of functions, training algorithms to minimize the cost of an error function over the parameters in the graph, and methods for reading hierarchical data representations from the resulting graph. Another exemplary embodiment learns hierarchical representations from unlabeled data. The method and system, in another exemplary embodiment, may be employed for biometric identity verification by combining multimodal data collected using many sensors, including, data, for example, such as anatomical characteristics, behavioral characteristics, demographic indicators, artificial characteristics. In other exemplary embodiments, the system and method may learn hypercomplex function approximations in one environment and transfer the learning to other target environments. Other exemplary applications of the hypercomplex deep learning framework include: image segmentation; image quality evaluation; image steganalysis; face recognition; event embedding in natural language processing; machine translation between languages; object recognition; medical applications such as breast cancer mass classification; multispectral imaging; audio processing; color image filtering; and clothing identification.

A computer implemented system for, and a computer implemented method of calculating indicators to reflect real-life interactions between people, among those are connectivity, social proximity, best paths and trust level. The system comprises a server connected via a communication link to users associated with communication and web based environments and to the web based environments and communication platforms themselves. The server is arranged to receive data relating to users, their profiles, connections and related data in the communication and web based environments as well as large scale data from these environments. The server comprises an application arranged to convert the data into a standard numeric format quantifying the connectivity, the social proximity, the trust level and other indicators to reflect real-life interactions between people. The computer implemented method collects information about the users and their connectivity, and analyses and maps the information as a virtual network spanning a plurality of the web based environments and communication platforms.

The invention discloses a network bandwidth self-adaptive QOS (quality of service) transmission method and system and a terminal device. The network bandwidth self-adaptive QOS transmission method comprises the following steps of: presetting first network state information and a code stream grade in a streaming media session process; obtaining second network state information in real time by utilizing a network period feedback protocol; and comparing the second network state information with the first network state information to determine whether the current network is jammed, if so, down-regulating the code stream grade or sending speed, and if not, keeping or up-regulating the code stream grade or the speed to realize dynamic regulation to real-time communication flows of mobile streaming media. According to the network bandwidth self-adaptive QOS transmission method and system and the terminal device disclosed by the invention, the size of code stream and the sending speed of a streaming media business source can be automatically regulated according to fluctuation of mobile internet bandwidth through a large-scale data acquisition and model test to ensure the stability of real-time transmission of streaming media audio-video data in a mobile internet environment, therefore, different network conditions can be adapted, and user experience can be enhanced.

The invention provides a construction method of a GPU and CPU combined processor, which comprises the following steps: a CPU and a GPU are coupled to construct a combined processor, wherein the CPU is responsible for the general-purpose processing tasks of an operation system, system software, general-purpose application program, and the like which have complex instruction scheduling, circulation, branches and logical judgment; and the GPU is responsible for the highly-parallel calculating processing of large-scale data without logical relation; and the CPU and the GPU jointly finish the same large-scale parallel calculating application. In the GPU and CPU combined processor, a plurality of cores of CPU are communicated with each other by a memory bus and carry out calculation, and the cores of the GPU exchange data and are calculated by a uniform shared memory (a video memory on the GPU); the GPU and the CPU are connected by a high-speed serial bus and exchange calculating data by the memory of the CPU and the shared memory of the GPU; and the construction method can fully utilize the parallel processing capacity of the cores of the GPU and realize the rapid parallel calculation of enormous data volume.

The invention belongs to the technical field of multimedia information searching, and particularly relates to a commodity image category forecasting method based on an online shopping platform. The commodity image category forecasting method mainly involves six modules and comprises corresponding algorithms, namely training image obtaining, image characteristic extracting, irrelevant image filtering, image characteristic training, multilevel image classifying and relevant image selecting. According to the commodity image category forecasting method, based on real data obtained from the online shopping platform, commodity category information in images can be automatically analyzed through large-scale data training, shopping guide can be provided for a user, and therefore online shopping procedures can be simplified for the user, user experience is enhanced, and the commodity image category forecasting method has broad application value in the field of image searching.

The invention discloses a high-availability distribution type full-text index method. The method comprises the following steps of: firstly, starting a basic service system, and then starting an index cluster service and an inquiry cluster service on each node; establishing, updating and deleting an index on full-text data; and finally, inquiring the index. According to the high-availability distribution type full-text index method, an inquiring and indexing process can be separated and the increment or batch type establishment of existing indexes can be simultaneously supported; the batch type indexes can be used for establishing an index for large-scale data in short time; and the increment type indexes avoid reestablishing the index. Index files can be divided into three layers of structures comprising an index file, an index fragment and an index sub-fragment, thus enhancing the expandability and the availability of the index file. According to the high-availability distribution type full-text index method, a dynamic index task configuration task is provided; and parameters in the index task are arranged to dynamically meet requirements of establishing the index by a user through different types of data.

The invention relates to a data compensation method for real time traffic information, which comprises the following steps: optimizing a road net according to the traffic information generated by real-time processing; conducting the abnormal data rejection to the traffic information generated by real-time processing, so as to obtain data which complies with current traffic tendency; choosing historical data which complies with the change of the traffic tendency from a historical database according to the obtained data of real-time traffic tendency to be taken as an auxiliary information source for compensating vacant information; taking a vacant road chain road as a center to construct a compensatory area according to the auxiliary information source and the road net after being optimized, taking the area as a unit to conduct compensatory calculation to road chain which is not covered with traveling track information; determining a filling mode according to the road chain role of a vacant road chain and the number of the vacant road chain in the compensatory area, thereby accomplishing the compensation. The data compensation method fully utilizes the characteristic of the change of traffic flow tendency to develop the information in the historical database to realize the real-time calculation of large-scale data, and the data compensation method has the advantages of high computational efficiency and versatility which is not restricted by areas.

The invention provides a Hadoop-based commodity recommendation system comprising five layers: a data access layer, a data model layer, a recommendation algorithm layer, a recommendation engine layer, and a recommendation application layer. The insides of all layers employ modularization designs and all layers interact with each other based on interfaces, thereby realizing high cohesion and low coupling of the system architecture and high extensibility. The database of the system employs Hive; convenient on-line shopping functions like commodity browsing, purchasing payment, logistics allocation, after-sales service and the like can be provided for users. On the basis of the high-efficiency, reliable, and convenient-extension distributed system infrastructure, the system is suitable for large-scale data storage and computing processing; the system based on a novel design and a good design plan has the broad market promotion prospects.

The invention relates to an intelligent face tracking system and method based on depth learning and large-scale clustering. The system comprises a video input unit, a distribution server, a face identification server cluster, a streaming media server, a distributed file server, a message center server, a web front server and a client of a common operation system. According to the invention, by use of large-scale cluster servers, based on depth learning based face identification technology, a quite high identification rate can still be maintained under the condition of degraded image quality, and more important, a quite low false alarm rate and a quite low missed examination rate are maintained in a large-scale database, such that the reliability and the robustness of the intelligent tracking system can be ensured, and the intelligent tracking system based on face identification can be really applied to the field of safety protection.

The invention discloses a massive data continuous analysis system suitable for stream processing, which comprises a metadata management module, a query plan generation module, a data import task generation module, an increment processing module, an MR (MapReduce) message processing module and a database connection module, wherein the metadata management module is used for managing meta-information of data tables and databases; the query plan generation module is used for receiving a query request and generating an optimal query plan; the data import task generation module is used for receiving a data import request and generating a data import MR operation set; the increment processing module is used for incrementally committing data import and query operations to a Hadoop system in parallel; the MR message processing module is used for receiving a result of a Map or Reduce function of the Hadoop system and outputting the result to a Reduce end or the next operation; and the database connection module is used as an interface between the Hadoop system and the databases. According to the invention, the Hadoop system is used for organically organizing the databases in nodes and simultaneously executing data import and data query and a pipeline technology is used for improving the MR execution flow, so that the data query is executed in a continuous stream mode and the time of analyzing massive data is greatly shortened.

The invention discloses a method and system for classifying data by adopting a decision tree. The method comprises the following steps of: parallel computing the information gain of each attribute in training data based on a MapReduce mechanism, and selecting optimum division decision attributes as nodes to construct the decision tree; based on the decision tree, classifying input data records. Aparallel decision tree ID3 algorithm is realized based on the MapReduce. A large data set can be processed, and the parallel efficiency is high. The parallel computing is realized for the nodes in the decision tree and the nodes in the same layer.

A large-scale data processing system, a large-scale data processing method, and a non-transitory tangible machine-readable medium are provided. The large-scale data processing system comprises an interface and a processor. The interface accesses a multi-dimensional data model, wherein the multi-dimensional data model comprises a plurality of dimensions, the dimensions form a multi-dimensional space of measures, each dimension is a single space comprising a plurality of members with a common set of attributes, and each measure is a data element organized and accessible through the multi-dimensional space of the cross-product of all dimensions. The processor builds at least one Tree Object (TO), wherein the TO is derived by converting the multi-dimensional data model into an N-level tree data structure according to a level order of N attributes, each tree node in the TO meets all conditions of attributes for all ancestor nodes, and N is a positive integer.

It is a purpose of this invention to achieve Scale-Out of the distributed database system that assumes a real-time update to be a requirement and which is achieved by dividing the database system into two or more database domains. This is to achieve handling of even larger scale databases while providing even higher performance. Assuming that the large-scale database system has been distributed to two or more of above-mentioned data base domains, in multi transaction processing with real-time update of the database object across two or more of above-mentioned database domain, this invention is achieved by executing the above-mentioned multi transaction processing to the database meta information storage management part in the database meta information management repository device by applying partition topology technology or replication topology technology for exchange and synchronization of meta information such as status information etc. at even higher speeds.

The invention discloses a face recognition method and system based on a large-scale face database. The recognition method comprises the following steps: obtaining a recognition image; image preprocessing: carrying out illumination compensation, graying, filtering denoising and normalization processing on an image to be detected to obtain a high-quality grey level image; face detection: detecting and positioning a face in a scene image, and separating the face from a background; characteristic extraction and expression: describing and modeling a face mode, and expressing the face by a characteristic vector; face matching: utilizing a similarity between the characteristic vector used for calculating the face image to be detected and a training sample to obtain discrimination information; and outputting a result. The face recognition method and system based on the large-scale face database not only improves the speed and the performance of face recognition under a large-scale database. Meanwhile, the face recognition method and system can be conveniently applied to the recognition field of other modes including facial expression recognition, face tracking and the like.

A data infrastructure for graph-based computing that combines the natural language expressiveness of the Semantic Web and the mathematical rigor of graph theory to discover meaningful associations across multiple sources towards computer-assisted serendipitous insight discovery. The process automatically integrates massive size datasets accessed using Semantic Web standards and technologies and normalizes data in graphs. The process generates a plurality of conditional probability distributions based on type-triple meta-data and triple statistics to model saliency and automatically construct and evaluate a plurality of sub-graphs based on the plurality of conditional probabilities for contextual-saliency. The process then renders a plurality of paths (i.e. sequence of associations) that model meaningful pairwise relations between objects of the normalized integrated data. The pluralities of conditional probabilities reveal and rank previously unknown associations between entities of user-interest in the knowledge graph.

A data distribution network includes a distribution-coordinating server and a plurality of client nodes. Each of the client nodes is configured to pull or push data from other clients or servers. The distribution-coordinating server monitors the locations of data sources and the statuses of data transfers between clients. By tracking the locations of data on the network and client data requests, the distribution-coordinating server can instruct the clients to perform data pushes and / or pulls to coordinate large-scale distributions of data among a multitude of clients. Metadata describing the data and clients can be exchanged between the clients and the distribution-coordinating server to coordinate the data distributions. Also, the distribution-coordinating server can be configured to coordinate data distributions through network security firewalls.

The monitoring device provides value to parents by assisting with monitoring their infants via a convenient sensor package, straightforward interface, and informative data. With movement, orientation, and temperature data, some strong indicators of general well-being can be monitored and conclusions extracted without the parent needing to be constantly involved. The low-power transceiver technology also means that the device can integrate with smart devices for even more convenience. Such a smart device can in turn communicate with modem large-scale data storage and analysis centers for data logging and analytics, which allows useful analytics to be passed back to the user. The advanced sensing, displaying, and analyzing of data makes the invention stands out in the field of infant monitoring devices.

The invention relates to an image super-resolution method based on overcomplete dictionary learning and sparse representation. The method comprises the following steps of: extracting two overcomplete dictionaries (a low-resolution image block dictionary and a high-resolution image block dictionary) in a large-scale dataset and utilizing the two overcomplete dictionaries to realize super-resolution reconstruction of image sparse representation. Simultaneously, in order to further improve the super-resolution effect of color images, the invention also proposes UV chromaticity super-resolution reconstruction based on super-resolution luminance information. The image super-resolution method has wide application prospect in the fields of video monitoring, medical imaging, remote sensing image and the like.

A method to manage the secure download of data (102) to a very large number of network connected client devices (104, 110, 112, 114, 116, 118, 1120, 122) distributed within large geographic areas. The method provides adaptive data throttling and makes optimal use of network data bandwidth. The method can be advantageously applied for managing simultaneous data downloading to millions of network connected remote devices (104, 110, 112, 114, 116, 118, 1120, 122) via private networks, public networks and the Internet (124). A close-loop download regulation algorithm achieves the highest download capacity for the files without exceeding the maximum network capacity and without degrading the transactional and operational traffic performance between the server and the connected client devices.

The invention provides a data parallel processing method. The data parallel processing method comprises the following steps that 1, a main management node receives data and acquires the incidence relation of the data; 2, the main management node calculates allocatable GPUs and GPU work loads of work computing nodes; 3, the main management node partitions the data and distributes the partitioned data to all the work computing nodes; 4, the work computing nodes perform parallel processing on the received data and transmit processing results back to the main management node; 5, the main management node merges the results and then outputs the results. The data parallel processing method has the following advantages that a master-slave architectural pattern is adopted to be used for high-performance large-scale data parallel processing, operation stage partition is performed on specific operations converted by application programs according to DNA feature modeling, node granularity grade operation deployment is performed according to a partition result, and the execution efficiency of a parallel task of data flow in a single node is improved by adopting a thread parallel optimization mechanism and fully utilizing multiple computing kernels.

An individual identity identification method based on deep learning comprises the steps of: utilizing individual data with marked identities to carry out pairing, establishing positive and negative training samples, utilizing the training samples to train an established dual-channel convolution neural network model until the neural network model is convergent, and obtaining neural network model parameters after the training; and utilizing the trained neural network model and related parameters to carry out matching between individual images to be identified and registered individual images, determining the identities of the individual images to be identified according to the magnitude ordering of matching similarities, and outputting a result. According to the invention, characteristic extraction and distance measuring are unified in one one-to-end network, and the global optimization of the whole body is realized; a deep network is utilized to learn characteristic and measuring matrixes, the generalization is higher, and the individual identity identification problem under complex large-scale data is overcome; in addition, the real-time speed is reached, and the practical value is higher.