169 results about "Data discovery" patented technology

A system for facilitating data discovery on a network, wherein the network has one or more data storage devices. The system may include a crawler program configured to select at least a first set of files and a second set of files, each of the first set of files and the second set of files being stored in at least one of the one or more data storage devices. The system may also include a data fetcher program configured to obtain a copy of the first set of files, the data fetcher program being further configured to resist against obtaining a copy of the second set of files. The system may also include circuit hardware implementing one or more functions of one or more of the crawler program and the data fetcher program.

A method and system to remotely install management software on client machines of a network without requiring user-intervention such as a logon. A data discovery manager determines which discovered machines in a network should be configured for client installation, and queues a client configuration request (CCR) for each such machine. A client configuration manager dequeues each CCR, and attempts to establish a connection with the remote machine corresponding to the CCR. If successful, a bootstrap install service is downloaded to the client machine from a server having administrative rights, after which the bootstrap service is run to install the remainder of the management software on the client machine from a client access point server. If unsuccessful, retries may be attempted. Client machines having users with sufficient security rights may install the management software via a logon or other process at the client machine.

A method and system for the integration of disparate data stored within an Information Technology infrastructure of a company is provided. The system and method enables holistic, real time control of data discovery, retrieval and analysis. The system combines data mining, retrieval and analytics at the source of the data, thereby solving traditional problems with disparate and distributed data, systems, business processes and analytics across an organization. The system enables a user to configure and target data, then apply rules, workflows and analytics from one central source. This process is accomplished by distributing functions in the form of software pneurons against the existing infrastructure for maximum processing while preserving a robust and extendable suite of definitions. The system uses the existing application, network and hardware assets and enables connection to the native data, maps only the data fields that need to be mapped to carry out the desired analysis, runs the analysis and then returns the data to a central location to be assembled, analyzed, organized and / or reported.

Described invention and its embodiments, in part, facilitate discovery of ‘Most Recent Common Ancestors’ in the family trees between a massive plurality of individuals who have been predicted to be related according to amount of deoxyribonucleic acids (DNA) shared as determined from a plurality of 3rd party genome sequencing and matching systems. This facilitation is enabled through a holistic set of distributed software Agents running, in part, a plurality of cooperating Machine Learning systems, such as smart evolutionary algorithms, custom classification algorithms, cluster analysis and geo-temporal proximity analysis, which in part, enable and rely on a system of Knowledge Management applied to manually input and data-mined evidences and hierarchical clusters, quality metrics, fuzzy logic constraints and Bayesian network inspired inference sharing spanning across and between all data available on personal family trees or system created virtual trees, and employing all available data regarding the genome-matching results of Users associated to those trees, and all available historical data influencing the subjects in the trees, which are represented in a form of Competitive Learning network. Derivative results of this system include, in part, automated clustering and association of phenotypes to genotypes, automated recreation of ancestor partial genomes from accumulated DNA from triangulations and the traits correlated to that DNA, and a system of cognitive computing based on distributed neural networks with mobile Agents mediating activation according to connection weights.

A method and a scripting paradigm for automatically integrating disparate information systems (e.g., web services and databases) within a given enterprise into a service-oriented architecture. A script writer generates a script using a scripting paradigm, and the resulting script automatically derives new data models, new ontological structures, new mappings, and a new web service that integrates disparate information systems. In addition to integrating disparate information systems, the scripts may be harvested to automate the metadata discovery and retrieval process. The scripting paradigm builds upon existing open-source scripting languages and is compatible with existing internet browsers, thus encouraging mass participation in the integration process.

The present invention is a method for detecting anomalies against normal profiles and for fusing and visualizing the results from multiple anomaly detection systems in a quantifying and unifying user interface. The knowledge patterns discovered from historical data serve as the normal profiles, or baselines or references (hereinafter, called “normal profiles”). The method assesses a piece of information against a collection of the normal profiles and decides how anomalous it is. The normal profiles are calculated from historical data sources, and stored in a collection of mining models. Multiple anomaly detection systems generate a collection of mining models using multiple data sources. When a piece of information is newly observed, the method measures the degree of correlation between the observed information and the normal profiles. The analysis is expressed and visualized through anomaly scores and critical event notifications that are triggered by fusion rules, thus allowing a user to see multiple levels of complexity and detail in a single view.

The invention discloses a database masking system based on big data. The system comprises a system management module for constructing a role for the database masking system, and performing function authorization and management on the role, a data source management module for providing data source management for the database masking system, a sensitive data discovery module for automatically discovering sensitive data in the database and submitting the sensitive data to a user to adjust and confirm, a masking task management module for configuring and managing the masking task of the database masking system; a masking configuration management module for managing and defining the sensitive data and performing masking algorithm and masking strategy management, a masking verification module for verifying the masking by comparing data before the masking with data after the masking, a multi-stage management module for realizing the status monitoring and strategy issuing and data synchronization of multi-stage masking equipment, and a security auditing module for analyzing the acquired masking log data in real time to guarantee the security of the database masking system. The invention further discloses a database masking method based on the big data.

In embodiments of the present invention improved capabilities are described for developing, training, validating and deploying discovery avatars embodying mathematical models that may be used for document and data discovery and deployed within large data repositories.

The invention discloses a desensitization system used for big data. The desensitization system comprises a database type management module, a system management module, a data source management module, a sensitive data discovery module, a desensitization task management module, a desensitization configuration management module, a multi-stage management module, a big data desensitization analysis module and a sensitive data classification module, wherein the database type management module manages the type of desensitization data; the system management module constructs a role for a desensitization system and carries out function authorization and management on the role; the data source management module provides data source management for the desensitization system; the sensitive data discovery module automatically discovers the sensitive data in the data source and submits the sensitive data to the user for regulation and confirmation; the desensitization task management module configures and manages the desensitization task of a big data desensitization system; the desensitization configuration management module manages and defines the sensitive data and carries out desensitization algorithm and desensitization strategy management; the multi-stage management module realizes the state monitoring, the strategy issuing and the data synchronization of multi-stage desensitization equipment; the big data desensitization analysis module carries out real-time analysis on collected desensitization log data to guarantee the security of the desensitization system; and the sensitive data classification module carries out characteristic analysis on the sensitive data to classify the sensitive data. The invention also discloses a desensitization method used for the big data.

A high availability, scalable cluster server system for enterprise data management. The server is a cluster of two or more nodes. Each node runs one or more virtual servers. A virtual server consists of network resources and resources for enterprise data management. Failover is based on moving virtual servers from a failed node to a healthy node. The invention provides for network failover within the same node. Enterprise data management consists of data discovery, data categorization and applying enterprise policies on categorized data. One of the problems of data management is the diversity of policies and in some cases their conflicting requirements. According to one aspect of the invention, enterprise policies are broking into policy entities. Policy entities represent the simplest policy unit. Some of the policy entities are shared between the diversified enterprise policies. Identifying a data management policy with the highest priority and applying the policy entity that belongs to it resolve conflict in policy requirements.

The invention pertains to the technical field of information processing, in particular relates to an integration method of distributed heterogeneous information resources. The method of the invention realizes the integration and share of the distributed heterogeneous information resources through the operations of an information resource provider end and an information resource collection end. The realization steps are as follows: (1) the following steps are carried out in sequence at the information resource provider end: heterogeneous data discovery, metadata configuration, catalog generation, catalog integration, data package as well as data upload that is uploading the packaged data in the form of data packet to the information resource collection end; (2) when the information resource collection end receives the data packet sent by the information resource provider end, data packet decompression, the analysis of configuration files and data storage are carried out. The method of the invention takes full consideration of the flexibility and scalability of structure; database type heterogeneity and data table heterogeneity are solved; centralized storage or distributed management of the distributed information sources is solved; the unified catalog storage of distributed heterogeneous information resources is realized; the problem of 'Information Island' that is increasingly serious is solved; the integration and share of the distributed heterogeneous information resources are effectively finished.

The invention belongs to the technical field of large data platforms, in particular to a unified data resource management system and method for a large data platform. In order to solve the problem that the data resources are difficult to manage due to the presence of different types of data resources in the large data platform, the unified metadata description for different types of data resources and the unified adapter interface criterion for different types of data management components are provided, and the unified data resource management method and system are designed on such basis to support unified data upload and download, unified data discovery, unified data access application and authorization, and other functions. The unified data resource management system and method provided by the invention can realize the dynamic extension, multi-tenancy management and uniform access control of the system, so that a user can manage and use different types of data resources in the large data platform.

Methods and systems for creating ensembles of hypersurfaces in high-dimensional feature spaces, and to machines and systems relating thereto. More specifically, exemplary aspects of the invention relate to methods and systems for generating supervised hypersurfaces based on user domain expertise, machine learning techniques, or other supervised learning techniques. These supervised hypersurfaces may optionally be combined with unsupervised hypersurfaces derived from unsupervised learning techniques. Lower-dimensional subspaces may be determined by the methods and systems for creating ensembles of hypersurfaces in high-dimensional feature spaces. Data may then be projected onto the lower-dimensional subspaces for use, e.g., in further data discovery, visualization for display, or database access. Also provided are tools, systems, devices, and software implementing the methods, and computers embodying the methods and / or running the software, where the methods, software, and computers utilize various aspects of the present invention relating to analyzing data.

The invention relates to a discovery method of a track data hot spot based on local multilayer grids. The technical scheme includes dividing track motion space locally and in a multilayer mode according to distribution characteristics of track data and calculating cell densities; screening and expanding a density cell under a given threshold and calculating a candidate hot spot; and screening the hot spot based on the residence time of candidate hot spot track support counts and track data in the candidate hot spot. According to the method, self-adaptive effects of a divided grid coverage motion plane are good, suitable grid coverage motion can always be obtained in an iterative classification process as long as a sample point number threshold value in a cell is assigned, sample data are divided meticulously, initial classification parameters cannot affect discovery results greatly as long as a set density threshold is fixed, the method can be applied to mass data discovery, and the efficiency and the adaptability of an algorithm are guaranteed.

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph.

A system and methods for capacity planning of electronic systems storage are provided. Features include a non-invasive data discovery and collection mechanism, an adaptive grouping mechanism, a flexible analysis mechanism, predictive modeling and forecasting mechanisms, and a business metric and correlation mechanism. A discovery engine that ascertains the availability of collectable entities. A collection engine in turn gathers information from and about entities discovered by the discovery engine. A grouping module groups together entities according to parameters specified by the user. An analysis and forecasting module allows a user to create, modify and save forecast scenarios from which a model is generated by a modeling module. Correlations can be identified between specified key business metrics and historical data. Once a model has been created, a forecast is generated and can then be analyzed using Bold Freehand Extrapolation, time-series analysis, and business trend-based forecasting.

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph.

The invention discloses a detection and matching mechanism for recognition of handwritten letters using WiFi signals. The invention mainly makes use of the characteristics of wireless signal multipath propagation, and acquires CSI (Channel State Information) data that carry information reflecting external environmental characteristics, including hand movements when letters are written. The CSI data after de-noising (via Butterworth filtering and principal component analysis filtering) show that in the stage of writing letters, the CSI data fluctuate significantly, and the CSI data are relatively stable in the stage without handwriting, and on the basis of the characteristics, the invention provides a waveform feature extraction algorithm to detect and extract waveforms only including information on 26 handwritten letters. Finally, the invention provides a solution combining feature matching and context error correction, takes the waveform as a matching feature, carries out machine learning classification for the extracted 26 feature waveforms, matches the waveforms by using a DTW (dynamic time warping), and improves the recognition accuracy of the system through the context error correction algorithm.

A system for facilitating data discovery on a network, wherein the network has one or more data storage devices. The system may include a crawler program configured to select at least a first set of files and a second set of files, each of the first set of files and the second set of files being stored in at least one of the one or more data storage devices. The system may also include a data fetcher program configured to obtain a copy of the first set of files, the data fetcher program being further configured to resist against obtaining a copy of the second set of files. The system may also include circuit hardware implementing one or more functions of one or more of the crawler program and the data fetcher program.

A system and a method used for data discovery in accordance with an inquiry in which multiple sources, which may be web sites or other data sources, are examined for data relevant to the inquiry. The process and method is performed recursively an indeterminate number of iterations, using data and metadata from multiple sources to corroborate discovered data and metadata from other sources, until no further relevant data or sources are found, or adjudication or exception rules have been met. Discovered data and metadata are curated, adjudicated to assess reliability, synthesized, and clustered into composite records using precedence rules and provenance to determine the most reliable data sources as well as terms of use for each source. Data, metadata, and information about each search are retained and can be used for subsequent purposes, such as subsequent searches or other downstream activities.

A discovery application may obtain, from a remote storage system configured to host storage structures, first data that identifies relationships among the storage structures. The discovery application may generate, based on the first data, a mapping of the storage structures, store the mapping as one or more configuration items, and provide, to the remote storage system, instructions configured to cause the remote storage system to notify the discovery application of modification events associated with the storage structures. The discovery application may receive, from the remote storage system, a notification of a modification event associated with a particular storage structure of the storage structures and, in response, obtain, from the remote storage system, second data that identifies a modification to the particular storage structure. The discovery application may modify the mapping based on the second data and store the modified mapping by updating the one or more configuration items.

A method and a sensitive data discovery engine (SDDE) are provided for discovering substantial sensitive data in source systems spurning across similar and variant data sources and applications. The SDDE configures a scanning pathway for scanning data based on a predefined or configurable unique data classification. The scanning pathway defines a sequence of one or more match operations including master data field, dictionary, code, pattern and exact data matches to be performed on the data for the unique data classification. The SDDE executes the match operations in the scanning pathway on the data based on a scan level, scores the data, and determines sensitive data. The SDDE generates a sensitive data discovery map report including locations of the sensitive data and discovery metadata including information of users and programs that access the sensitive data, generated by inspecting application codes, for use in downstream data protection and governance operations.

Methods and systems for creating ensembles of hypersurfaces in high-dimensional feature spaces, and to machines and systems relating thereto. More specifically, exemplary aspects of the invention relate to methods and systems for generating supervised hypersurfaces based on user domain expertise, machine learning techniques, or other supervised learning techniques. These supervised hypersurfaces may optionally be combined with unsupervised hypersurfaces derived from unsupervised learning techniques. Lower-dimensional subspaces may be determined by the methods and systems for creating ensembles of hypersurfaces in high-dimensional feature spaces. Data may then be projected onto the lower-dimensional subspaces for use, e.g., in further data discovery, visualization for display, or database access. Also provided are tools, systems, devices, and software implementing the methods, and computers embodying the methods and / or running the software, where the methods, software, and computers utilize various aspects of the present invention relating to analyzing data.

Interface code is produced to use services of an enterprise information system. An application specific information schema is provided that defines the format and shape of application specific information usable in a business object schema defining the business object and the application specific information for the enterprise information system. An enterprise metadata discovery schema is also provided for anchoring the application specific information schema to complex type, element, attribute and the business object schema. The business object schema is generated using a metatdata editor. The interface code is produced using the business object schema.

A document desensitization system based on big data is used to desensitize Word, Excel, PPT, TXT, PDF, XML format documents desensitization processing, the system is mainly composed of system management module, data source management module, sensitive data discovery module, desensitization task management module, desensitization configuration management module, desensitization verification module,multi-level management module, security audit module seven modules. By means of natural language processing and semantic analysis, the invention solves the identification problem of sensitive data ina document, and the identification accuracy is high. The invention provides a method for solving static desensitization and dynamic desensitization of unstructured data such as documents, which ensures the safety of sharing and exchanging documents under the environment of big data. By analyzing the document, identifying the sensitive data in the document and desensitizing the document, the invention ensures that the original format of the document is not destroyed, and effectively solves the difficulty of desensitizing the document.

The invention discloses a sensitive data discovery method based on text recognition. The sensitive data discovery method comprises the following steps of S01, extracting the sample data; S02, constructing a training sample, collecting a text data set, and constructing the training sample; S03, training a sample annotation model, obtaining a training sample based on S02, and training a text annotation model; S04, constructing data features; S05, constructing a training set, carrying out label description on the data set obtained in the S04 to form a training set for constructing a classification judgment model; S06, constructing a classification judgment model, and forming a variable prediction model according to the training set obtained in the S05; S07, testing the model. Through the identification of the data variables, the sensitive data can be accurately and efficiently judged and identified under the condition that the data dictionary and the matching rules are incomplete, and theconsistency of identification and classification results is ensured.

The invention discloses an intelligent backup method based on data discovery. The method is applied to a data backup system composed of a backup client side and a backup server. The method includes that the backup client side records data object description information in a backup protecting path in real time on the basis of the data discovery technology and sends a backup request to the backup server according to backup strategy conditions; the backup server acquires the data object description information, matches a sub backup strategy from a preset backup strategy according to the data object description information and backups data in the backup protecting path of the backup client side according to the matched sub backup strategy.