Structured Data Discovery Through Contextual Metadata Disambiguation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large organizations face challenges in comprehensively processing vast amounts of data due to the impracticality of deep profiling tools, leading to inconsistent and outdated data certification, compliance issues, and the development of 'dark pools' of data, exacerbated by manual methods and evolving team dynamics.
Innovation Solution
Utilizing machine learning techniques for contextual metadata disambiguation, including parsing, probabilistic parsing, and bi-directional encoding to transform ambiguous metadata into human-readable formats, enabling efficient and standardized data understanding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep profiling tools are used to comprehensively process vast amounts of data, then data understanding completeness is improved, but processing time becomes impractically long (over a decade)
Solution Approach 1:
The patent extracts and processes only the metadata (data about data) rather than the actual data values. By taking out the structural information from the data itself and focusing only on metadata parsing, the system achieves comprehensive data understanding without needing to scan every data record, thereby reducing processing time from decades to manageable periods.
Solution Approach 2:
The patent creates a virtual copy of the data structure through metadata extraction and parsing. Instead of physically scanning and profiling every data record, the system analyzes the metadata representation (schema, data types, relationships) which serves as a copy sufficient for understanding data structure, sensitivity, and movement patterns without processing the actual data volumes.
2Reliability
If manual certification methods are used by application owners, then compliance can be maintained, but consistency and freshness of data certification deteriorates
Solution Approach 1:
The patent implements self-service through automated metadata parsing and analysis. The system automatically extracts, parses, and analyzes metadata from data sources without requiring manual intervention from application owners. This automation maintains consistent and fresh certification by continuously processing metadata changes, eliminating the variability and delays inherent in manual certification processes while preserving compliance through standardized automated analysis.
3Reliability
If manual data certification is performed by application teams, then compliance tasks can be completed, but productivity of business problem solving is reduced
Solution Approach 1:
The patent replaces the mechanical manual certification process with an automated computational system. Instead of human application owners manually reviewing and certifying data, the system uses automated metadata parsing, tokenization, and analysis algorithms to perform compliance assessment. This substitution eliminates the time-consuming manual tasks that blocked productivity while maintaining compliance through standardized automated evaluation processes.
4Quantity of substance
If comprehensive data profiling is performed on petabytes of data, then data inventory completeness is improved, but the inventory becomes outdated by the time processing completes
Solution Approach 1:
The patent performs preliminary action by extracting and analyzing metadata structure information before any data changes occur. By focusing on the schema, data types, and structural metadata that define how data is organized and named, the system can quickly inventory data structures without needing to scan actual data values. This preliminary structural analysis completes rapidly and maintains freshness because it analyzes the defining characteristics of data rather than the data contents themselves.
Data Source
AI summary
Systems and methods for generalized structured data discovery utilizing contextual metadata disambiguation via machine learning are disclosed. A method may include receiving physical application metadata for an attribute, a database object, or a database; receiving reference data including tokens and their associated abbreviations and acronyms; parsing the physical application metadata into application tokens including known application tokens and unknown application tokens; identifying unknown application tokens by comparing the parsed application tokens to a corpus; performing probabilistic parsing on the unknown application tokens using the reference data resulting in token phrases; identifying monosemous tokens in the token phrases using a look-up dictionary; replacing the monosemous tokens with their expansions from the look-up dictionary; and outputting a mapping of the physical application metadata to enhanced physical application metadata that includes an expression for the physical application metadata comprising the expansions for monosemous tokens in a supported language.