Metadata Portal for Enterprise Data Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Organizations face inefficiencies in searching across multiple data sources due to rapidly changing data and inadequate context in query results, leading to difficulties in finding relevant data assets within their systems.
Innovation Solution
A software platform that ingests metadata from various data sources without accessing the underlying data, processes it, and makes it searchable through a data portal, allowing users to tag, rate, and comment on data assets, and utilizes data lineage to determine relevance in search results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If queries are performed against all data stored across the organization, then completeness of search results is improved, but query response time deteriorates
Solution Approach 1:
The patent extracts and stores metadata describing data assets in a separate search service, allowing queries to be performed on the metadata index rather than scanning all underlying data. This extraction enables fast query response times while maintaining completeness of search results through the metadata descriptions.
Solution Approach 2:
The system performs preliminary actions by pre-processing and storing metadata about data assets before queries are executed. This includes creating searchable indexes of metadata descriptions, tags, and attributes in advance, so that when queries arrive, the system can quickly retrieve relevant results without scanning all data.
2Object-affected harmful factors
If metadata is obtained without accessing underlying data, then security is improved, but measurement precision of data quality deteriorates
Solution Approach 1:
The patent uses metadata as an intermediary representation of the underlying data. The metadata provides a simplified description that can be accessed and processed without direct access to the sensitive underlying data, maintaining security while still enabling quality assessment through metadata attributes like completeness, recency, and source reliability.
3Adaptability or versatility
If data changes rapidly and database structure changes, then adaptability of the system is improved, but reliability of query results deteriorates
Solution Approach 1:
The patent segments the data system into multiple independent components: data sources, metadata descriptions, and search service. This segmentation allows each component to change independently - data sources can update their structures and the metadata can be regenerated without affecting the core search functionality, maintaining reliability through decoupling.
Solution Approach 2:
The system embraces dynamics by automatically updating metadata when underlying data changes. The metadata is dynamically regenerated to reflect current data states, ensuring that search results remain reliable even as data structures evolve. Users can also dynamically add tags and ratings that adapt to changing data environments.
Data Source
AI summary
The subject matter of this disclosure relates to a software platform that ingests data from a variety of data sources, processes the data, and makes the data available for search. In addition, a data portal is described through which users can quickly find information and improve metadata to take advantage of the enterprise's data assets. Various aspects of disclosed software platforms include technologies for obtaining metadata without touching underlying data assets, technologies for using table or database lineage as a factor in determining document relevance, processes for receiving data pushed from data sources, and processes for exposing what a user needs to do to access particular data assets.