Automated Cloud Data Asset Discovery and Catalog Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud computing environments, manually managing and configuring hundreds to thousands of data assets for a tenant is time-consuming and tedious, requiring significant effort from data owners and administrators.
Innovation Solution
An automated discovery process that identifies new data sources and adds them to a data catalog, leveraging existing connections and services within the cloud computing ecosystem, allowing for periodic or scheduled updates without manual intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual creation and management of data assets is performed, then data assets can be added to the data catalog, but the process becomes time-consuming and tedious as the number of assets increases
Solution Approach 1:
The system performs preliminary actions by automatically discovering data assets through monitoring cloud computing resources and analyzing metadata before manual intervention is needed. The data management service proactively identifies data sources, extracts metadata, and prepares data asset information for catalog integration, eliminating the need for manual asset creation and significantly reducing the time required for data asset management.
2Quantity of substance
If the number of data assets increases to hundreds or thousands, then more comprehensive data coverage is achieved, but the manual management process becomes increasingly tedious and time-consuming
Solution Approach 1:
The data management service implements self-service by automatically monitoring cloud computing resources, discovering data sources, extracting metadata, and integrating data assets into the catalog without requiring manual intervention. The system autonomously manages the entire data asset lifecycle including creation, updates, and synchronization, making the management of large numbers of data assets straightforward and eliminating the tedium associated with manual processes.
Solution Approach 2:
The system replaces the mechanical manual process of data asset creation and management with an automated computational system. The data management service uses software-based mechanisms to monitor cloud resources, parse metadata, and integrate data assets programmatically, substituting manual human operations with automated digital processes that scale efficiently regardless of the number of assets.
3Productivity
If automated discovery process is implemented, then data asset integration is faster and more efficient, but the system complexity increases
Solution Approach 1:
The data management service implements multi-functionality by combining multiple capabilities into a single unified system: monitoring cloud computing resources, discovering data sources, extracting metadata, validating data quality, and integrating assets into the catalog. This universal approach consolidates what would otherwise require multiple separate systems into one cohesive platform, achieving fast automated data asset integration while managing system complexity through functional integration.
Data Source
AI summary
System and methods discussed herein are directed to detecting the existence or modification of one or more data assets within a cloud-computing tenancy. The method may include obtaining, by a data management service, a plurality of connection data instances associated with a tenant. Cloud-computing data assets associated with the tenant can be identified from the plurality of connection data instances and presented to the user at a user interface. The user may select which data assets to add to a data catalog managed by the data management service. The data management service may be configured to monitor for new data assets or connections and/or for changes in data and/or connections of previously-selected data assets.