Automated Data Dictionary Generation for Secure Cloud Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data sharing methods are cumbersome, slow, and expensive, particularly for smaller entities, as they require manual data cleaning, de-identification, and aggregation, and do not allow scalable sharing or real-time access to updated data, limiting accessibility for smaller businesses and startups.
Innovation Solution
A data exchange platform using cloud computing services allows data providers to share data without copying it, enabling secure, scalable, and real-time access through automated metadata generation and updates, using a data dictionary system that describes shared data, including tables, schemas, and objects, facilitating easier discovery and usage by consumers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual data cleaning, de-identification, and aggregation are performed, then data security and quality are improved, but time consumption and operational complexity increase
Solution Approach 1:
The system performs data cleaning, de-identification, and aggregation in advance before data sharing, so that when data is shared, the processing is already complete. This resolves the contradiction by preparing the data beforehand, ensuring security and quality without adding time to the actual sharing process.
Solution Approach 2:
The system implements automated metadata generation and data processing that operates without manual intervention. The metadata system automatically discovers, generates, and updates data descriptions, resolving the contradiction by making the system self-sufficient rather than requiring manual processing at each step.
2Ease of operation
If data is copied for sharing, then data accessibility and portability are improved, but storage costs and data synchronization complexity increase
Solution Approach 1:
The system creates and manages metadata copies that describe the data without requiring copies of the actual data. The metadata dictionary contains all necessary information about the data structures, schemas, and relationships, allowing data to be accessed and understood without duplicating the data itself, thus reducing storage costs while maintaining accessibility.
Solution Approach 2:
The metadata dictionary acts as an intermediary between data providers and data consumers. It provides descriptions, schemas, and contextual information about the data without requiring direct data copying or transfer, resolving the contradiction by enabling data accessibility through metadata mediation rather than physical data replication.
3Loss of information
If comprehensive data descriptions are provided, then data consumer understanding and data value are improved, but metadata generation complexity and processing time increase
Solution Approach 1:
The metadata generation system operates autonomously, automatically discovering data structures, generating descriptions, and updating metadata without manual intervention. This self-service capability provides comprehensive data descriptions while keeping the generation process simple and automated, resolving the contradiction between completeness and complexity.
Solution Approach 2:
The system implements automated feedback loops where metadata is continuously generated, updated, and validated based on data changes. This feedback mechanism ensures comprehensive and accurate data descriptions are maintained automatically, reducing the complexity of manual metadata management while ensuring completeness.
4Speed
If real-time data access is enabled, then data freshness and operational efficiency are improved, but system resource consumption and infrastructure costs increase
Solution Approach 1:
The system copies and distributes lightweight metadata information rather than duplicating large volumes of actual data. The metadata dictionary contains structural information, schemas, and descriptions that are small in size but provide real-time access to data understanding, enabling fast data access while minimizing resource consumption compared to full data replication.
Solution Approach 2:
The system segments data information into metadata (descriptive information) and actual data (content). By separating these components and providing real-time access to the metadata segment, the system enables fast data understanding and access planning without requiring simultaneous access to all data resources, thus reducing overall system resource consumption while maintaining operational efficiency.
Data Source
AI summary
A data dictionary generation system utilizes a background service that is programmed to automatically populate and update a data dictionary for listings offering shared data. A data dictionary includes metadata describing the shared data overall as well as the individual objects included in the listing, such as the individual tables, schemas, views, and functions. To generate the data dictionary, the data dictionary generation system analyzes the shared data to identify objects, identifies a set of data fields associated with each identified object and populates the set of data fields associated with each identified object based on the shared data offered by the listing. To ensure that a data dictionary for each listing remains up to date, the data dictionary generation system periodically scans the listings to identify any changes to share access granted to the listings.