Asset Catalog Update via Scan Summary Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data cataloging systems face challenges in reliably removing assets from data stores due to incomplete or failed scans, leading to inaccurate catalog updates and potential removal of assets still present in the data stores.
Innovation Solution
A system that includes a scan initiator, a scan result validator, and a catalog updater, where the scanning agent generates a summary record of scanned assets, and the validator compares it with an independently generated asset summary to ensure scan completeness before removing assets from the catalog.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If scan results are used to update the data catalog, then new assets can be added to the catalog, but assets may be incorrectly removed due to scan failures or lost results
Solution Approach 1:
The system performs a preliminary validation step before removing assets from the catalog. The scan validator independently generates an asset summary and compares it with the summary received from the scanning agent to verify scan completeness and accuracy before proceeding with asset removal, preventing incorrect removals due to scan failures or lost results
Solution Approach 2:
The system implements feedback mechanisms where the scan validator receives and validates summary records from scanning agents, compares them with independently generated summaries, and uses this feedback to determine whether to proceed with asset removal. This feedback loop ensures reliability by verifying scan results before catalog updates
2Measurement precision
If multiple scans are aggregated to improve completeness, then scan accuracy improves, but the complexity of the cataloging system increases
Solution Approach 1:
The system extracts only the essential validation information (asset summaries) from the scan results and uses these extracted summaries for validation purposes. By focusing on comparing summaries rather than processing complete scan result sets, the system achieves validation accuracy without the full complexity of processing multiple aggregated scans
Solution Approach 2:
The scan validator creates a copy of the asset summary from the scanning agent's results and compares it with an independently generated summary. This copying approach allows validation without requiring multiple full scans, reducing system complexity while maintaining measurement precision through summary comparison
Data Source
AI summary
Methods, systems, apparatuses, and computer program products are provided for updating a catalog of assets. A scanning agent may scan for a set of assets in a data store. A scan validator may receive, from the scanning agent, a list of assets scanned in the data store and determine that a summary record is received from the scanning agent. The summary record may include an asset summary generated by the scanning agent corresponding to the list of scanned assets. In response to determining that the summary record is received, the scan validator may determine whether the asset summary in the summary record matches an asset summary generated from the list of assets. If a match is determined between the asset summaries, a match validation may be provided. In response to the match validation being provided, assets absent from the list of assets may be removed from the asset catalog.