Automated Dataset Dependency Discovery for Risk-Based Protection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in managing complex dataflows with circular and multiple dependencies, making it difficult to protect against data corruption and allocate resources efficiently, especially with sensitive data types, as manual efforts to identify dependencies are inaccurate and non-scalable.
Innovation Solution
A system automatically discovers dependencies between datasets using logs of read and write operations to generate a dependency graph, calculating a risk metric for each dataset, and provides an indication of risk to prioritize resource allocation and backup processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual methods are used to identify data dependencies, then developers can declare data sources, but the process requires significant manual effort and does not scale well
Solution Approach 1:
The system performs self-service by automatically discovering data dependencies through log analysis without requiring developer intervention. The dependency discovery mechanism autonomously parses logs, identifies read/write operations, and constructs dependency graphs, eliminating the need for manual declaration while maintaining high accuracy.
Solution Approach 2:
The patent replaces the mechanical manual process of dependency declaration with an automated computational system. Instead of developers manually tracking and declaring dependencies, the system uses log parsing algorithms and graph construction mechanisms to automatically discover and represent dependencies, significantly improving scalability and efficiency.
2Reliability
If backup resources are allocated to all datasets, then data corruption can be protected against, but limited resources are wasted on non-critical data
Solution Approach 1:
The system applies local quality by differentiating backup resource allocation based on individual dataset characteristics. Instead of uniform protection, the system calculates risk scores for each dataset considering its specific dependencies, sensitivity, and criticality, then allocates backup resources proportionally to actual risk levels, ensuring critical data receives adequate protection while non-critical data receives minimal resources.
Solution Approach 2:
The patent implements partial action by providing backup protection only to datasets that exceed certain risk thresholds rather than all datasets. The system identifies high-risk datasets through dependency analysis and risk calculation, then applies backup resources selectively to these partial cases, avoiding excessive resource consumption on low-risk data while maintaining adequate overall protection.
3Loss of information
If comprehensive dependency tracking is implemented, then all data flows can be monitored, but the system complexity increases significantly
Solution Approach 1:
The system segments the complex dependency tracking problem into manageable components: log parsing module, dependency extraction module, graph construction module, and risk calculation module. Each module handles a specific aspect of dependency analysis independently, reducing overall system complexity while maintaining comprehensive data flow visibility through the coordinated operation of these segmented components.
Solution Approach 2:
The patent introduces intermediary structures such as dependency graphs and risk score metrics that mediate between raw log data and final protection decisions. These intermediaries simplify the representation of complex dependencies, transforming detailed operational logs into structured graphs that are easier to analyze and process, thereby reducing system complexity while preserving information.
4Object-affected harmful factors
If sensitive data is protected with strict access controls, then data misuse is prevented, but legitimate operations may be hindered
Solution Approach 1:
The system implements feedback mechanisms by continuously monitoring data access patterns and dependency relationships. When sensitive data is accessed, the system evaluates the context, checks dependency requirements, and dynamically adjusts access permissions. This feedback loop ensures that strict protection rules are applied only when necessary while allowing legitimate operations to proceed smoothly, balancing security with operational convenience.
Data Source
AI summary
To assess risk for a target dataset including a plurality of records, a system receives, from a data source, information related to a plurality of datasets including the target dataset; automatically determines dependencies on the target dataset using logs indicative of read and write operations related to the target dataset; generates, using the determined dependencies, a dependency graph indicative of dependencies, within the plurality of datasets, on the target dataset; determines, using the generated dependency graph, a level of risk associated with the target dataset; and provides an indication of the determined level of risk.