Data Lake Ownership Detection via Write Access Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining ownership of cloud computing resources, particularly data lake tables, are computationally expensive and inefficient, especially for organizations with numerous and frequently changing instances.
Innovation Solution
A method and system that utilize write access logs to identify the owner of data lake tables by associating IP addresses with cloud instance identifiers, verifying the instance's creation time, and using cost allocation tags to determine ownership, thereby updating central ownership records efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional methods are used to determine cloud computing resource ownership, then ownership information can be maintained, but the computational cost and complexity increase significantly
Solution Approach 1:
The system uses write access logs to automatically determine ownership without requiring manual intervention or complex computational processes. The cloud computing instance itself provides the necessary information (write access rights, IP addresses, timestamps) that enables automatic ownership determination, eliminating the need for complex external verification systems.
Solution Approach 2:
The patent extracts ownership determination logic from complex computational processes and simplifies it by using directly available data from write access logs. Instead of performing complex computations to determine ownership, the system extracts relevant information (IP address, timestamp, write access rights) already present in the logs and uses this to identify the owner.
2Reliability
If comprehensive ownership tracking is implemented for all cloud instances, then ownership accuracy improves, but the system scalability decreases
Solution Approach 1:
Each cloud computing instance automatically provides its own identification information (IP address, timestamp, write access rights) through its write access logs. This self-providing mechanism eliminates the need for a centralized system to actively track and verify ownership of each instance, enabling scalable operation without sacrificing accuracy.
Solution Approach 2:
The system segments the ownership determination process by handling each cloud instance independently through its own write access logs. Rather than maintaining a monolithic ownership tracking system, the patent processes ownership information on a per-instance basis using the instance's own log data, which improves scalability.
3Reliability
If manual ownership verification processes are used, then ownership can be determined, but the time required for determination increases
Solution Approach 1:
The system performs preliminary action by continuously collecting and storing write access log data (IP addresses, timestamps, write access rights) as cloud instances operate. When ownership determination is needed, this pre-collected data is immediately available for quick analysis, eliminating the need for time-consuming manual verification processes.
Solution Approach 2:
The write access logs automatically record the necessary ownership information as part of normal cloud instance operation. This self-recording mechanism provides ready-to-use data for ownership determination without requiring manual intervention or time-consuming verification processes.
Data Source
AI summary
Systems and methods for determining ownership of cloud computing resources are disclosed. An example method includes identifying a first active table whose ownership is not defined in a central repository, determining, based on a write log associated with the first active table, a first timestamp and a first internet address associated with a most recent write to the first active table, determining, based on the first internet address, whether or not the first timestamp is more recent than a creation time of a first cloud computing instance corresponding to the most recent write, and in response to the first timestamp being more recent than the creation time of the first cloud computing instance, identifying a first owner of the first active table based on a first cost allocation tag associated with the first cloud computing instance.