Observability Metrics for Data Lake Usage Based on Access Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data lakes face challenges with data table management due to continuous data dumping, leading to storage inefficiencies, compliance risks, and difficulty in determining up-to-date and relevant data, as well as unused data tables, which complicates analytics and increases storage costs.
Innovation Solution
The solution involves analyzing data table access at the data layer to determine usage, generating object and table path datasets, and iteratively matching them to create observability metrics, which enables controlling data table access, moving unused data to cold storage, and decommissioning unused pipelines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is continuously dumped into the data lake to ensure nothing is lost, then data completeness is improved, but storage efficiency deteriorates due to cluttered useless information
Solution Approach 1:
The system implements feedback mechanisms by continuously monitoring data access patterns and usage metrics at the data layer. Observability metrics are generated and fed back to identify unused data tables, enabling automated decisions to move or delete such data, thus maintaining storage efficiency while preserving important data
Solution Approach 2:
The system automatically identifies and discards useless data tables through observability metrics analysis. Unused data is moved to cold storage or deleted, while frequently accessed data is retained in hot storage,实现ing dynamic data lifecycle management that improves storage efficiency without losing valuable information
2Adaptability or versatility
If multiple copies of data tables are created for different computation points, then data availability is improved, but device complexity worsens due to jumbled forest of data tables
Solution Approach 1:
The patent introduces an intermediary observability metrics system that sits between the data lake and computation points. This intermediary analyzes access patterns and provides visibility into data usage, enabling automated management of data table copies without increasing manual management complexity
Solution Approach 2:
The system changes the parameter of data table management from static manual tracking to dynamic automated monitoring based on access patterns. By continuously monitoring and analyzing usage metrics, the system automatically determines which data tables to retain, move, or delete, simplifying management despite multiple copies existing in the system
3Loss of information
If all data tables are retained in the data lake, then data access completeness is improved, but storage cost increases due to retained unused data
Solution Approach 1:
The system uses feedback from observability metrics to continuously monitor data access patterns. When data tables are identified as unused based on these metrics, automated actions are triggered to move them to cold storage or delete them, reducing storage costs while maintaining access to frequently used data
Solution Approach 2:
The system implements automated discarding of unused data tables through the observability metrics analysis. Data that is not accessed within certain timeframes is moved to cold storage or deleted, while important frequently accessed data is retained in hot storage, optimizing storage cost without losing access to necessary information
Data Source
AI summary
Incoming data requests from the perspective of the data lake, are gathered and analyzed to determine the usage of the data. Using the perspective of the data lake avoids the technical challenge of analyzing data usage by different computation points, which are at different locations, perform hard-to-track different operations, and are often reachable only through complicated access protocols. Another technical challenge of mapping between an object path and a table path is solved by generating object path datasets and table path datasets at different levels of abstractions. A comparison is performed, iteratively, from a lower level of granularity and the granularity is increased in the progressive steps. Matches from iterations are unionized to generate a final matching data. Observability metrics are generated using final matching data and are used to perform downstream operations such as controlling data table access, moving data tables to cold storage, decommissioning unused pipelines, etc.