Partial Data Lineage Generation for Data Lake Investigation

Solution Overview

Problem

In data lakes, managing and investigating the influence range of problems across large-scale data lineages is inefficient due to the need to check all accumulated data and processes, leading to extended investigation times and increased complexity.

Innovation Solution

A computer-readable recording medium stores an information processing program that updates and stores update dates and times for data and processes, allowing for the generation of partial data lineages focused on specific data of interest, which includes display objects and cycle information, enabling efficient problem handling by isolating relevant data and processes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the complete data lineage is displayed to manage all data in a data lake, then comprehensive data management is achieved, but investigation time and system complexity increase significantly

Engineering Contradiction:

Improvecomprehensive data managementVSAvoidinvestigation time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent segments the complete data lineage into multiple partial data lineages based on time periods. Each partial data lineage contains only the data and processes relevant to a specific time window, allowing investigators to focus on smaller, manageable subsets rather than examining the entire data lake history. This directly reduces investigation time while maintaining comprehensive coverage when needed.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent extracts and displays only the relevant portion of the data lineage corresponding to the time period when a problem occurred. By taking out and isolating the specific time-windowed data lineage, the system eliminates the need to examine unrelated historical data, thereby significantly reducing investigation time and cognitive load.

Inventive Principle:

Principle #2Taking out (Extraction)

2Reliability

If the complete data lineage is displayed to manage all data in a data lake, then comprehensive data management is achieved, but the complexity of the system increases

Engineering Contradiction:

Improvecomprehensive data managementVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent segments the complete data lineage into multiple partial data lineages based on time periods. Each partial data lineage contains only the data and processes relevant to a specific time window, allowing investigators to focus on smaller, manageable subsets rather than examining the entire data lake history. This directly reduces investigation time while maintaining comprehensive coverage when needed.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies partial action by displaying only the necessary portion of the data lineage (the time-windowed segment) rather than the complete lineage. This partial display reduces system complexity and improves usability by showing only what is needed for the current investigation, while the ability to generate complete lineages remains available when required.

Inventive Principle:

Principle #16Partial or excessive action

3Reliability

If all data and processes are checked to investigate problems, then complete coverage is achieved, but investigation efficiency decreases

Engineering Contradiction:

Improvecomplete coverageVSAvoidinvestigation efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent segments the complete data lineage into multiple partial data lineages based on time periods. Each partial data lineage contains only the data and processes relevant to a specific time window, allowing investigators to focus on smaller, manageable subsets rather than examining the entire data lake history. This directly reduces investigation time while maintaining comprehensive coverage when needed.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary action by pre-organizing data lineage information into time-windowed segments and storing metadata about when each segment is applicable. When a problem occurs, the system can immediately retrieve and display the relevant time-windowed lineage without needing to filter through all data, thereby improving investigation efficiency while maintaining complete coverage capability.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11775497B2Computer-readable recording medium storing information processing program for generating partial data lineage, method of generating partial data lineage, and information processing device for generating partial data lineage

Publication Date: 2023.10.03 FUJITSU LTD

AI summary

A non-transitory computer-readable recording medium stores an information processing program for causing a computer to execute processing including: every time a plurality of data to be managed in a data lineage indicating histories of the plurality of data are updated, storing an update date and time in association with updated data; in response to an input of a designated date and time or a designated period, based on the update date and time stored in association with each of the plurality of data, determining data updated within a predetermined period including the designated date and time or within the designated period as data of interest; and generating a partial data lineage that is a part of the data lineage and includes a display object corresponding to the data of interest.