Log Score-Based Task Logging in Distributed Stream Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed stream data processing platforms, developers face challenges in predicting log output during task operation, leading to excessive or insufficient log acquisition, which hampers failure investigation and increases processing load.
Innovation Solution
An information processing apparatus calculates a log score for each task based on failure occurrence likelihood and adjusts the output frequency of logs considering upstream and downstream tasks' scores, optimizing log collection and reducing platform load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If log output frequency is increased for each task to ensure sufficient log acquisition for failure investigation, then measurement precision of task status is improved, but processing load and system performance deteriorate
Solution Approach 1:
The patent applies local quality by differentiating log output frequencies across different tasks based on their individual failure probabilities. Instead of uniform log collection, each task receives customized log output frequency tailored to its specific risk profile, ensuring sufficient logs for high-risk tasks while minimizing overhead for low-risk tasks.
Solution Approach 2:
The patent changes the parameter of log output frequency dynamically based on calculated failure probabilities. The system adjusts this parameter for each task according to its upstream and downstream task relationships, transforming a static uniform logging approach into a dynamic adaptive one that responds to system state.
2Productivity
If log output frequency is decreased to reduce processing load, then productivity is improved, but measurement precision of task status deteriorates leading to insufficient log acquisition
Solution Approach 1:
The patent performs preliminary action by calculating failure probabilities for each task before determining log output frequencies. This advance assessment allows the system to pre-establish appropriate logging levels for each task, ensuring that sufficient logs are captured proactively rather than reactively after failures occur.
Solution Approach 2:
The patent implements feedback by considering both upstream and downstream task failure probabilities when determining log output frequency for each task. This feedback mechanism ensures that tasks connected to high-risk operations automatically receive higher logging levels, creating a responsive system that adapts to system-wide risk patterns.
3Ease of operation
If uniform log output frequency is applied to all tasks, then ease of operation is improved, but manufacturing precision of log collection efficiency deteriorates
Solution Approach 1:
The patent transforms the static uniform logging approach into a dynamic system that automatically adjusts log output frequencies based on calculated failure probabilities. The system evolves from a fixed configuration to an adaptive one that responds to task-specific and system-wide conditions, optimizing efficiency without requiring manual intervention.
Data Source
AI summary
An information processing apparatus includes: a memory and a processor. The processor, coupled to the memory, is configured to: determine, for each task of a plurality of tasks executed in a distributed stream data processing platform, a log score based on an indication associated with easiness of occurrence of a failure; and output a log message for each task of the plurality of tasks at an output frequency based on a log score of each task and a log score of at least one of an upstream task located upstream of each task and a downstream task located downstream of each task.