Log File Bundle Assignment with Timestamps

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Large-scale log file processing systems face challenges in implementing multi-site processing while minimizing processing redundancy and ensuring fault tolerance, as they struggle to efficiently assign and manage log file bundles across multiple processing pipelines without duplicative processing.

Innovation Solution

A log file processing system that includes a data storage device and a processing circuit, which retrieves log file data, creates bundles, and assigns them to multiple processing pipelines with global and pipeline available timestamps to manage eligibility and reassignment, using commit identifiers to track and prevent duplicate processing, and prioritizes data chunks based on event times.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If multiple processing pipelines are implemented to improve robustness and fault tolerance, then system reliability is improved, but processing redundancy increases

Engineering Contradiction:

Improvesystem robustnessVSAvoidprocessing redundancy

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The system performs preliminary actions by setting global available timestamps and pipeline available timestamps before bundle assignment. These timestamps pre-determine eligibility criteria that prevent redundant processing decisions during operation, allowing the system to achieve fault tolerance without unnecessary processing redundancy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements feedback mechanisms through commit identifiers and timestamp checks. When a pipeline completes processing, the system receives feedback via commit requests with commit identifiers, allowing the system to track processed bundles and prevent reassignment to pipelines that have already processed the same bundle, thus eliminating redundancy while maintaining reliability.

Inventive Principle:

Principle #23Feedback

2Reliability

If bundles are assigned to multiple processing pipelines to ensure fault tolerance, then system reliability is improved, but duplicate processing occurs

Engineering Contradiction:

Improvefault toleranceVSAvoidduplicate processing

Core Design Contradiction:

ReliabilityVSLoss of information

Solution Approach 1:

The system creates copies of bundle assignment state through commit identifiers and timestamps. Each pipeline receives a unique commit identifier when assigned a bundle, and the system maintains copy information about which pipelines have processed which bundles. This copying mechanism allows the system to track and prevent duplicate processing while preserving the ability to reassign to different pipelines for fault tolerance.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The processing circuit acts as an intermediary between bundle creation and pipeline execution. It mediates the assignment process by checking global available timestamps and pipeline available timestamps before assigning bundles, and by managing commit identifiers to track processed bundles. This intermediary role prevents duplicate processing decisions while maintaining the multi-pipeline fault tolerance architecture.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If pipeline available timestamps are extended to allow reassignment, then system robustness is improved, but processing time increases

Engineering Contradiction:

ImproverobustnessVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system implements dynamic timestamp management where pipeline available timestamps and global available timestamps can be adjusted based on actual pipeline performance. The timestamps are not fixed but can be extended or shortened dynamically, allowing the system to balance robustness with processing time efficiency based on real-time conditions.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes the parameter of timestamp values dynamically. By adjusting global available timestamps and pipeline available timestamps based on pipeline processing speed and system conditions, the system can optimize the balance between allowing sufficient time for robust processing and preventing excessive delays. This parameter adjustment resolves the contradiction between robustness and processing time.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS8719462B1Systems and methods for distributed log file processing

Publication Date: 2014.05.06 GOOGLE LLC

AI summary

A log file processing system includes a data storage device and a processing circuit configured to retrieve log file data from the data storage device and to create a bundle including a plurality of data chunks using the log file data. The processing circuit is configured to assign the bundle to a first processing pipeline of a set of multiple processing pipelines in response to a request from the first processing pipeline. The processing circuit is configured to store a global available timestamp for the bundle upon assigning the bundle to the first processing pipeline. The global timestamp is a time threshold before which the bundle is ineligible to be assigned to any of the multiple processing pipelines.