LeakyReLU Activation in CNN Video Filters

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing neural network-based filters in video coding face instability during training due to the use of Parametric Rectified Linear Unit (PReLU) activation functions, which can lead to performance issues and training divergence.

Innovation Solution

Employing a Convolutional Neural Network (CNN)-based filter with a Leaky Rectified Linear Unit (ReLU) activation function, which provides better performance and stability compared to PReLU by maintaining a fixed Alpha value, thereby enhancing video coding efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If PReLU activation function is used in CNN-based filter, then adaptability is improved, but training stability deteriorates

Engineering Contradiction:

ImproveadaptabilityVSAvoidtraining stability

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent changes the activation function from PReLU to LeakyReLU, modifying the parameter behavior by using a fixed Alpha value instead of a trainable parameter. This parameter change resolves the training instability while preserving adaptability through the leaky ReLU mechanism.

Inventive Principle:

Principle #35Parameter changes

2Manufacturing precision

If PReLU activation function is used in CNN-based filter, then performance is improved, but training stability deteriorates

Engineering Contradiction:

ImproveperformanceVSAvoidtraining stability

Core Design Contradiction:

Manufacturing precisionVSReliability

Solution Approach 1:

The patent applies parameter changes by switching from PReLU with trainable parameters to LeakyReLU with fixed Alpha parameter. This resolves the training divergence issue while maintaining performance through the leaky ReLU activation mechanism that prevents neuron death.

Inventive Principle:

Principle #35Parameter changes

3Reliability

If LeakyReLU activation function is used in CNN-based filter, then training stability is improved, but device complexity increases

Engineering Contradiction:

Improvetraining stabilityVSAvoiddevice complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent extracts the trainable parameter aspect from the activation function by using a fixed Alpha value in LeakyReLU instead of a learnable parameter. This simplifies the device complexity by removing the need for parameter storage and computation while maintaining training stability.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS11778213B2Activation function design in neural network-based filtering process for video coding

Publication Date: 2023.10.03 QUALCOMM INC

AI summary

A method of coding video data, the method comprising: reconstructing a block of the video data; and applying a Convolutional Neural Network (CNN)-based filter to the reconstructed block, wherein the CNN-based filter uses a LeakyReLU activation function.