Network Space Search for Pareto-Efficient Neural Architectures

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current Neural Architecture Search (NAS) methods rely heavily on human expertise and manual effort to define effective search spaces, which are often reused without exploring untailored spaces, leading to inefficiencies and increased computational costs.

Innovation Solution

The method partitions an expanded search space into multiple network spaces characterized by ranges of network depths and widths, evaluates their performance using a multi-objective loss function, and identifies a target space based on model complexity, reducing human expertise and improving efficiency by automatically searching for Pareto-efficient network spaces.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If human expertise and manual effort are used to define search spaces, then design principles can be established, but the process requires extensive experiments and is time-consuming

Engineering Contradiction:

Improvedesign principle validityVSAvoidexperiment time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs automatic network space search without requiring human expertise to define search spaces. The automated algorithm explores the search space, evaluates architectures, and identifies Pareto-efficient spaces independently, eliminating the need for extensive manual experimentation while maintaining design quality

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual human expertise and experimental validation with an automated computational system. The mechanism substitutes human-driven design principles with algorithmic search and evaluation processes that automatically discover effective network spaces

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If tailored search spaces from previous works are reused, then development time is reduced, but untailored spaces with potential are ignored

Engineering Contradiction:

Improvedevelopment efficiencyVSAvoidspace exploration coverage

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The automated network space search system serves multiple functions: it can search in untailored spaces, adapt to different task requirements, and generate search spaces suitable for various deployment platforms. This universal approach replaces the need for task-specific tailored spaces while maintaining high productivity

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Reliability

If a new effective search space is defined, then search effectiveness is improved, but tremendous prior knowledge and manual effort are required

Engineering Contradiction:

Improvesearch space effectivenessVSAvoidmanual effort requirement

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system automatically defines effective search spaces without requiring tremendous prior knowledge or manual effort. The automated search algorithm explores the space, evaluates architectures based on performance and complexity, and identifies Pareto-efficient spaces independently

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent changes the approach from manually defining search space parameters to automatically optimizing them. The system dynamically adjusts search space boundaries and architecture parameters during the automated search process, eliminating the need for extensive prior knowledge about optimal parameter settings

Inventive Principle:

Principle #35Parameter changes

4Reliability

If extensive experiments are conducted for validation, then design principles are validated, but computational costs increase

Engineering Contradiction:

Improvedesign principle validationVSAvoidcomputational cost

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent replaces extensive experimental validation with an automated computational evaluation system. The algorithm efficiently validates design principles through systematic search and evaluation processes that consume significantly fewer computational resources than traditional experimental approaches

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20230064692A1Network Space Search for Pareto-Efficient Spaces

Publication Date: 2023.03.02 MEDIATEK INC

AI summary

According to a network space search method, an expanded search space is partitioned into multiple network spaces. Each network space includes a plurality of network architectures and is characterized by a first range of network depths and a second range of network widths. The performance of the network spaces is evaluated by sampling respective network architectures with respect to a multi-objective loss function. The evaluated performance is indicated as a probability associated with each network space. The method then identifies a subset of the network spaces that has the highest probabilities, and selects a target network space from the subset based on model complexity.