Neural Architecture Search with Sequence-Dependent Edge Probabilities
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural architecture search methods are limited in learning complex dependencies between edges in the search space, leading to suboptimal architecture selection due to fully factorized probability distributions that do not account for sequential dependencies.
Innovation Solution
A novel parameterization of probability distributions that depends on the sequence of previously drawn edges, allowing for more informative edge probability calculations based on the unique coding of the path drawn so far, using functions such as affine transformations or neural networks to optimize the architecture search.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If fully factorized probability distributions are used for edge selection, then the computational complexity is reduced and the method is simpler to implement, but the ability to learn complex dependencies between edges is limited
Solution Approach 1:
The probability distribution is segmented into multiple components: a base distribution that handles individual edge probabilities and additional dependency terms that capture relationships between edges. This segmentation allows the system to manage complexity while preserving dependency information by treating different aspects of the probability distribution separately.
Solution Approach 2:
The patent introduces an additional dimension to the probability distribution parameterization by incorporating sequence-dependent terms. Instead of only considering individual edge features, the method adds a temporal/sequential dimension that captures how previously drawn edges influence current edge probabilities, thereby learning complex dependencies without excessive computational burden.
2Measurement precision
If sequence-dependent probability distributions are used to capture dependencies between edges, then the learning of complex patterns is improved, but the computational complexity and training time increase
Solution Approach 1:
The patent pre-computes and stores edge features and their interactions during a preprocessing phase. By preparing dependency-related information in advance, the training process can efficiently access this pre-computed data without performing expensive calculations during each training step, thereby reducing overall training time while maintaining pattern recognition accuracy.
Solution Approach 2:
The method dynamically adjusts the complexity of the probability distribution parameterization based on the training progress and data characteristics. By changing parameters such as the depth of sequence dependency consideration or the number of dependency terms, the system can balance between learning complex patterns and controlling computational cost during different phases of training.
Data Source
AI summary
Method for creating a machine learning system. The method includes: providing a directed graph with an input node and output node, wherein each edge is assigned a probability which characterizes with which probability an edge is drawn. The probabilities are ascertained depending on a coding of the currently drawn edges.