Experience-Layered Evolutionary Data Mining Novelty Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Genetic algorithms used in data mining often converge prematurely to local optima, failing to identify global patterns due to homogeneous populations and the inability to accurately assess fitness early in the process, leading to optimization of 'lucky' individuals rather than those with true fitness.
Innovation Solution
Implementing an experience-layered structure in evolutionary data mining systems, where individuals are stratified by testing experience and only considered for elitist pools after reaching a minimum experience threshold, and using novelty search to promote phenotypical diversity and prevent premature convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If genetic algorithms use homogeneous populations and evaluate fitness early, then convergence speed is improved, but premature convergence to local optima occurs and global patterns are missed
Solution Approach 1:
The population is segmented into experience layers (L0, L1, L2, etc.) based on testing experience. Individuals in different layers are evaluated and selected differently, with L0 individuals undergoing novelty competition rather than fitness-based selection. This segmentation prevents premature convergence by maintaining diverse exploration strategies across layers while still enabling fast convergence within each layer.
Solution Approach 2:
The system performs preliminary novelty-based competition in the L0 layer before individuals are promoted to higher experience layers where fitness-based selection applies. This preliminary action ensures that only genuinely novel and diverse individuals advance, preventing early convergence to local optima while maintaining efficient selection in later stages.
2Productivity
If genetic algorithms promote elite individuals early, then optimization efficiency is improved, but diversity of the population decreases leading to premature convergence
Solution Approach 1:
The selection mechanism is made dynamic by adjusting the promotion criteria based on experience layer. In L0, promotion is based on novelty competition results, maintaining diversity. In higher layers (L1, L2, etc.), promotion is based on fitness evaluation, improving efficiency. This dynamic adjustment of selection pressure across different experience stages resolves the contradiction between diversity maintenance and optimization efficiency.
Solution Approach 2:
The system changes the selection parameter from novelty-based (in L0) to fitness-based (in L1+). This parameter change allows the algorithm to maintain population diversity during early exploration phases while enabling efficient optimization in later phases, thus resolving the contradiction between productivity and adaptability.
3Measurement precision
If genetic algorithms evaluate all individuals extensively before selection, then accuracy of fitness assessment is improved, but computational time and resources increase
Solution Approach 1:
The system applies partial evaluation by using novelty competition as a sufficient statistic for L0 individuals without requiring full fitness evaluation. Individuals are promoted to higher layers based on this partial assessment, and only then undergo more extensive fitness evaluation. This partial action approach maintains adequate assessment accuracy while significantly reducing computational time and resources.
Solution Approach 2:
The system performs preliminary novelty-based assessment in L0 before committing to extensive fitness evaluation. This preliminary action filters out non-promising individuals early, so that only those with demonstrated novelty potential undergo time-consuming full fitness assessment in higher layers, thus improving measurement precision for selected individuals while minimizing overall time loss.
Data Source
AI summary
Roughly described, an evolutionary data mining system includes at least two processing units, each having a pool of candidate individuals in which each candidate individual has a fitness estimate and experience level. A first processing unit tests candidate individuals against training data, updates an individual's experience level, and assigns each candidate to one of multiple layers of the candidate pool based on the individual's experience level. Individuals within the same layer of the same pool compete with each other to remain candidates. The first processing unit selects a set of candidates to retain based on the relative novelty of their responses to the training data. The first processing unit reports successful individuals to the second processing unit, and receives individuals for further testing from the second processing unit. The second processing unit selects individuals to retain based on their fitness estimate.