Data Mining Fitness Stratification for Evolutionary Algorithm Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Evolutionary algorithms in data mining environments face challenges in determining the fitness of individuals, particularly when the fitness estimation is uncertain and improves gradually with more samples, leading to optimization for 'lucky' individuals rather than those with true fitness, especially in scenarios like financial asset trading where rules for position acquisition and divestiture are unevenly tested.
Innovation Solution
Implementing a data mining system that stratifies individuals into experience layers based on testing experience, ensuring that elitist pool competition occurs only within similar experience levels, and setting a minimum fitness threshold for entry into higher layers to prevent lucky individuals from displacing more experienced ones, thereby improving the generalization ability and performance of the evolutionary process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If evolutionary algorithms test individuals on limited samples initially, then the algorithm can quickly evaluate fitness and proceed with evolution, but the fitness estimation has significant uncertainty and may optimize for 'lucky' individuals rather than truly fit ones
Solution Approach 1:
The patent segments the evolution process into distinct phases: an initial phase with smaller population sizes and fewer generations to quickly establish a baseline, followed by an extended phase with larger population sizes and more generations to refine fitness estimates. This segmentation allows the algorithm to balance initial speed with later accuracy, preventing lucky individuals from dominating the entire evolution process.
Solution Approach 2:
The patent implements dynamic adjustment of evolution parameters, specifically increasing population size and generation count as the evolution progresses. This dynamic approach allows the algorithm to start with faster initial evaluation and then invest more computational resources in later stages when fitness estimates need greater precision, effectively resolving the contradiction between speed and accuracy.
2Quantity of substance
If rules for position acquisition are tested on all data, then these rules receive sufficient testing data, but rules for position divestiture are triggered only when specific states occur, significantly undermining their generalization ability
Solution Approach 1:
The patent applies preliminary environmental alteration by intentionally modifying test environments to create states that trigger divestiture rules before they would naturally occur. This allows divestiture rules to be tested on sufficient data samples during the evolution process, improving their generalization ability without affecting the natural operation of acquisition rules.
Solution Approach 2:
The patent changes environmental parameters during testing to induce specific states that trigger divestiture rules. By systematically varying parameters such as market conditions or system states, the algorithm ensures that divestiture rules receive adequate testing opportunities, balancing the testing data volume between acquisition and divestiture rules.
Data Source
AI summary
Roughly described, a data mining system for use in evolving individuals based on data samples in a training database. The individuals have a plurality of conditions and, for each of a plurality of the data items in a sample, output an action which depends upon application of the individual's conditions to the data item. The conditions include a state of the individual, and the actions include at least one which affects the state of the individual if asserted. Each candidate individual further has a fitness estimate which the system updates in dependence upon the testing results. Testing involves applying the conditions of the individual to data items of the samples to assert actions, and for a subset of at least one but less than all of the data items to which the conditions are applied, modifying the asserted action for the data items in the subset.