N-Pool Evolution Data Mining for Fitness Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data mining environments, the fitness estimation of individuals in evolutionary algorithms is initially inaccurate and improves over time, making it challenging to determine suitable individuals for harvesting until they reach a sufficient testing experience level, and existing methods lack effective stratification and validation processes.
Innovation Solution
The implementation of an elitist pool with multiple layers based on testing experience levels, where individuals are stratified and promoted based on their fitness and experience, with a minimum fitness threshold to ensure accurate selection and reduce resource allocation to inferior individuals, and the use of n-pool cross-validation to validate and harvest individuals effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If individuals are selected for harvesting based on early fitness estimates, then the selection process is faster, but the accuracy of fitness estimation is insufficient leading to poor generalization
Solution Approach 1:
The testing database is divided into multiple segments, with each segment assigned to a separate pool. Individuals are tested on their home segment first for quick initial evaluation, then validated on other segments for final selection. This segmentation allows early time-saving evaluations while ensuring later accuracy through cross-validation.
Solution Approach 2:
Individuals undergo preliminary testing on their home segment before full validation. This preliminary action provides an early fitness estimate that can quickly eliminate poor performers, saving time. Only individuals passing this preliminary test proceed to full cross-validation, ensuring accuracy for final harvesting decisions.
2Device complexity
If a single validation step is used for all individuals, then the validation process is simpler, but resource allocation is inefficient due to lack of stratification
Solution Approach 1:
The validation process is segmented into multiple pools corresponding to different database segments. Each pool handles validation against a specific segment, creating a structured multi-stage process that is slightly more complex but dramatically improves resource allocation efficiency by preventing redundant testing.
Solution Approach 2:
Different pools perform different validation functions based on their assigned database segments. Each pool is optimized for its specific validation task, allowing efficient resource allocation where each validation operation focuses on its local expertise rather than all pools performing all validations.
3Measurement precision
If all database segments are used for testing each individual, then the fitness estimation is most accurate, but the testing process becomes excessively time-consuming
Solution Approach 1:
The database is segmented and assigned to different pools, with each individual tested primarily on its home segment for initial evaluation. This segmentation reduces the testing burden per individual while maintaining overall accuracy through the collective coverage of all pools validating on their respective segments.
Solution Approach 2:
Individuals are first tested on their home segment as a preliminary screening step. This preliminary action provides sufficient initial accuracy to filter out poor performers quickly, eliminating the need for all individuals to undergo the full time-consuming process of testing on all segments.
4Device complexity
If no stratification is applied to the candidate pool, then the pool management is simpler, but inferior individuals consume excessive computational resources
Solution Approach 1:
The candidate pool is stratified into multiple pools based on which database segment each individual is primarily tested on. This segmentation creates a structured management system that is slightly more complex but prevents inferior individuals from consuming excessive resources by distributing them across multiple specialized pools rather than allowing them to persist in a single undifferentiated pool.
Solution Approach 2:
Each pool is specialized for validating individuals against a specific database segment. This local quality ensures that each pool focuses its computational resources on its specific validation task, preventing waste from inferior individuals being repeatedly tested across all segments by all pools.
Data Source
AI summary
Roughly described, a training database contains N segments of data samples. Candidate individuals identify a testing experience level, a fitness estimate, a rule set, and a testing set TSi of the data samples on which it is tested. The testing sets have fewer than all of the data segments and they are not all the same. Testing involves testing on only the individual's assigned set of data segments, updating the fitness estimates and testing experience levels, and discarding candidates through competition. If an individual reaches a predetermined maturity level of testing experience, then validating involves further testing it on samples of the testing data from a testing data segment other than those in the individual's testing set TSi. Those individuals that satisfy validation criteria are considered for deployment.