ML
Genetic Algorithm Hyperparameter Search
A Python optimizer that evolves classifier settings through cross-validation, selection, crossover, and mutation while handling dependencies between parameters.
- Genetic Algorithms
- Hyperparameter Tuning
- scikit-learn
- Python
- Cross-Validation
Searching through generations
Exhaustive hyperparameter search grows expensive as options multiply. I implemented an evolutionary optimizer that scores a population of classifier settings and breeds candidates for the next generation.
The search preserves its strongest candidate and handles model-specific dependencies, including logistic-regression penalties that restrict which solvers can be used.
Evolution across generations
- Generation 1
- Generation 2
- Generation 3
How candidates evolve
I built a genetic algorithm that searches for classifier settings by evolving a population of candidates. Each candidate is a set of hyperparameter values, and its fitness is the mean cross-validation score returned by scikit-learn.
Tournament selection chooses parents by comparing a sampled group of candidates. Crossover takes each parameter from either parent, while mutation can replace one value to explore a different configuration. The best candidate found across all generations is copied into the next population, so breeding cannot discard it.
The optimizer records average and best-so-far fitness after each generation, then fits the selected configuration on the supplied training data. Categorical options, numeric samplers, and tuple-valued neural-network layer sizes give it several kinds of search space to work with.
Handling dependent parameters
Some settings cannot be chosen independently. In logistic regression, the penalty determines which solvers are valid. I added compatibility rules to candidate creation and the genetic operators, so changing a penalty also chooses a compatible solver.
For example, an L1 candidate can use liblinear or saga, while L2 permits a broader solver set. A final fitness check rejects incompatible pairs and catches selected fitting failures. Encoding that dependency in the search keeps it from repeatedly proposing combinations the estimator cannot train.
Comparing search cost and accuracy
I compared the optimizer with grid, randomized, and halving-grid search in classifier experiments. The final report’s diabetes classification comparison gives SVC the same test accuracy with genetic, grid, and randomized search, with substantially different elapsed times.
Randomized search was slightly faster than the genetic algorithm in this run. Search budgets and fold parallelism differed, so the table describes those experiments rather than an inherent speed advantage. The custom loop evaluates candidates serially and lets scikit-learn handle parallel fitting within cross-validation.
| Search method | Test accuracy | Elapsed (s) |
|---|---|---|
| Genetic | 0.7416 | 160.09 |
| Grid | 0.7416 | 940.42 |
| Randomized | 0.7416 | 155.49 |
| Halving grid | 0.6854 | 194.74 |
Tools and technologies
- Python
- scikit-learn
- NumPy
- pandas
- Jupyter
- Matplotlib
- seaborn
Python implements selection, crossover, mutation, and compatibility handling. scikit-learn supplies classifiers, cross-validation, and baseline searches. NumPy and pandas support experiment data; Matplotlib and seaborn support fitness and result plots.