Hyperparameter Optimization 3 - Automated Hyperparameter Tuning with Python Libraries, Optuna, Hyperopt, and Scikit-Optimize

In the previous blog posts, we introduced the concept of hyperparameter optimization and explored basic techniques like Grid Search and Random Search. In this post, we will dive into more advanced and automated techniques for hyperparameter tuning using popular Python libraries: Optuna, Hyperopt, and Scikit-Optimize. These libraries implement advanced optimization algorithms that can efficiently search for the best hyperparameters in large search spaces.

Optuna

Optuna is a powerful Python library for hyperparameter optimization that uses a combination of Tree-structured Parzen Estimator (TPE) and pruning strategies to efficiently search for the best hyperparameters. Optuna is designed to be easy to use and highly customizable, making it suitable for a wide range of optimization problems.

Hyperopt

Hyperopt is another popular Python library for hyperparameter optimization. It uses the TPE algorithm to efficiently search for the best hyperparameters. Hyperopt is designed to be highly flexible and can be used for a wide range of optimization problems, including deep learning and reinforcement learning.

Scikit-Optimize

Scikit-Optimize is a library for sequential model-based optimization (SMBO) in Python. It provides several optimization algorithms, including Bayesian optimization, which is a powerful technique for finding the global optimum of a function with minimal evaluations. Scikit-Optimize is designed to be easy to use and integrates well with Scikit-learn, making it a popular choice for hyperparameter tuning in machine learning.

Example: Hyperparameter Tuning with Optuna, Hyperopt, and Scikit-Optimize

In this example, we will demonstrate hyperparameter tuning using Optuna, Hyperopt, and Scikit-Optimize on the famous Iris dataset with the Support Vector Machine (SVM) algorithm.

1. Import necessary libraries and load the dataset:

import numpy as np
import pandas as pd
from sklearn import datasets
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.svm import SVC
from sklearn.metrics import classification_report
import optuna
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from skopt import BayesSearchCV

iris = datasets.load_iris()
X = iris.data
y = iris.target

2. Split the dataset into training and testing sets:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

3. Define the objective function for Optuna:

def optuna_objective(trial):
    C = trial.suggest_loguniform('C', 1e-2, 1e2)
    kernel = trial.suggest_categorical('kernel', ['linear', 'rbf'])
    gamma = trial.suggest_loguniform('gamma', 1e-4, 1e1)

    svm = SVC(C=C, kernel=kernel, gamma=gamma)
    score = np.mean(cross_val_score(svm, X_train, y_train, cv=5))
    return 1.0 - score

4. Perform hyperparameter optimization using Optuna:

optuna_study = optuna.create_study()
optuna_study.optimize(optuna_objective, n_trials=50)

5. Define the search space for Hyperopt:

hyperopt_space = {
    'C': hp.loguniform('C', np.log(1e-2), np.log(1e2)),
    'kernel': hp.choice('kernel', ['linear', 'rbf']),
    'gamma': hp.loguniform('gamma', np.log(1e-4), np.log(1e1))
}

6. Define the objective function for Hyperopt:

def hyperopt_objective(params):
    svm = SVC(**params)
    score = np.mean(cross_val_score(svm, X_train, y_train, cv=5))
    return {'loss': 1.0 - score, 'status': STATUS_OK}

7. Perform hyperparameter optimization using Hyperopt:

trials = Trials()
best_hyperopt = fmin(hyperopt_objective, hyperopt_space, algo=tpe.suggest, max_evals=50, trials=trials)

8. Perform hyperparameter optimization using Scikit-Optimize:

param_grid = {
    'C': (1e-2, 1e2, 'log-uniform'),
    'kernel': ['linear', 'rbf'],
    'gamma': (1e-4, 1e1, 'log-uniform')
}

bayes_search = BayesSearchCV(SVC(), param_grid, n_iter=50, cv=5, n_jobs=-1)
bayes_search.fit(X_train, y_train)

9. Compare the results of Optuna, Hyperopt, and Scikit-Optimize:

best_optuna_model = SVC(**optuna_study.best_params)
best_hyperopt_model = SVC(**best_hyperopt)
best_skopt_model = bayes_search.best_estimator_

best_optuna_model.fit(X_train, y_train)
best_hyperopt_model.fit(X_train, y_train)

y_pred_optuna = best_optuna_model.predict(X_test)
y_pred_hyperopt = best_hyperopt_model.predict(X_test)
y_pred_skopt = best_skopt_model.predict(X_test)

print("Optuna Results:")
print(classification_report(y_test, y_pred_optuna))

print("Hyperopt Results:")
print(classification_report(y_test, y_pred_hyperopt))

print("Scikit-Optimize Results:")
print(classification_report(y_test, y_pred_skopt))

Conclusion

In this blog post, we explored advanced and automated techniques for hyperparameter tuning using popular Python libraries: Optuna, Hyperopt, and Scikit-Optimize. We demonstrated their usage with a practical example on the Iris dataset and the SVM algorithm. These libraries provide powerful optimization algorithms that can efficiently search for the best hyperparameters in large search spaces, making them a valuable tool for machine learning practitioners. In the next blog post, we will explore more advanced techniques for hyperparameter optimization, such as genetic algorithms and population-based training.
Continue your learning by reading:
Leveraging Genetic Algorithms for Hyperparameter Tuning in Python