ハイパーパラメータを手動で指定する

初期学習率の値やリーフ数など、最初に試すべきハイパーパラメータのセットが具体的に決まっている場合もあります。 また、Optunaがより良いハイパーパラメータセットを見つける前に、これらのセットを既に試している場合もあります。

Optunaではこのようなケースに対応するため、以下の2つのAPIを提供しています:

  1. これらのハイパーパラメータセットをOptunaに渡し、評価させる - enqueue_trial()

  2. これらのセットの結果を完了した Trial として追加する - add_trial()

最初のシナリオ:Optunaにハイパーパラメータを評価させる

このシナリオでは、既製のハイパーパラメータセットは持っているものの、まだ評価しておらず、 Optunaを使ってより良いハイパーパラメータセットを見つけたいとします。

Optunaには optuna.study.Study.enqueue_trial() という機能があり、 この機能を使えば、Optunaにこれらのハイパーパラメータセットを渡して評価させることができます。

このセクションでは、 LightGBM を使った このAPIの使用方法を説明します。

import lightgbm as lgb
import numpy as np
import sklearn.datasets
import sklearn.metrics
from sklearn.model_selection import train_test_split

import optuna

目的関数を定義します。

def objective(trial):
    data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)
    train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25)
    dtrain = lgb.Dataset(train_x, label=train_y)
    dvalid = lgb.Dataset(valid_x, label=valid_y)

    param = {
        "objective": "binary",
        "metric": "auc",
        "verbosity": -1,
        "boosting_type": "gbdt",
        "bagging_fraction": min(trial.suggest_float("bagging_fraction", 0.4, 1.0 + 1e-12), 1),
        "bagging_freq": trial.suggest_int("bagging_freq", 0, 7),
        "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
    }

    gbm = lgb.train(param, dtrain, valid_sets=[dvalid])

    preds = gbm.predict(valid_x)
    pred_labels = np.rint(preds)
    accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels)
    return accuracy

次に、ハイパーパラメータ最適化のための Study を作成します。

study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner())

ここでは、 "bagging_fraction" の値を大きくしたセットとデフォルト値のセットを Optunaに評価させます。

study.enqueue_trial(
    {
        "bagging_fraction": 1.0,
        "bagging_freq": 0,
        "min_child_samples": 20,
    }
)

study.enqueue_trial(
    {
        "bagging_fraction": 0.75,
        "bagging_freq": 5,
        "min_child_samples": 20,
    }
)

import logging
import sys

# Optunaの動作を確認できるように、stdoutのストリームハンドラを追加します。
optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout))
study.optimize(objective, n_trials=100, timeout=600)
Trial 0 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 1.0, 'bagging_freq': 0, 'min_child_samples': 20}. Best is trial 0 with value: 0.958041958041958.
Trial 1 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.75, 'bagging_freq': 5, 'min_child_samples': 20}. Best is trial 0 with value: 0.958041958041958.
Trial 2 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.5762105593893179, 'bagging_freq': 5, 'min_child_samples': 53}. Best is trial 0 with value: 0.958041958041958.
Trial 3 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8957677659544876, 'bagging_freq': 5, 'min_child_samples': 40}. Best is trial 3 with value: 0.965034965034965.
Trial 4 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7626113703208248, 'bagging_freq': 0, 'min_child_samples': 81}. Best is trial 3 with value: 0.965034965034965.
Trial 5 finished with value: 0.9230769230769231 and parameters: {'bagging_fraction': 0.7003956505812776, 'bagging_freq': 5, 'min_child_samples': 76}. Best is trial 3 with value: 0.965034965034965.
Trial 6 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.6224993464542131, 'bagging_freq': 2, 'min_child_samples': 28}. Best is trial 3 with value: 0.965034965034965.
Trial 7 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6406306796951904, 'bagging_freq': 3, 'min_child_samples': 76}. Best is trial 7 with value: 0.972027972027972.
Trial 8 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.5353684017624838, 'bagging_freq': 2, 'min_child_samples': 89}. Best is trial 7 with value: 0.972027972027972.
Trial 9 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.8205671493592026, 'bagging_freq': 6, 'min_child_samples': 85}. Best is trial 9 with value: 0.9790209790209791.
Trial 10 finished with value: 0.6153846153846154 and parameters: {'bagging_fraction': 0.40260289500535407, 'bagging_freq': 7, 'min_child_samples': 100}. Best is trial 9 with value: 0.9790209790209791.
Trial 11 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8365368023267711, 'bagging_freq': 7, 'min_child_samples': 66}. Best is trial 9 with value: 0.9790209790209791.
Trial 12 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8609193038158727, 'bagging_freq': 3, 'min_child_samples': 63}. Best is trial 9 with value: 0.9790209790209791.
Trial 13 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.6644698198680586, 'bagging_freq': 3, 'min_child_samples': 99}. Best is trial 9 with value: 0.9790209790209791.
Trial 14 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.4275891093300166, 'bagging_freq': 6, 'min_child_samples': 72}. Best is trial 9 with value: 0.9790209790209791.
Trial 15 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.5268303366029861, 'bagging_freq': 4, 'min_child_samples': 86}. Best is trial 9 with value: 0.9790209790209791.
Trial 16 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.9572865039672702, 'bagging_freq': 2, 'min_child_samples': 55}. Best is trial 9 with value: 0.9790209790209791.
Trial 17 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.7816967462241873, 'bagging_freq': 4, 'min_child_samples': 38}. Best is trial 9 with value: 0.9790209790209791.
Trial 18 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6702063480059465, 'bagging_freq': 1, 'min_child_samples': 91}. Best is trial 9 with value: 0.9790209790209791.
Trial 19 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8066352216245811, 'bagging_freq': 6, 'min_child_samples': 66}. Best is trial 9 with value: 0.9790209790209791.
Trial 20 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.7169075585381659, 'bagging_freq': 6, 'min_child_samples': 77}. Best is trial 9 with value: 0.9790209790209791.
Trial 21 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.8500596787562705, 'bagging_freq': 7, 'min_child_samples': 65}. Best is trial 21 with value: 0.986013986013986.
Trial 22 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8883315944856498, 'bagging_freq': 7, 'min_child_samples': 47}. Best is trial 21 with value: 0.986013986013986.
Trial 23 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.932548209465277, 'bagging_freq': 6, 'min_child_samples': 71}. Best is trial 21 with value: 0.986013986013986.
Trial 24 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8188318818908541, 'bagging_freq': 4, 'min_child_samples': 7}. Best is trial 21 with value: 0.986013986013986.
Trial 25 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6284881713681107, 'bagging_freq': 7, 'min_child_samples': 61}. Best is trial 21 with value: 0.986013986013986.
Trial 26 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7377550554831195, 'bagging_freq': 3, 'min_child_samples': 83}. Best is trial 21 with value: 0.986013986013986.
Trial 27 finished with value: 0.9300699300699301 and parameters: {'bagging_fraction': 0.8762727729511293, 'bagging_freq': 6, 'min_child_samples': 92}. Best is trial 21 with value: 0.986013986013986.
Trial 28 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.938890888939646, 'bagging_freq': 1, 'min_child_samples': 59}. Best is trial 21 with value: 0.986013986013986.
Trial 29 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.9861651922248336, 'bagging_freq': 7, 'min_child_samples': 73}. Best is trial 21 with value: 0.986013986013986.
Trial 30 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8167256662103061, 'bagging_freq': 4, 'min_child_samples': 48}. Best is trial 21 with value: 0.986013986013986.
Trial 31 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.8312618422301531, 'bagging_freq': 7, 'min_child_samples': 67}. Best is trial 21 with value: 0.986013986013986.
Trial 32 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.839243165058366, 'bagging_freq': 7, 'min_child_samples': 81}. Best is trial 21 with value: 0.986013986013986.
Trial 33 finished with value: 0.993006993006993 and parameters: {'bagging_fraction': 0.7789161361127772, 'bagging_freq': 6, 'min_child_samples': 80}. Best is trial 33 with value: 0.993006993006993.
Trial 34 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7855113104232999, 'bagging_freq': 5, 'min_child_samples': 94}. Best is trial 33 with value: 0.993006993006993.
Trial 35 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.9056563776195286, 'bagging_freq': 6, 'min_child_samples': 82}. Best is trial 33 with value: 0.993006993006993.
Trial 36 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.7643137227071812, 'bagging_freq': 6, 'min_child_samples': 80}. Best is trial 33 with value: 0.993006993006993.
Trial 37 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.84964030457924, 'bagging_freq': 5, 'min_child_samples': 85}. Best is trial 33 with value: 0.993006993006993.
Trial 38 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.720669666164601, 'bagging_freq': 7, 'min_child_samples': 57}. Best is trial 33 with value: 0.993006993006993.
Trial 39 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.9019984907596724, 'bagging_freq': 5, 'min_child_samples': 95}. Best is trial 33 with value: 0.993006993006993.
Trial 40 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.7503894470416554, 'bagging_freq': 6, 'min_child_samples': 77}. Best is trial 33 with value: 0.993006993006993.
Trial 41 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7954689069438796, 'bagging_freq': 5, 'min_child_samples': 88}. Best is trial 33 with value: 0.993006993006993.
Trial 42 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7805159916829466, 'bagging_freq': 5, 'min_child_samples': 95}. Best is trial 33 with value: 0.993006993006993.
Trial 43 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.8614215143492617, 'bagging_freq': 7, 'min_child_samples': 95}. Best is trial 33 with value: 0.993006993006993.
Trial 44 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6887235299980492, 'bagging_freq': 6, 'min_child_samples': 80}. Best is trial 33 with value: 0.993006993006993.
Trial 45 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.77386931127688, 'bagging_freq': 7, 'min_child_samples': 88}. Best is trial 33 with value: 0.993006993006993.
Trial 46 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8340103468900272, 'bagging_freq': 6, 'min_child_samples': 70}. Best is trial 33 with value: 0.993006993006993.
Trial 47 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.734629249028901, 'bagging_freq': 7, 'min_child_samples': 100}. Best is trial 33 with value: 0.993006993006993.
Trial 48 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7208374385177253, 'bagging_freq': 7, 'min_child_samples': 99}. Best is trial 33 with value: 0.993006993006993.
Trial 49 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.5751720546627385, 'bagging_freq': 7, 'min_child_samples': 51}. Best is trial 33 with value: 0.993006993006993.
Trial 50 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7453030133750455, 'bagging_freq': 7, 'min_child_samples': 85}. Best is trial 33 with value: 0.993006993006993.
Trial 51 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8004137728477251, 'bagging_freq': 6, 'min_child_samples': 91}. Best is trial 33 with value: 0.993006993006993.
Trial 52 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8608037868273216, 'bagging_freq': 6, 'min_child_samples': 97}. Best is trial 33 with value: 0.993006993006993.
Trial 53 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.6876333195369577, 'bagging_freq': 5, 'min_child_samples': 92}. Best is trial 33 with value: 0.993006993006993.
Trial 54 finished with value: 0.993006993006993 and parameters: {'bagging_fraction': 0.7997187544112262, 'bagging_freq': 7, 'min_child_samples': 100}. Best is trial 33 with value: 0.993006993006993.
Trial 55 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8371466899673545, 'bagging_freq': 7, 'min_child_samples': 99}. Best is trial 33 with value: 0.993006993006993.
Trial 56 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.8835673913723108, 'bagging_freq': 7, 'min_child_samples': 74}. Best is trial 33 with value: 0.993006993006993.
Trial 57 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8119993475114536, 'bagging_freq': 7, 'min_child_samples': 37}. Best is trial 33 with value: 0.993006993006993.
Trial 58 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.9175779274700631, 'bagging_freq': 6, 'min_child_samples': 68}. Best is trial 33 with value: 0.993006993006993.
Trial 59 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.6631820832877102, 'bagging_freq': 7, 'min_child_samples': 88}. Best is trial 33 with value: 0.993006993006993.
Trial 60 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.6015365045692954, 'bagging_freq': 6, 'min_child_samples': 88}. Best is trial 33 with value: 0.993006993006993.
Trial 61 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.6588720960037724, 'bagging_freq': 7, 'min_child_samples': 79}. Best is trial 33 with value: 0.993006993006993.
Trial 62 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.7574875368059412, 'bagging_freq': 7, 'min_child_samples': 84}. Best is trial 33 with value: 0.993006993006993.
Trial 63 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7186283684028795, 'bagging_freq': 7, 'min_child_samples': 90}. Best is trial 33 with value: 0.993006993006993.
Trial 64 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7355334859630993, 'bagging_freq': 7, 'min_child_samples': 100}. Best is trial 33 with value: 0.993006993006993.
Trial 65 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.4530094507680036, 'bagging_freq': 6, 'min_child_samples': 76}. Best is trial 33 with value: 0.993006993006993.
Trial 66 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7035195190185258, 'bagging_freq': 7, 'min_child_samples': 87}. Best is trial 33 with value: 0.993006993006993.
Trial 67 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.6484498104344185, 'bagging_freq': 7, 'min_child_samples': 82}. Best is trial 33 with value: 0.993006993006993.
Trial 68 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.7960481611288496, 'bagging_freq': 6, 'min_child_samples': 65}. Best is trial 33 with value: 0.993006993006993.
Trial 69 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8654740381455157, 'bagging_freq': 0, 'min_child_samples': 11}. Best is trial 33 with value: 0.993006993006993.
Trial 70 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8177942380001669, 'bagging_freq': 7, 'min_child_samples': 97}. Best is trial 33 with value: 0.993006993006993.
Trial 71 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7853638892655148, 'bagging_freq': 4, 'min_child_samples': 93}. Best is trial 33 with value: 0.993006993006993.
Trial 72 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.7679156532475977, 'bagging_freq': 6, 'min_child_samples': 94}. Best is trial 33 with value: 0.993006993006993.
Trial 73 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8369562921977329, 'bagging_freq': 5, 'min_child_samples': 96}. Best is trial 33 with value: 0.993006993006993.
Trial 74 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6767439390732843, 'bagging_freq': 6, 'min_child_samples': 86}. Best is trial 33 with value: 0.993006993006993.
Trial 75 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8481801526286147, 'bagging_freq': 7, 'min_child_samples': 91}. Best is trial 33 with value: 0.993006993006993.
Trial 76 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.787473839359229, 'bagging_freq': 6, 'min_child_samples': 83}. Best is trial 33 with value: 0.993006993006993.
Trial 77 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7354391610890638, 'bagging_freq': 7, 'min_child_samples': 80}. Best is trial 33 with value: 0.993006993006993.
Trial 78 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8071962240888538, 'bagging_freq': 5, 'min_child_samples': 89}. Best is trial 33 with value: 0.993006993006993.
Trial 79 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8245703438547684, 'bagging_freq': 7, 'min_child_samples': 97}. Best is trial 33 with value: 0.993006993006993.
Trial 80 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.848521442473186, 'bagging_freq': 6, 'min_child_samples': 63}. Best is trial 33 with value: 0.993006993006993.
Trial 81 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7821601738026013, 'bagging_freq': 4, 'min_child_samples': 94}. Best is trial 33 with value: 0.993006993006993.
Trial 82 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7764698182512582, 'bagging_freq': 5, 'min_child_samples': 97}. Best is trial 33 with value: 0.993006993006993.
Trial 83 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.7567265581625382, 'bagging_freq': 4, 'min_child_samples': 100}. Best is trial 33 with value: 0.993006993006993.
Trial 84 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7048569321143909, 'bagging_freq': 5, 'min_child_samples': 93}. Best is trial 33 with value: 0.993006993006993.
Trial 85 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.801438795358525, 'bagging_freq': 6, 'min_child_samples': 78}. Best is trial 33 with value: 0.993006993006993.
Trial 86 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7702924425155302, 'bagging_freq': 7, 'min_child_samples': 90}. Best is trial 33 with value: 0.993006993006993.
Trial 87 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.626655422557457, 'bagging_freq': 3, 'min_child_samples': 86}. Best is trial 33 with value: 0.993006993006993.
Trial 88 finished with value: 0.9300699300699301 and parameters: {'bagging_fraction': 0.6077041210141325, 'bagging_freq': 3, 'min_child_samples': 74}. Best is trial 33 with value: 0.993006993006993.
Trial 89 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.6329245998443328, 'bagging_freq': 2, 'min_child_samples': 85}. Best is trial 33 with value: 0.993006993006993.
Trial 90 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.5807737592643109, 'bagging_freq': 3, 'min_child_samples': 70}. Best is trial 33 with value: 0.993006993006993.
Trial 91 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8232862724845683, 'bagging_freq': 2, 'min_child_samples': 87}. Best is trial 33 with value: 0.993006993006993.
Trial 92 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6187252498532209, 'bagging_freq': 3, 'min_child_samples': 82}. Best is trial 33 with value: 0.993006993006993.
Trial 93 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.7336537983788823, 'bagging_freq': 4, 'min_child_samples': 94}. Best is trial 33 with value: 0.993006993006993.
Trial 94 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.8724560244878429, 'bagging_freq': 7, 'min_child_samples': 98}. Best is trial 33 with value: 0.993006993006993.
Trial 95 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8733356400550585, 'bagging_freq': 7, 'min_child_samples': 98}. Best is trial 33 with value: 0.993006993006993.
Trial 96 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8846697513762853, 'bagging_freq': 7, 'min_child_samples': 92}. Best is trial 33 with value: 0.993006993006993.
Trial 97 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.9141055420585937, 'bagging_freq': 7, 'min_child_samples': 43}. Best is trial 33 with value: 0.993006993006993.
Trial 98 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.5389254490707451, 'bagging_freq': 7, 'min_child_samples': 100}. Best is trial 33 with value: 0.993006993006993.
Trial 99 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8946764356623897, 'bagging_freq': 7, 'min_child_samples': 89}. Best is trial 33 with value: 0.993006993006993.

第2のシナリオ: 既に評価済みのハイパーパラメータをOptunaに活用する

このシナリオでは、事前に設定したハイパーパラメータセットがあり、 既に評価済みだが結果が思わしくないため、Optunaの活用を検討する場合を考えます。

Optunaには optuna.study.Study.add_trial() という機能があり、 これらの結果をOptunaに登録することで、Optunaがそれらを考慮したハイパーパラメータのサンプリングを行います。

このセクションでは、 objective は第1のシナリオと同じです。

study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner())
study.add_trial(
    optuna.trial.create_trial(
        params={
            "bagging_fraction": 1.0,
            "bagging_freq": 0,
            "min_child_samples": 20,
        },
        distributions={
            "bagging_fraction": optuna.distributions.FloatDistribution(0.4, 1.0 + 1e-12),
            "bagging_freq": optuna.distributions.IntDistribution(0, 7),
            "min_child_samples": optuna.distributions.IntDistribution(5, 100),
        },
        value=0.94,
    )
)
study.add_trial(
    optuna.trial.create_trial(
        params={
            "bagging_fraction": 0.75,
            "bagging_freq": 5,
            "min_child_samples": 20,
        },
        distributions={
            "bagging_fraction": optuna.distributions.FloatDistribution(0.4, 1.0 + 1e-12),
            "bagging_freq": optuna.distributions.IntDistribution(0, 7),
            "min_child_samples": optuna.distributions.IntDistribution(5, 100),
        },
        value=0.95,
    )
)
study.optimize(objective, n_trials=100, timeout=600)
A new study created in memory with name: no-name-397000d6-f908-41fe-8c3b-9bb29c98bdf2
Trial 2 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8287794133716305, 'bagging_freq': 6, 'min_child_samples': 47}. Best is trial 2 with value: 0.965034965034965.
Trial 3 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.41973491602959784, 'bagging_freq': 1, 'min_child_samples': 18}. Best is trial 3 with value: 0.9790209790209791.
Trial 4 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.716192990541751, 'bagging_freq': 2, 'min_child_samples': 21}. Best is trial 3 with value: 0.9790209790209791.
Trial 5 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7962273598172719, 'bagging_freq': 2, 'min_child_samples': 72}. Best is trial 3 with value: 0.9790209790209791.
Trial 6 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7445120695688103, 'bagging_freq': 0, 'min_child_samples': 57}. Best is trial 3 with value: 0.9790209790209791.
Trial 7 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.6380391090577919, 'bagging_freq': 2, 'min_child_samples': 94}. Best is trial 3 with value: 0.9790209790209791.
Trial 8 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.4225265627762822, 'bagging_freq': 3, 'min_child_samples': 27}. Best is trial 3 with value: 0.9790209790209791.
Trial 9 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6731819227551485, 'bagging_freq': 4, 'min_child_samples': 32}. Best is trial 3 with value: 0.9790209790209791.
Trial 10 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.48792857815836566, 'bagging_freq': 7, 'min_child_samples': 6}. Best is trial 3 with value: 0.9790209790209791.
Trial 11 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.8999208723458911, 'bagging_freq': 7, 'min_child_samples': 47}. Best is trial 11 with value: 0.986013986013986.
Trial 12 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.9440474095876933, 'bagging_freq': 7, 'min_child_samples': 46}. Best is trial 11 with value: 0.986013986013986.
Trial 13 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.5678069728336548, 'bagging_freq': 5, 'min_child_samples': 71}. Best is trial 11 with value: 0.986013986013986.
Trial 14 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8891077517758941, 'bagging_freq': 1, 'min_child_samples': 7}. Best is trial 11 with value: 0.986013986013986.
Trial 15 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.5692950180303371, 'bagging_freq': 4, 'min_child_samples': 37}. Best is trial 11 with value: 0.986013986013986.
Trial 16 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.4050538802647866, 'bagging_freq': 3, 'min_child_samples': 57}. Best is trial 11 with value: 0.986013986013986.
Trial 17 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8607425559281152, 'bagging_freq': 1, 'min_child_samples': 72}. Best is trial 11 with value: 0.986013986013986.
Trial 18 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.5822087549368136, 'bagging_freq': 6, 'min_child_samples': 94}. Best is trial 11 with value: 0.986013986013986.
Trial 19 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.49180595072910216, 'bagging_freq': 1, 'min_child_samples': 39}. Best is trial 11 with value: 0.986013986013986.
Trial 20 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.9556917359378647, 'bagging_freq': 5, 'min_child_samples': 84}. Best is trial 11 with value: 0.986013986013986.
Trial 21 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.4999779969001831, 'bagging_freq': 4, 'min_child_samples': 37}. Best is trial 11 with value: 0.986013986013986.
Trial 22 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.5566613541764371, 'bagging_freq': 6, 'min_child_samples': 48}. Best is trial 11 with value: 0.986013986013986.
Trial 23 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.6269107520656674, 'bagging_freq': 6, 'min_child_samples': 64}. Best is trial 11 with value: 0.986013986013986.
Trial 24 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.4552143929582165, 'bagging_freq': 7, 'min_child_samples': 49}. Best is trial 11 with value: 0.986013986013986.
Trial 25 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.5365422696998082, 'bagging_freq': 6, 'min_child_samples': 16}. Best is trial 11 with value: 0.986013986013986.
Trial 26 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.5344556437023709, 'bagging_freq': 7, 'min_child_samples': 29}. Best is trial 11 with value: 0.986013986013986.
Trial 27 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6305351346088414, 'bagging_freq': 7, 'min_child_samples': 43}. Best is trial 11 with value: 0.986013986013986.
Trial 28 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.5232169025136384, 'bagging_freq': 7, 'min_child_samples': 30}. Best is trial 11 with value: 0.986013986013986.
Trial 29 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.9688783726249413, 'bagging_freq': 6, 'min_child_samples': 60}. Best is trial 11 with value: 0.986013986013986.
Trial 30 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6687574165603, 'bagging_freq': 5, 'min_child_samples': 52}. Best is trial 11 with value: 0.986013986013986.
Trial 31 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.43133586170454896, 'bagging_freq': 0, 'min_child_samples': 16}. Best is trial 11 with value: 0.986013986013986.
Trial 32 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.45911341412441997, 'bagging_freq': 7, 'min_child_samples': 25}. Best is trial 11 with value: 0.986013986013986.
Trial 33 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.596688861362626, 'bagging_freq': 6, 'min_child_samples': 14}. Best is trial 11 with value: 0.986013986013986.
Trial 34 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.5339711369845572, 'bagging_freq': 7, 'min_child_samples': 34}. Best is trial 11 with value: 0.986013986013986.
Trial 35 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.7642289923342896, 'bagging_freq': 5, 'min_child_samples': 21}. Best is trial 11 with value: 0.986013986013986.
Trial 36 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.783679113358901, 'bagging_freq': 6, 'min_child_samples': 43}. Best is trial 11 with value: 0.986013986013986.
Trial 37 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.913188951282721, 'bagging_freq': 5, 'min_child_samples': 24}. Best is trial 11 with value: 0.986013986013986.
Trial 38 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8223613557078178, 'bagging_freq': 5, 'min_child_samples': 12}. Best is trial 11 with value: 0.986013986013986.
Trial 39 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7221676440417057, 'bagging_freq': 6, 'min_child_samples': 22}. Best is trial 11 with value: 0.986013986013986.
Trial 40 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.7648192473368567, 'bagging_freq': 6, 'min_child_samples': 53}. Best is trial 11 with value: 0.986013986013986.
Trial 41 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7507419667908646, 'bagging_freq': 6, 'min_child_samples': 65}. Best is trial 11 with value: 0.986013986013986.
Trial 42 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.7768610899633173, 'bagging_freq': 7, 'min_child_samples': 52}. Best is trial 11 with value: 0.986013986013986.
Trial 43 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7041596384154856, 'bagging_freq': 5, 'min_child_samples': 43}. Best is trial 11 with value: 0.986013986013986.
Trial 44 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8084280561283894, 'bagging_freq': 7, 'min_child_samples': 29}. Best is trial 11 with value: 0.986013986013986.
Trial 45 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.7438281556525111, 'bagging_freq': 6, 'min_child_samples': 57}. Best is trial 11 with value: 0.986013986013986.
Trial 46 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.8486691533769398, 'bagging_freq': 5, 'min_child_samples': 46}. Best is trial 11 with value: 0.986013986013986.
Trial 47 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.90796504364778, 'bagging_freq': 4, 'min_child_samples': 81}. Best is trial 11 with value: 0.986013986013986.
Trial 48 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.6616662603367605, 'bagging_freq': 7, 'min_child_samples': 35}. Best is trial 11 with value: 0.986013986013986.
Trial 49 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.992893575979243, 'bagging_freq': 6, 'min_child_samples': 9}. Best is trial 11 with value: 0.986013986013986.
Trial 50 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.8594160649978191, 'bagging_freq': 7, 'min_child_samples': 20}. Best is trial 11 with value: 0.986013986013986.
Trial 51 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6717045674384281, 'bagging_freq': 7, 'min_child_samples': 34}. Best is trial 11 with value: 0.986013986013986.
Trial 52 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.599511910394134, 'bagging_freq': 7, 'min_child_samples': 50}. Best is trial 11 with value: 0.986013986013986.
Trial 53 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7597816069712677, 'bagging_freq': 6, 'min_child_samples': 39}. Best is trial 11 with value: 0.986013986013986.
Trial 54 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6564594147554124, 'bagging_freq': 7, 'min_child_samples': 56}. Best is trial 11 with value: 0.986013986013986.
Trial 55 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7252795737641106, 'bagging_freq': 6, 'min_child_samples': 28}. Best is trial 11 with value: 0.986013986013986.
Trial 56 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.6883823402139331, 'bagging_freq': 3, 'min_child_samples': 46}. Best is trial 11 with value: 0.986013986013986.
Trial 57 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.5535464750755014, 'bagging_freq': 6, 'min_child_samples': 39}. Best is trial 11 with value: 0.986013986013986.
Trial 58 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.6107622456226458, 'bagging_freq': 7, 'min_child_samples': 35}. Best is trial 11 with value: 0.986013986013986.
Trial 59 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.6456640799443583, 'bagging_freq': 5, 'min_child_samples': 31}. Best is trial 11 with value: 0.986013986013986.
Trial 60 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8307334914469044, 'bagging_freq': 7, 'min_child_samples': 64}. Best is trial 11 with value: 0.986013986013986.
Trial 61 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.9914361551691683, 'bagging_freq': 6, 'min_child_samples': 10}. Best is trial 11 with value: 0.986013986013986.
Trial 62 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.9318791322756064, 'bagging_freq': 6, 'min_child_samples': 7}. Best is trial 11 with value: 0.986013986013986.
Trial 63 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.9874481595828899, 'bagging_freq': 7, 'min_child_samples': 18}. Best is trial 11 with value: 0.986013986013986.
Trial 64 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.51259239479931, 'bagging_freq': 4, 'min_child_samples': 25}. Best is trial 11 with value: 0.986013986013986.
Trial 65 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.9002963299857814, 'bagging_freq': 6, 'min_child_samples': 42}. Best is trial 11 with value: 0.986013986013986.
Trial 66 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.5607520407998945, 'bagging_freq': 6, 'min_child_samples': 54}. Best is trial 11 with value: 0.986013986013986.
Trial 67 finished with value: 0.9300699300699301 and parameters: {'bagging_fraction': 0.8742309687504811, 'bagging_freq': 7, 'min_child_samples': 49}. Best is trial 11 with value: 0.986013986013986.
Trial 68 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.4660687136037562, 'bagging_freq': 5, 'min_child_samples': 60}. Best is trial 11 with value: 0.986013986013986.
Trial 69 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.9304227710834421, 'bagging_freq': 5, 'min_child_samples': 100}. Best is trial 11 with value: 0.986013986013986.
Trial 70 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.7920459549025229, 'bagging_freq': 6, 'min_child_samples': 12}. Best is trial 11 with value: 0.986013986013986.
Trial 71 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8606743803692901, 'bagging_freq': 7, 'min_child_samples': 20}. Best is trial 11 with value: 0.986013986013986.
Trial 72 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8394645700859747, 'bagging_freq': 7, 'min_child_samples': 20}. Best is trial 11 with value: 0.986013986013986.
Trial 73 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.81003995307184, 'bagging_freq': 7, 'min_child_samples': 23}. Best is trial 11 with value: 0.986013986013986.
Trial 74 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.9697277666581252, 'bagging_freq': 7, 'min_child_samples': 5}. Best is trial 11 with value: 0.986013986013986.
Trial 75 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.8819066758730431, 'bagging_freq': 7, 'min_child_samples': 17}. Best is trial 11 with value: 0.986013986013986.
Trial 76 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.7703588516503306, 'bagging_freq': 6, 'min_child_samples': 27}. Best is trial 11 with value: 0.986013986013986.
Trial 77 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7355442574168279, 'bagging_freq': 6, 'min_child_samples': 10}. Best is trial 11 with value: 0.986013986013986.
Trial 78 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.6920139321965123, 'bagging_freq': 7, 'min_child_samples': 32}. Best is trial 11 with value: 0.986013986013986.
Trial 79 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.9552264989927621, 'bagging_freq': 2, 'min_child_samples': 14}. Best is trial 11 with value: 0.986013986013986.
Trial 80 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.576180130073076, 'bagging_freq': 7, 'min_child_samples': 36}. Best is trial 11 with value: 0.986013986013986.
Trial 81 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6946413752357792, 'bagging_freq': 3, 'min_child_samples': 48}. Best is trial 11 with value: 0.986013986013986.
Trial 82 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.682330732408349, 'bagging_freq': 3, 'min_child_samples': 44}. Best is trial 11 with value: 0.986013986013986.
Trial 83 finished with value: 0.9370629370629371 and parameters: {'bagging_fraction': 0.713564369587944, 'bagging_freq': 2, 'min_child_samples': 46}. Best is trial 11 with value: 0.986013986013986.
Trial 84 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.5408797503952553, 'bagging_freq': 4, 'min_child_samples': 41}. Best is trial 11 with value: 0.986013986013986.
Trial 85 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6141131555229542, 'bagging_freq': 3, 'min_child_samples': 51}. Best is trial 11 with value: 0.986013986013986.
Trial 86 finished with value: 0.951048951048951 and parameters: {'bagging_fraction': 0.47759127029493675, 'bagging_freq': 3, 'min_child_samples': 55}. Best is trial 11 with value: 0.986013986013986.
Trial 87 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.8114857247008684, 'bagging_freq': 6, 'min_child_samples': 61}. Best is trial 11 with value: 0.986013986013986.
Trial 88 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.8531345455001603, 'bagging_freq': 7, 'min_child_samples': 33}. Best is trial 11 with value: 0.986013986013986.
Trial 89 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.7642521686817061, 'bagging_freq': 4, 'min_child_samples': 40}. Best is trial 11 with value: 0.986013986013986.
Trial 90 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6476647670785332, 'bagging_freq': 5, 'min_child_samples': 27}. Best is trial 11 with value: 0.986013986013986.
Trial 91 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.8531386086827378, 'bagging_freq': 7, 'min_child_samples': 33}. Best is trial 11 with value: 0.986013986013986.
Trial 92 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.8733971998217991, 'bagging_freq': 7, 'min_child_samples': 37}. Best is trial 11 with value: 0.986013986013986.
Trial 93 finished with value: 0.958041958041958 and parameters: {'bagging_fraction': 0.9223819105053068, 'bagging_freq': 7, 'min_child_samples': 45}. Best is trial 11 with value: 0.986013986013986.
Trial 94 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.7941049013092384, 'bagging_freq': 6, 'min_child_samples': 30}. Best is trial 11 with value: 0.986013986013986.
Trial 95 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.8923721206674549, 'bagging_freq': 7, 'min_child_samples': 53}. Best is trial 11 with value: 0.986013986013986.
Trial 96 finished with value: 0.9440559440559441 and parameters: {'bagging_fraction': 0.5033203143437779, 'bagging_freq': 6, 'min_child_samples': 25}. Best is trial 11 with value: 0.986013986013986.
Trial 97 finished with value: 0.965034965034965 and parameters: {'bagging_fraction': 0.6606225789442622, 'bagging_freq': 7, 'min_child_samples': 21}. Best is trial 11 with value: 0.986013986013986.
Trial 98 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.8261233040082921, 'bagging_freq': 7, 'min_child_samples': 38}. Best is trial 11 with value: 0.986013986013986.
Trial 99 finished with value: 0.9790209790209791 and parameters: {'bagging_fraction': 0.7051491770047957, 'bagging_freq': 6, 'min_child_samples': 48}. Best is trial 11 with value: 0.986013986013986.
Trial 100 finished with value: 0.986013986013986 and parameters: {'bagging_fraction': 0.743776056303211, 'bagging_freq': 7, 'min_child_samples': 29}. Best is trial 11 with value: 0.986013986013986.
Trial 101 finished with value: 0.972027972027972 and parameters: {'bagging_fraction': 0.8452801152388681, 'bagging_freq': 7, 'min_child_samples': 39}. Best is trial 11 with value: 0.986013986013986.

Total running time of the script: (0 minutes 9.431 seconds)

Gallery generated by Sphinx-Gallery