Automatically adjust superparameters with Optuna and Pytorch

Tram Ho


What is metameter?

For those of you who have built deep learning models, it is certainly no stranger to this concept of metameter. In machine learning and deep learning, metameter is the parameters for which its value can be. This model controls the training process. It is not fixed, varies by problem, training process and can change during that training. In particular, it plays an important role in determining the model’s performance. Examples of metadata such as the number of units and layers in the neural network, or in a network trainer with metadata such as batchsize, learning rate schedule, optimizer, momentum, adam alpha, …

What is Optuna?

Optuna is a framework that automatically adjusts model parameters so that the best model can achieve the best performance.

Difficulty encountered when having to adjust parameters manually

Problem: you need to create an MLP model, how to choose the number of hidden layers, the number of units in a layer appropriately and accurately?

Choosing super parameters when doing the training model seems to be based on your feelings and experiences. And this doesn’t always get the best performance out of a model. And for those of you who do not have experience in super-parameter selection, this is really a challenge.

How to use the Optuna library in Pytorch


Define objective function

Trial object definition

Trial is an instance of a class that is implemented in Optuna.It is used to define the optimized super parameters.The value is taken within the scope you define, the information of the parameters is Past searches are retained and new values ​​will be based on that information. Trial is defined as follows:

name is the type of string and the value is the name of the parameter. Choicesis is a list form and it is represented as the choice of names of many types. Low and high are minimum and maximum values ​​of parameter. Example:

Defines study object

To search for metameter, you need to initialize an object study. This object saves your optimal results.

Then, using the optimize method:

In which, the first parameter is the Objective function, the second parameter is the number of tests. This optimization process is done in Study object and will perform to find the minimum value of the parameters in Objective function by optimization method. At the end of the above process, the optimal value will be saved and you can view it with the following command:

Experiment and comparison between model using Optuna and selecting super parameter rice on Iris set

Install Optuna

You can install Optuna using pip or conda

Check version

Super-parameter optimization

Experiment with the model to select the test parameters

I will perform testing the random forest model to perform classification on the Iris dataset and manually select the parameters as follows:

The result achieved by the above model is: 0.966. Fortunately, the results are quite good with n_estimators = 5 and max_depth = 3.

Experiment using Optuna to automatically adjust parameters

The superparameter of the above model is n_estimators and max_depth. Use trial objects to define them. Then create a study object to optimize this meta-parameter and finally get the best one.

After finishing the optimization process after 100 trials, we have the best test.

The following results :

As such, the accuracy will increase by more than 1%, and the metrics will change and not be the same as I manually selected above.

Test datasets on different algorithms

Until now, testing on different algorithms to get the most objective assessment takes a lot of coding and training, now Optuna has supported you with that. Let’s get it!

Let’s see the results:

After only a few lines of code using Optuna , the accuracy is 0.987%, and as above, the SVC model has better results than the Random forest.

Visualization with Optuna

Not only supports the automatic optimization of the super parameters, she also assists us in visualizing the state of the tests.

  1. Visualize history of study object

2. Visualize the precision of a super-parameter at each test

  1. Visualize accuracy surface for many parameters of the random forest model


If you are wondering or wasting time choosing parameters, give Optuna a try. If this article is interested by everyone, I will try to learn and write more articles about prunning model using Optuna. Thank you for watching, please leave 1 vote for me.


Share the news now

Source : Viblo