Step-by-Step Guide to Hyperparameter Tuning with Azure ML SDK

You want to make your machine learning models better by using hyperparameter tuning with Azure ML SDK. This process helps you find the best settings for your model faster. It also means you do not have to do as much work by hand. You need an Azure account and a workspace. You also need to know some basic Python and machine learning. You will see clear steps, code examples, and helpful tips in this guide.

Key Takeaways

• Set up your Azure ML workspace. Get the compute resources you need. This helps your machine learning experiments run well.

• Prepare your data carefully. Write a training script that takes hyperparameters as input. This lets you tune your model.

• Make a clear search space with different hyperparameters. Pick a sampling method that works for you.

• Use early stop rules to end bad runs soon. This saves time and compute money.

• Run your sweep job and watch it closely. Check the results to find the best model. Register and deploy your model for real use.

Workspace and Compute

Create Workspace

You need a workspace to use Azure Machine Learning. The workspace is like a home for your projects. You can make a workspace in the Azure Portal. You can also make one with Python code.

Here is how you make a workspace with the Azure ML SDK:

from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

ml_client = MLClient(
    DefaultAzureCredential(),
    subscription_id="your-subscription-id",
    resource_group_name="your-resource-group",
    workspace_name="your-workspace-name"
)

Tip: Change "your-subscription-id", "your-resource-group", and "your-workspace-name" to your own info.

If you do not have a workspace, you can make one in the Azure Portal. Click "Create a resource" and search for "Machine Learning". Follow the steps you see on the screen.

Set Up Compute

You need compute resources to run your experiments. Azure ML lets you use different types of compute. You can use your own computer, a virtual machine, or a cluster.

To make a compute cluster with the SDK, use this code:

from azure.ai.ml.entities import AmlCompute

cpu_cluster = AmlCompute(
    name="cpu-cluster",
    size="STANDARD_DS11_V2",
    min_instances=0,
    max_instances=4
)
ml_client.begin_create_or_update(cpu_cluster)

Note: The cluster will get bigger when you send jobs. It will get smaller when not used. This helps you save money.

Install SDK

You need the Azure ML SDK to use these tools. You can install it with pip. Open your terminal or command prompt and type:

pip install azure-ai-ml

Check if it works by importing the package in Python:

import azure.ai.ml

If you do not see errors, you are ready! Now you have your workspace, compute, and SDK set up. You can start getting your data and training script ready.

Data and Training Script

Prepare Data

You must get your data ready before tuning. Good data helps your model learn well. You can use data from your computer or the cloud. Azure Blob Storage is one cloud option.

Here is how you upload data to your Azure ML workspace:

1. Save your data as a CSV file.

2. Use the Azure ML SDK to upload your data.

from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes

data_asset = Data(
    path="data/train.csv",
    type=AssetTypes.URI_FILE,
    description="Training data for hyperparameter tuning"
)
ml_client.data.create_or_update(data_asset)

Tip: Give your data files clear names. This helps you find them later.

You can also use built-in datasets from Azure ML. Try these if you want to practice first.

Write Training Script

You need a training script for Azure ML. The script tells Azure ML how to train your model. It should take hyperparameters as arguments. You can use Python’s argparse library for this.

Here is a simple example for a scikit-learn model:

import argparse
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

parser = argparse.ArgumentParser()
parser.add_argument('--n_estimators', type=int, default=100)
parser.add_argument('--max_depth', type=int, default=5)
args = parser.parse_args()

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

model = RandomForestClassifier(n_estimators=args.n_estimators, max_depth=args.max_depth)
model.fit(X_train, y_train)
accuracy = model.score(X_test, y_test)
print(f"Accuracy: {accuracy}")

Note: Always print the metric you want to improve. Azure ML will watch this value during tuning.

Now your data and training script are ready. Next, you will set up hyperparameter tuning.

Hyperparameter Tuning Setup

Define Search Space

You need to tell Azure ML which hyperparameters to tune. This step is important. It helps Azure ML know what values to try. There are different types of search spaces you can use:

• Continuous: These are numbers in a range. For example, learning rate can be any number from 0.01 to 0.1.

• Discrete: These are numbers with set choices. For example, you might use 50, 100, or 200 trees in a random forest.

• Categorical: These are picked from a list. For example, you can choose "gini" or "entropy" for a decision tree.

Here is how you set up a search space in Azure ML SDK v2:

from azure.ai.ml.sweep import Choice, Uniform, LogUniform

search_space = {
    "n_estimators": Choice(values=[50, 100, 200]),  # Discrete
    "max_depth": Choice(values=[5, 10, 15]),        # Discrete
    "criterion": Choice(values=["gini", "entropy"]),# Categorical
    "learning_rate": Uniform(min_value=0.01, max_value=0.1)  # Continuous
}

Tip: Use Choice for discrete and categorical values. Use Uniform or LogUniform for continuous values.

You can add more hyperparameters if you want. Make sure your training script can take these as arguments.

Choose Sampling Method

Azure ML lets you pick how it tries different hyperparameter values. This is called the sampling method. There are three main ways to do this:

Here is how you set the sampling method in your sweep job:

from azure.ai.ml.sweep import RandomSampling, GridSampling, BayesianSampling

# Example: Use random sampling
sampling = RandomSampling()

Note: Bayesian sampling can help you find good hyperparameters faster. It works well if you do not have a lot of compute resources.

You can use these methods with Azure ML sweep jobs. HyperDrive is the older tool for hyperparameter tuning. The new SDK v2 uses sweep jobs instead. You can also try AutoML if you want less manual setup.

Set Early Termination

You can save time and money by stopping bad runs early. This is called early termination. Azure ML has different early termination policies:

• Bandit Policy: Stops runs that do much worse than the best run so far.

• Median Stopping Policy: Stops runs if they do worse than the middle score of finished runs.

• Truncation Selection Policy: Stops a set percent of the worst runs at certain times.

Here is how you set an early termination policy:

from azure.ai.ml.sweep import BanditPolicy

early_termination_policy = BanditPolicy(evaluation_interval=2, slack_factor=0.2)

Tip: Early termination helps you use your compute resources better. Try different policies to see which one works best for you.

Now you can put together your search space, sampling method, and early termination policy. This setup helps you find the best model settings faster and with less work.

Run and Monitor Sweep Job

Configure Sweep

You need to set up your sweep job before you can start hyperparameter tuning. A sweep job tells Azure ML which training script to use, what compute to run on, and how to search for the best hyperparameters. You can use the Command class to define your training job, then wrap it with a SweepJob to add the search space, sampling method, and early termination policy.

Here is an example of how to configure a sweep job:

from azure.ai.ml import command, sweep

# Define the command job
job = command(
    code="./src",  # Path to your training script
    command="python train.py --n_estimators ${{inputs.n_estimators}} --max_depth ${{inputs.max_depth}}",
    inputs={
        "n_estimators": 100,
        "max_depth": 5
    },
    environment="AzureML-sklearn-1.0-ubuntu20.04-py38-cpu:1"
)

# Wrap with sweep job
sweep_job = sweep(
    job=job,
    sampling=sampling,
    search_space=search_space,
    early_termination=early_termination_policy,
    objective="accuracy",
    compute="cpu-cluster",
    max_total_trials=20,
    max_concurrent_trials=4
)

Tip: Set max_concurrent_trials to run several experiments at once. This speeds up your search.

Submit Job

You can now submit your sweep job to Azure ML. Use the ml_client.jobs.create_or_update() method. This will start the hyperparameter tuning process.

returned_job = ml_client.jobs.create_or_update(sweep_job)
print(f"Job submitted. ID: {returned_job.name}")

Azure ML will handle the rest. It will try different hyperparameter values and run your training script many times.

Monitor Progress

You can watch your sweep job in real time. Azure ML Studio gives you a dashboard to see each run, the best scores, and which hyperparameters work best. You can also check progress in your notebook:

from azure.ai.ml.entities import JobStatus

job_status = ml_client.jobs.get(returned_job.name).status
print(f"Current job status: {job_status}")

Azure ML lets you stop jobs early or adjust settings if needed. You can view logs, download outputs, and compare runs. This helps you learn which settings improve your model.

Note: Always monitor your jobs to make sure you use your compute resources well.

Results and Model Registration

Analyze Results

When your sweep job is done, you need to see which run did the best. Azure ML helps you do this. You can look at the Azure ML Studio dashboard to see all runs and their scores. You can also use code to find the best run.

Here is how you get the best run with the SDK:

best_run = ml_client.jobs.get(returned_job.name).outputs['best_trial']
print(f"Best run ID: {best_run}")

You should check the metrics for every run. Look at which hyperparameters gave the highest accuracy or your chosen metric. You can also download logs and outputs if you want to look deeper.

Tip: Use tables or charts in Azure ML Studio to compare runs. This helps you see patterns and pick the best settings.

Register Model

After you find the best run, you should register the model. Registering saves your model in the Azure ML workspace. This makes it easy to use again or share with others.

Here are the steps to register your model:

1. Download the model file from the best run.

2. Use the SDK to register it.

Here is an example:

from azure.ai.ml.entities import Model

model = Model(
    path="outputs/model.pkl",
    name="best_random_forest_model",
    description="Random Forest model from hyperparameter tuning"
)
ml_client.models.create_or_update(model)

Note: Give your model a clear name and description. This helps you find it later.

Deploy Model

Now you can deploy your registered model as a web service. This lets you send data and get predictions.

You can deploy your model in a few steps:

• Pick a compute target for deployment.

• Make an endpoint.

• Deploy the model.

Here is a simple example:

from azure.ai.ml.entities import ManagedOnlineEndpoint, ManagedOnlineDeployment

endpoint = ManagedOnlineEndpoint(name="rf-endpoint")
deployment = ManagedOnlineDeployment(
    name="rf-deployment",
    endpoint_name=endpoint.name,
    model=model
)
ml_client.online_endpoints.begin_create_or_update(endpoint)
ml_client.online_deployments.begin_create_or_update(deployment)

Tip: Try your endpoint with sample data to make sure it works before using it for real.

---

You now know how to do hyperparameter tuning with Azure ML SDK. This helps you find the best settings for your model fast. It also saves you time. You can use these steps in your own projects. Try using hyperparameter tuning in your machine learning work. You can also check out AutoML for more features. Azure ML has tools to help you run bigger experiments and get better results.

FAQ

How do you choose which hyperparameters to tune?

You should start with the most important hyperparameters for your model. For example, pick learning rate or number of trees. Try not to tune too many at once. Focus on the ones that affect accuracy the most.

Can you stop a sweep job early if you see good results?

Yes, you can stop a sweep job at any time in Azure ML Studio or with the SDK. This helps you save compute resources if you already found a good model.

What happens if your sweep job fails?

Azure ML shows error messages in the logs. You should check the logs to find the problem. Common issues include wrong data paths or missing packages. Fix the error and try again.

Do you need to change your training script for hyperparameter tuning?

Yes, your script must accept hyperparameters as arguments. Use Python’s argparse library. This lets Azure ML pass different values to your script during each run.

Can you use AutoML for hyperparameter tuning?

Yes, you can use AutoML in Azure ML. AutoML tries many models and hyperparameters for you. This saves time and helps you find the best model with less manual work.