Optimize agent instructions, skills, tools, and models (preview)

The agent optimizer improves four aspects of your hosted agent: instructions, skills, tools, and model selection. It automatically detects which of these targets to optimize from your agent's baseline configuration.

This article shows how to run an optimization, configure and monitor the run, and deploy the results. For what each target does and when it activates, see Optimization targets. To set up the baseline inputs, see Make your agent optimizer-ready. For a quick reference on what the optimizer changes, see What each target changes.

Prerequisites

• A Foundry project with a deployed hosted agent
• The azure.ai.agents CLI extension installed (see Quickstart: Optimize a hosted agent)
• A model deployed for evaluation (for example, gpt-4.1-mini) and an optimization model from the supported list (for example, gpt-5.1)
• Your agent is optimizer-ready (calls load_config())

Run an optimization

Start an optimization run with a single command:

azd ai agent optimize

The optimizer evaluates your baseline, generates candidates, evaluates them, and ranks the results. For the full evaluate-and-improve cycle, see How the agent optimizer works. Which targets run depends on your baseline configuration—instruction tuning, skill improvement, and tool optimization activate automatically when the matching baseline files are present. See Optimization targets.

To control the run with a config file, pass an eval.yaml that references your dataset, evaluators, and options:

azd ai agent optimize --config eval.yaml

For the full eval.yaml schema, see Configure the optimization run.

Target a specific agent

How the CLI resolves the agent depends on whether you run the command from an azd project:

The deployed agent name must match a hosted agent in the target Foundry project.

Optimize an existing agent without AZD project files

You can optimize an existing hosted agent without running azd ai agent init and without creating azure.yaml or a .azure environment directory. In this standalone flow, provide the Foundry project endpoint and deployed agent name explicitly.

1. Ensure the deployed agent is optimizer-ready. In a local working directory, create the instruction file, dataset, evaluators, and eval.yaml described in Configure the optimization run.

   Run the command from this working directory. Without an azd project, relative paths in eval.yaml resolve from the current working directory.

   For this standalone flow, omit agent.config. The CLI asks for the baseline instruction when you run the command:

   # eval.yaml
   agent:
     name: my-support-agent
     kind: hosted
     model: gpt-4.1-mini
   dataset:
     local_uri: ./eval.jsonl
   evaluators:
     - builtin.task_adherence
   options:
     eval_model: gpt-4.1-mini
     optimization_model: gpt-5.1
     max_candidates: 2
   

2. Authenticate:

   az login
   azd auth login
   

3. Copy the project endpoint from the Foundry project's Overview page. Use the project endpoint URL, not the Azure resource ID.

4. Save the endpoint in your user-level azd config so subsequent commands can resolve the same project from any directory:

   azd ai project set "<project-endpoint>"
   azd ai project show
   

   This step writes the default endpoint to ~/.azd/config.json. For the full resolution order and commands to inspect or clear the saved context, see Set the Foundry project context for azd commands.

5. Run the optimization with the deployed agent name:

   azd ai agent optimize --agent "<deployed-agent-name>" --config eval.yaml
   

   When prompted for the agent instruction, provide it inline or select a file such as .agent_configs/baseline/instructions.md.

   Note

   In the current preview, a standalone run doesn't expand agent.config from eval.yaml. Run the command interactively so you can provide the baseline instruction. Don't use --no-prompt for this flow. Loading file-based skill and tool baselines also requires an azd project.

   For a one-off command that shouldn't change your user-level config, pass --project-endpoint:

   azd ai agent optimize \
     --project-endpoint "<project-endpoint>" \
     --agent "<deployed-agent-name>" \
     --config eval.yaml
   

   You can also set the endpoint for the current shell:

   export FOUNDRY_PROJECT_ENDPOINT="<project-endpoint>"
   azd ai agent optimize --agent "<deployed-agent-name>" --config eval.yaml
   

6. Save the operation ID from the command output. Because this flow has no azd environment, the CLI doesn't persist the last operation ID locally. Pass the operation ID to follow-up commands:

   azd ai agent optimize status <operation-id> --watch
   
   azd ai agent optimize list
   
   azd ai agent optimize cancel <operation-id>
   

   These commands use the endpoint saved by azd ai project set. If you used the one-off --project-endpoint form instead, pass the flag again to each follow-up command.

Configure the optimization run

Configure optimization runs through an eval.yaml file that ties together your dataset, evaluators, and run options. The command azd ai agent eval generate writes this file for you, or you can create it by hand. The optimizer auto-detects eval.yaml in your project root, or you can pass it explicitly with --config eval.yaml.

# eval.yaml
name: my-optimization              # Optional label for the run
agent:
  name: my-agent                   # Deployed hosted agent name
  kind: hosted
  version: "1"                     # Agent version (optional)
  model: gpt-4.1-mini              # Baseline model deployment
  config: .agent_configs/baseline/metadata.yaml
dataset:
  local_uri: ./eval.jsonl          # A local JSONL file...
  # name: my-foundry-dataset       # ...OR a registered Foundry dataset
  # version: "1"
# validation_dataset:              # Optional held-out dataset
#   name: my-validation-dataset
#   version: "1"
evaluators:
  - builtin.task_adherence         # A built-in evaluator...
  # - name: my-custom-evaluator    # ...or a custom evaluator
  #   version: "1"
  #   local_uri: ./my_evaluator.json
options:
  eval_model: gpt-4.1-mini         # Scores responses
  optimization_model: gpt-5.1      # Generates candidates
  max_candidates: 4
  optimization_config:
    model_search_space:            # Optional: compare model deployments
      - gpt-4.1

Author the dataset and evaluators separately; see Create an evaluation dataset and evaluators. The following sections describe the run options.

Choose the eval and optimization models

The optimizer uses two models: an eval model that scores agent responses against criteria, and an optimization model that generates candidate configurations. Set them in eval.yaml or use CLI flags.

options:
  eval_model: gpt-4.1-mini
  optimization_model: gpt-5.1

azd ai agent optimize --eval-model gpt-4.1-mini --optimize-model gpt-5.1

Any chat-completion model deployed in your project works as the eval model. The optimization model must be from the supported list. For roles and supported models, see Models.

Set the number of candidates

The max_candidates option sets the expected number of candidate configurations for the run. The optimizer typically returns after it reaches that count, unless the run stops early because of an error or another stopping condition.

Higher values explore more variations but take longer. The optimizer learns from earlier candidates, so later candidates tend to score higher.

Evaluate multiple models

To compare model deployments in a single run, list them under optimization_config.model_search_space. The optimizer evaluates your agent with each model against the same dataset and ranks the results by score and token cost.

# eval.yaml
options:
  optimization_config:
    model_search_space:
      - gpt-4.1
      - gpt-4.1-mini
      - gpt-4o

Each model listed under model_search_space must be deployed in your Foundry project.

Model selection runs alongside the targets that activate automatically from your baseline. A single run can produce candidates that combine improved instructions, skills, and tool descriptions with different model options - you don't configure the combination yourself.

Monitor a running job

An optimization run is asynchronous. Use these commands when a job is long-running or you want to check its progress:

# Check status and stream progress
azd ai agent optimize status <operation-id> --watch

# List recent optimization jobs
azd ai agent optimize list

# Cancel a running job
azd ai agent optimize cancel <operation-id>

Capture the operation ID, portal URL, scores, and candidate IDs from the run output. You can also monitor the job in the Foundry portal using the URL shown when the run starts.

If you started the job without AZD project files, always pass the operation ID to status and cancel. The commands use the user-level endpoint saved by azd ai project set; otherwise, include --project-endpoint.

Interpret results

After optimization completes, review the results table. An asterisk (*) marks the best candidate. For the results table columns, scoring details, score-improvement thresholds, and the portal view, see Understand optimization results.

Deploy the winner

The recommended workflow is to apply the optimized config locally, then deploy:

# Apply the winning candidate locally
azd ai agent optimize apply --candidate <candidate-id>

# Deploy with the optimized config
azd deploy

This downloads the optimized configuration into .agent_configs/<candidate_id>/ in your project. On next deploy, your agent uses the improved instructions and tool descriptions.

Alternatively, you can deploy directly via the API (useful for quick A/B testing):

azd ai agent optimize deploy --candidate <candidate-id>

If all candidates score lower than the baseline, don't deploy any candidate. The baseline configuration remains active.

What each target changes

The optimizer automatically activates the targets that apply to your baseline. This section is a reference for what a run changes. Use the following table to anticipate what optimization does for your agent:

Your code stays the same across all targets because load_config() returns the optimized values automatically. Only the configuration the model sees changes.

Instructions

The optimizer rewrites the system prompt. Common improvements include:

• Adding explicit constraints that the original prompt implied but didn't state
• Restructuring instructions for clarity
• Adding output format specifications
• Strengthening safety and scope boundaries

For example, a minimal baseline prompt like You are a helpful assistant. might become:

You are a helpful coding assistant. Follow these guidelines:
1. Always include working code examples
2. Explain your reasoning step by step
3. If a question is outside your expertise, say so clearly
4. Use markdown formatting for code blocks
5. Handle edge cases in code examples

Skills

The optimizer refines each skill's description, body, and activation criteria while keeping the skill's purpose intact. The agent loads improved skills through load_config(), which appends them to the instruction set. Skills use the open Agent Skills format. For how your agent loads skills, see Make your agent optimizer-ready.

Tools

The optimizer refines your tools.json definitions. Common improvements include:

• Clearer function descriptions that help the model know when to call a tool
• More specific parameter descriptions that reduce inaccurate arguments
• Added constraints (enums, required fields) that prevent invalid inputs

Your tool implementation code stays the same. Only the definitions the model sees change.

Models

The optimizer ranks each candidate model by composite score and token cost, so you can choose the best quality-to-cost trade-off. To configure the candidates, see Evaluate multiple models.

Troubleshooting

Related content

• Agent optimizer overview
• Make your agent optimizer-ready
• Create an evaluation dataset and evaluators
• Quickstart: Optimize a hosted agent