Reliability in Azure Batch

Azure Batch runs large-scale parallel and high-performance computing (HPC) batch jobs in Azure. Batch creates and manages pools of compute nodes, installs the applications that you want to run, and schedules jobs to run on the nodes.

When you use Azure, reliability is a shared responsibility. Microsoft provides a range of capabilities to support resiliency and recovery. You're responsible for understanding how those capabilities work within all of the services you use, and selecting the capabilities you need to meet your business objectives and uptime goals.

This article describes how to make Azure Batch resilient to transient faults, compute node failures, availability zone failures, region-wide failures, and service maintenance. It also describes production deployment recommendations, the Batch reliability architecture, backup and restore, and key information about the Azure Batch service-level agreement (SLA).

Production deployment recommendations

For production workloads, follow the best practices for Azure Batch, including these reliability recommendations:

Reliability architecture overview

This section describes some of the important aspects of how the service works that are most relevant from a reliability perspective. The section introduces the logical architecture, which includes some of the resources and features that you deploy and use. It also discusses the physical architecture, which provides details on how the service works under the covers.

Logical architecture

Azure Batch uses several resource types to run batch workloads. The configuration of these resources determines how your workload responds to failures.

All Batch processing and resources are associated with a Batch account. A Batch solution uses the following logical components:

• Batch account: A regional resource with which pools, jobs, tasks, and other Batch resources are associated. You create the account and select its region, pool allocation mode, and authentication settings.

• Pool: A collection of compute nodes on which tasks run. A pool's configuration defines its VM image and size, target node count, scaling behavior, networking, and zone placement. You configure each pool to provide the capacity, placement, and operating environment that your workload requires. The target node count is the requested capacity, but the number of allocated nodes can be lower because of quota limits, capacity constraints, or allocation failures.

• Jobs and tasks: A job manages a collection of tasks and is associated with a pool. A task runs an application or script on a compute node. You define jobs, tasks, constraints, dependencies, retry behavior, and the pool on which each job runs.

• Associated storage account: An Azure Storage account that you optionally link to a Batch account. Most Batch solutions use an associated storage account for application files, task resource files, input data, and output data.

  After you link a storage account to a Batch account, Batch refers to it as the autostorage account. An autostorage account is required for application packages and can also store task resource files.

  The reliability of a Batch solution depends on its storage resources. You're responsible for selecting a storage redundancy configuration that meets your recovery requirements when you create the storage account. Move task output and other durable data from compute nodes into storage. For information about Azure Storage reliability options, see Reliability in Azure Blob Storage.

For more information, see Batch service workflow and resources and Batch accounts and Azure Storage accounts.

Physical architecture

Azure Batch has a regional control plane that processes management operations against Batch accounts and other Batch resources, such as pools, jobs, and tasks. It's separate from the compute nodes that run tasks.

Azure Batch pools use Azure Virtual Machines and Virtual Machine Scale Sets. A compute node is a VM that processes part of a Batch workload. Each node belongs to a pool and executes tasks that the Batch service assigns to it. The Batch node agent runs on each node and provides the command-and-control interface between the node and the Batch service. Compute nodes and their local storage are ephemeral. When a node leaves a pool, changes to its operating system and files are lost.

The Batch account's pool allocation mode determines where Batch creates the compute and VM-related resources:

• Batch service pool allocation mode: Batch allocates the resources in Batch service-managed Azure subscriptions.
• User subscription pool allocation mode: Batch creates the resources directly in the subscription that contains the Batch account.

Responsibilities for the physical architecture are divided as follows:

• Microsoft: Microsoft operates the regional Batch control plane and service infrastructure. Batch provisions compute nodes according to the pool configuration and available capacity, schedules tasks, monitors node health, and attempts to recover nodes after infrastructure failures. Batch can requeue interrupted tasks for execution on another available node.

• You: You select the VM image and size, node types, target node count, scaling configuration, networking, and availability zone placement.

  You're also responsible for maintaining enough capacity to tolerate node failures, monitoring node and task states, keeping the guest operating system and node agent current, designing tasks to tolerate reexecution, and moving durable data off the compute nodes.

Batch node recovery doesn't guarantee that a workload continues without interruption. For more information, see Resilience to compute node failures.

Resilience to transient faults

Transient faults are short, intermittent failures in components. They occur frequently in a distributed environment like the cloud, and they're a normal part of operations. Transient faults correct themselves after a short period of time. It's important that your applications can handle transient faults, usually by retrying affected requests.

All cloud-hosted applications should follow the Azure transient fault handling guidance when they communicate with any cloud-hosted APIs, databases, and other components. For more information, see Recommendations for handling transient faults.

Within a Batch solution, transient faults can affect both the applications and scripts that run as tasks and any applications that call Batch APIs to manage Batch resources. Batch provides retry mechanisms at the task level. You're responsible for configuring task constraints, making tasks idempotent (safe to run more than once), and configuring retry behavior for calls to the Batch service and other dependencies.

Task retries and time limits

Configure maxTaskRetryCount to control how many times Batch retries a task after its process exits with a nonzero exit code:

• The default value, 0, means that Batch doesn't retry the task after a nonzero exit code.
• A positive integer sets a maximum number of retries after the initial attempt.
• A value of -1 allows unlimited retries. This value isn't recommended because a persistent failure can cause the task to run indefinitely.

Design every task to be idempotent so that retries don't cause data corruption or duplicate output. For long-running tasks, use checkpointing so that a retry can continue from saved progress. For more information, see Design for retries and reexecution.

Set the maxWallClockTime constraint on a task or job to terminate work that doesn't make progress within an acceptable period. A time limit also bounds the effect of repeated failures. Select retry counts and time limits that align with the workload's recovery objectives.

Transient faults in task dependencies

Applications and scripts that run as Batch tasks often call other components, such as Azure Storage, databases, or APIs. Batch doesn't manage the individual requests that task code sends to these dependencies. Implement transient fault handling in the task code, or use retry features provided by the dependency's SDK. Follow each dependency's guidance to identify retryable errors, select appropriate backoff and timeout settings, and honor retry instructions such as the Retry-After header. Retry only operations that are safe to repeat.

If a dependency failure causes the task process to exit with a nonzero exit code, Batch can retry the entire task according to maxTaskRetryCount. However, Batch doesn't retry an individual dependency request within a running task.

Batch API request retries

Your applications running outside the service can call Batch APIs to create and manage pools, jobs, tasks, compute nodes, and other Batch resources. Configure these clients with retry policies that span at least five minutes to handle transient Batch service or network failures.

Use built-in SDK retry support when it's available. For example, the Azure.Compute.Batch client library for .NET exposes retry behavior through Azure.Core retry options.

If you interact directly with the Batch service's APIs from your code, wait several seconds before retrying, use exponential backoff with randomization (jitter) to avoid synchronized retry requests, and honor the Retry-After header returned with throttling responses such as 429 Too Many Requests and 503 Service Unavailable.

A request timeout doesn't necessarily mean that Batch failed to process the request. After a timeout, retry the operation when it's safe to do so, or retrieve the resource state to determine whether the original operation succeeded. For more information, see Error handling and detection in Azure Batch.

Resilience to compute node failures

A compute node can become unavailable because of an infrastructure failure, operating system problem, or configuration problem. Batch attempts to recover nodes affected by infrastructure failures, but it can't automatically recover every configuration-related failure. For example, if an application package or container fails to install, you must correct the underlying problem and might need to remove and replace the affected node.

If a node failure interrupts a task, Batch can retry the task on the same node, if possible, up to an internal limit. If the task can't continue on that node, Batch can requeue it for execution on another available node. These service-driven recovery retries are independent of maxTaskRetryCount and can occur even when its value is 0.

Microsoft-managed node recovery process

Microsoft manages service-driven node recovery and task requeuing. If the Batch service stops receiving heartbeats from a node long enough to trigger recovery, Batch can perform the following actions:

1. Isolate the node: Batch treats the node as unhealthy, sets its state to unusable, and stops scheduling new tasks on it.
2. Identify affected tasks: Batch identifies tasks that were running on the node as candidates for rescheduling.
3. Requeue eligible tasks: If an affected task is still persisted in the running state, Batch can reset it to active so that the scheduler can assign it to another node. Batch doesn't reset or rerun a task already persisted in the completed state.
4. Recover the node: Batch attempts to restore the node to a healthy state or replace the underlying VM.

Recovery isn't instantaneous. A task can remain in the running state while Batch detects the node failure and attempts to reschedule the task. If no other compute node is available, the task can remain in that state until Batch can reschedule it. For more information, see Known limitations for Batch tasks.

If the original node resumes healthy heartbeats before replacement begins, Batch can return it to a stable state. If replacement has already begun, Batch can instead remove the original VM and provision a replacement.

Your responsibilities

Service-driven recovery doesn't guarantee uninterrupted task execution or sufficient remaining capacity. You're responsible for the following actions:

• Maintain sufficient capacity: If your workload must continue while a node is unavailable, provision enough nodes for the remaining nodes to run rescheduled tasks. Without another usable node, interrupted tasks must wait.

• Monitor usable capacity: Track aggregate node counts and states so that you can detect when a pool no longer has enough usable nodes for your workload. States such as unusable, startTaskFailed, unknown, and preempted indicate reduced capacity. For more information, see Monitor Batch solutions by counting tasks and nodes by state.

• Investigate problem nodes: If tasks repeatedly fail, are requeued, or remain in the running state after their assigned node becomes unusable or leaves the pool, use task metadata to determine whether the problems are associated with a specific node.

  Before you remove a problem node, upload its Batch node agent logs and preserve relevant task logs because removing the node deletes its local files. Node agent logs can help Microsoft Support diagnose Batch service or node problems, while task logs such as stdout and stderr can help you identify application and configuration problems. For more information, see Node agent log files.

• Protect task progress and data: Design tasks to tolerate interruption and reexecution. Move checkpoints, output, and other durable data off compute nodes because their local storage is ephemeral.

For more information, see Azure Batch pool and node errors and Error handling and detection in Azure Batch.

Resilience to availability zone failures

Availability zones are physically separate groups of datacenters within an Azure region. When one zone fails, services can fail over to one of the remaining zones.

Batch supports availability zones in two main ways:

• Control plane: The Batch control plane is zone resilient in many regions. In those regions, management operations against Batch accounts and other Batch resources can continue during a zone failure. Microsoft doesn't publish which regions provide this control-plane resiliency.

• Pools: You can configure a Batch pool to use availability zone placement. If a zone becomes unavailable, compute nodes in healthy zones remain available for task scheduling.

  For compute-node placement, this support model is similar to zone redundancy in other Azure services because a single pool can contain compute nodes in multiple availability zones. However, Batch distributes nodes across zones on a best-effort basis. It attempts to use all zones in the region but doesn't guarantee placement in every zone or an even distribution. You can't select specific zones or control the number of nodes placed in each zone.

The following diagram shows an example Batch account. The account has a single pool with its nodes distributed across three availability zones:

You're responsible for making other dependent resources, such as storage accounts, zone resilient. Review the reliability guides for each service. For more information, see Reliability guides for Azure services.

Requirements

• Region support: Create the pool in an Azure region that supports availability zones.

• VM SKU availability: The selected VM SKU must be available in more than one zone in the region. Use the Resource SKUs List API to check the locationInfo field for the VM SKU.

• Subscription access: For a user subscription pool allocation mode, the subscription must not have a zone offer restriction for the selected VM SKU. Check ResourceSkuRestrictions in the Resource SKUs List API response. If a restriction exists, submit a support request to request its removal.

• Inter-node communication: A pool can't use an availability zone placement policy when inter-node communication is enabled with an InfiniBand-enabled VM SKU.

Instance distribution across zones

When you enable availability zone placement, Batch attempts to allocate compute nodes across all availability zones in the selected region. This allocation is performed on a best-effort basis. Batch controls the distribution, and the number of nodes in each zone might not be equal.

Cost

Batch doesn't add a charge for enabling availability zone placement. You're billed for the compute, storage, networking, and other Azure resources used by the workload. If you provision extra nodes to tolerate the loss of a zone, those nodes incur their normal charges. For more information, see Azure Batch pricing.

Configure availability zone support

• Create a new pool with availability zone placement: To create a pool with availability zone placement enabled, see Create a Batch pool across availability zones.

• Change availability zone placement on an existing pool: You must select availability zone placement when you create the pool. You can't change an existing pool to use availability zones.

  You can create a replacement pool and update each of your jobs to use it by using the following steps:

  1. Disable the job: Use the Disable Job operation, and select how Batch handles tasks that are running when you submit the request:

     • Wait: Batch stops starting new tasks and allows running tasks to finish on the existing pool. The job remains in the disabling state until the running tasks finish.
     • Requeue: Batch interrupts running tasks and returns them to the queue. The tasks can run again after you enable the job.
     • Terminate: Batch interrupts running tasks and marks them as completed with failure information that indicates that they were terminated. The tasks don't run again after you enable the job.

     Tasks that are already in the active state, which means that they're queued but not running, remain queued with every option.

     Wait until the job reaches the disabled state before continuing. At that point, no tasks are running and Batch won't schedule new tasks.

  2. Edit the job's pool: Use the Update Job operation to set the job's poolInfo.poolId to the replacement pool. Batch only permits this property to change while the job is in the disabled state. Ensure that the replacement pool exists and has usable capacity before you enable the job. If the pool doesn't exist, Batch accepts the update, but it can't schedule tasks until you create the pool.

  3. Enable the job: Use the Enable Job operation. The job transitions through the enabling state to the active state. Batch then schedules queued tasks, including tasks returned to the queue by the Requeue option, on the replacement pool. Tasks that finished while Batch applied the Wait option and tasks completed by the Terminate option don't run again.

Capacity planning and management

Batch's normal node recovery process attempts to replace nodes that become unusable during a zone failure and restore the pool to its configured target node count. It doesn't increase that target. To retain enough capacity for the workload to continue when one zone is unavailable, consider over-provisioning the pool's capacity by using autoscale or manually:

• If you use autoscale: Reference the $UsableNodeCount service-defined variable in the pool's autoscale formula to detect when usable capacity falls below the baseline that your workload requires. $UsableNodeCount is a Batch autoscale variable for the pool whose formula is being evaluated, not an Azure Monitor metric. Azure Monitor publishes Batch node-count metrics at the Batch account scope, and those metrics don't have a poolId dimension.

• If you manage capacity manually: Use the List Pool Node Counts operation to retrieve dedicated and Spot node counts in each state for each pool. The operation doesn't return a single usable-node count. Assess task-capable states such as idle and running against unavailable or problem states such as unusable, unknown, offline, and startTaskFailed. The returned counts might not be current, so enumerate the pool's compute nodes when you require an exact count.

Batch distributes nodes across zones on a best-effort basis, so the actual distribution might not be equal. Don't assume that nodes are distributed equally across zones when you plan capacity. Validate that your workload can meet its recovery objectives with reduced pool capacity.

Behavior when all zones are healthy

This section describes what to expect when a Batch pool uses availability zone placement and all zones are operational.

• Cross-zone operation: Batch schedules tasks on available compute nodes throughout the pool, regardless of the zone that contains each node.

• Cross-zone data replication: Batch compute nodes are ephemeral and Batch doesn't replicate task files between zones. You're responsible for moving input, output, checkpoints, and diagnostic data that must survive node loss to durable storage. Configure the storage account redundancy to meet your availability requirements.

Behavior during a zone failure

This section describes what to expect when a Batch pool uses availability zone placement and one availability zone is unavailable.

• Detection and response: Microsoft monitors the Batch service and compute-node health. During a zone failure, Batch detects sustained heartbeat loss from affected nodes and starts its normal node recovery process. Nodes in healthy zones remain available, and Batch can continue to schedule tasks on them.

  Batch attempts to restore the pool to its configured target node count through its normal recovery process, but it doesn't increase that target. You're responsible for requesting additional capacity if the target doesn't provide enough capacity for the workload.

• Notification: Microsoft doesn't automatically notify you when a zone is down. However, you can use Azure Service Health to understand the overall health of the service, including any zone failures, and you can set up Service Health alerts to notify you of problems.

• Active tasks: Tasks running on affected nodes are interrupted. Batch might reschedule tasks on available nodes as part of its internal recovery behavior. Design tasks to tolerate retries and reexecution, as described in Resilience to transient faults and Resilience to compute node failures.

• Expected data loss: Files on affected compute nodes are unavailable during the outage and are lost if the nodes are reimaged or removed from the pool. Batch doesn't replicate these files across zones. Data persisted to durable storage remains available according to that storage resource's redundancy configuration.

• Expected downtime: Tasks on healthy nodes continue. Interrupted tasks experience downtime until Batch can reschedule them to usable nodes. If no usable node is available, a task might remain in the running state until Batch can reschedule it. Recovery time depends on available pool capacity and the queued workload.

  Management operations against Batch accounts and other Batch resources continue if the account is in a region where the control plane is zone resilient. In other regions, management operations might be unavailable until the affected zone recovers.

• Redistribution: After sustained heartbeat loss, Batch marks affected nodes as unusable and submits them for replacement through its normal node recovery process. Batch attempts to restore the pool to its configured target node count, but replacement capacity depends on quota and VM availability in the healthy zones.

  Jobs remain associated with the pool during this process. Batch continues to schedule tasks from active jobs on usable nodes in healthy zones. If replacement nodes are allocated and become usable, Batch can also schedule tasks on those nodes.

  This recovery process doesn't increase the configured target node count. If your workload requires capacity beyond that target, increase it manually or configure the pool's autoscale formula to do so.

Zone recovery

Batch manages the recovery process when a zone recovers after a failure. Batch follows the same process that's defined in Resilience to compute node failures.

Batch doesn't guarantee that nodes are rebalanced evenly across availability zones after recovery. Zone placement remains on a best-effort basis.

Test for zone failures

You can't simulate an actual Azure availability zone outage for testing. Instead, use a nonproduction pool to interrupt representative tasks and take enough compute nodes out of service to approximate the expected capacity loss. Consider using the following supported node operations:

• Reboot selected nodes.
• Disable task scheduling on selected nodes.
• Remove selected nodes from the pool.

When an operation allows you to choose how to handle running tasks, select the Requeue option so that Batch can schedule the interrupted tasks on the remaining nodes.

During the test, verify that Batch reschedules interrupted tasks and that reexecution doesn't produce duplicate or inconsistent output. Confirm that checkpoints and output remain available from durable storage, monitoring detects node loss and reduced capacity, and the workload can finish with the remaining capacity.

These operations validate task interruption and rescheduling, idempotency, checkpointing, and durable output handling. However, they don't reproduce every effect of a zone-wide failure.

Resilience to region-wide failures

A Batch account and its pools are scoped to one Azure region. If the region experiences a failure, your Batch jobs and pools might be affected.

Batch doesn't replicate accounts, pool configuration, jobs, tasks, application packages, or task data to another region, and it doesn't provide built-in cross-region failover. You're responsible for deploying and operating a multiregion Batch solution when your workload must tolerate a region-wide failure.

Custom multiregion solutions for resilience

To build a multiregion solution, deploy at least one Batch account in each region that your recovery design uses. Batch accounts operate independently across regions. You're responsible for selecting the regions, deploying and operating the resources in each region, and coordinating configuration, replication, failover, and failback.

You can design an active-active approach that submits work in multiple regions during normal operation, or an active-passive approach that keeps the secondary deployment ready for failover. Active-active designs can reduce recovery time because both deployments process work during normal operations, while active-passive designs can reduce steady-state compute costs.

The following list provides examples of your responsibilities when you design the solution:

• Deployment: Precreate the Batch accounts, pools or pool deployment definitions, storage accounts, networking resources, and other dependencies in each region. Verify that each selected region supports the Batch features, VM sizes, and dependent services that your workload requires. Keep each Batch account and the storage resources used by its tasks in the same region when possible to avoid unnecessary latency. Define Batch accounts, pools, and supporting resources by using infrastructure as code so that configuration can be reproduced consistently across regions.

• Quotas and capacity: Verify the applicable Batch account, subscription, and VM family quotas in every recovery region, and request increases before an outage when needed. A quota is a limit, not a capacity guarantee. VM availability varies by region, VM family, and time. For more information, see Batch account and core quotas.

• Data replication: Keep application packages, container images, input data, and reference data available in every recovery region. Batch doesn't replicate these items for you.

• Storage replication: Configure storage redundancy and cross-region data replication based on your recovery point objective. Verify that replicated data is accessible from each recovery region. For storage reliability options, see Reliability in Azure Blob Storage.

• Region targeting: If your applications work with your Batch account, design them so that they can select another Batch account and its associated storage resources without requiring a code deployment.

• Failover: Batch doesn't automatically move your workload to another region. You're responsible for detecting an outage, determining how the outage affects your workload, stopping submissions to the affected Batch account, directing new work to a Batch account in a healthy region, and deciding whether to resubmit interrupted jobs. Work or task output that you haven't replicated outside the affected region might be unavailable or lost. Your recovery time depends on your deployment model, available capacity, data replication, and automation.

• Region recovery: After service availability is restored in the affected region, validate the Batch account, pools, storage dependencies, and job state before directing work to that region. Reconcile duplicate or partially completed tasks before resubmission. Batch doesn't coordinate this process or reconcile state between regional accounts.

• Testing: Regularly test the complete regional recovery process in a nonproduction environment. Simulate one regional deployment becoming unavailable, direct submissions to another regional Batch account, validate access to application packages and input data, run representative jobs, collect output, and practice returning the recovered deployment to service. Measure the results against your recovery time objective and recovery point objective.

Backup and restore

Azure Batch isn't a durable store for workload input and output data. Microsoft maintains the Batch service, but you're responsible for preserving the configuration and data needed to recreate your workload:

• Batch configuration: Define Batch accounts, pools, jobs, and supporting resources by using infrastructure as code, such as Bicep, and version-controlled job definitions. These artifacts provide a repeatable way to recreate configuration in the same region or another region. Regularly test that the definitions can deploy a pool and run a representative job.

• Batch metadata: Batch retains job and task records until you delete the associated job or task, but these records aren't a backup. If you need to retain this history, preserve the required records outside Batch.

• Data on compute nodes: Compute nodes are ephemeral, and their file systems aren't a backup or durable store. Data on a node is lost when the node is reimaged or leaves a pool. Batch can also delete task files after their retention period expires. Persist required output and diagnostic logs to durable storage before a task completes. For long-running tasks, persist checkpoints during execution. For more information, see Persist job and task output and Monitor Azure Batch.

• Data in autostorage accounts: An associated autostorage account can hold application packages and task resource files. In addition to configuring redundancy that meets your recovery objectives, consider whether you need to protect this data from accidental deletion or corruption. For more information, see Backup and recovery in Azure Blob Storage.

Restore and recovery testing

To recover, deploy the Batch and supporting resource configuration, restore or reconnect the required input data and application artifacts, and resubmit work that didn't complete. Periodically test both resource reconstruction and data restoration in an isolated or nonproduction recovery environment. Use version-controlled definitions and backed-up data, run a representative job, verify the resulting output and access to dependencies, and measure the achieved recovery time and recovery point against your objectives. Verify that tasks are idempotent so that resubmitting uncertain work doesn't corrupt data or create duplicate output.

Resilience to service maintenance

Microsoft performs platform maintenance and coordinates infrastructure updates for Azure Batch and other services. You're responsible for configuring pool update behavior and designing tasks to tolerate interruption.

• Host maintenance: Azure uses maintenance techniques such as live migration when possible. If maintenance requires a compute node to restart or move, running tasks can be interrupted and Batch might reschedule them. Make tasks idempotent, and persist checkpoints and output to durable storage.

• Guest operating system: You're responsible for keeping the guest operating system and Batch node agent current. Operating system update behavior depends on the image and update configuration. With appropriate configuration, automatic OS image upgrades for Batch pools can update pool nodes without interrupting running tasks. Automatic OS image upgrades don't support every Batch operating system image.

• Node agent updates: Automatic OS image upgrades don't automatically update the Batch node agent. Batch node agents aren't automatically upgraded while a pool has compute nodes. To receive a newer node agent, resize the pool to zero nodes and scale it up again, or replace the pool. Before removing nodes, preserve any required task output and diagnostic logs. For more information, see Batch node agent updates.

Follow the production deployment recommendations and compute node failure guidance to help your workload continue or recover when maintenance interrupts tasks or makes compute nodes unavailable.

Service-level agreement

The service-level agreement (SLA) for Azure services describes the expected availability of each service and the conditions that your solution must meet to achieve that availability expectation. For more information, see SLAs for online services.

The Azure Batch availability SLA applies to operations that you perform against the Batch API. For detailed information about the types of operations that are included, review the SLA wording. Your pool size, zone distribution, and dependent services affect the end-to-end availability of the workload but don't change the terms of the Batch SLA.

Related content

• Azure Batch documentation
• Best practices for Azure Batch
• Reliability in Azure