Create and manage container copy jobs in Azure Cosmos DB (Preview)

  • Applies to: ✅ NoSQL

Copy jobs help create copies of containers in Azure Cosmos DB accounts.

This article describes how to create, monitor, and manage copy jobs using Azure CLI commands.

Prerequisites

  • You can use the portal Cloud Shell to run container copy commands. Alternately, you can run the commands locally. Make sure you have Azure CLI installed on your machine.
  • Currently, container copy is only supported in these regions. Make sure your account's write region belongs to this list.
  • Install the Azure Cosmos DB preview extension, which contains the container copy commands. 
    az extension add --name cosmosdb-preview

Set shell variables

First, set all of the variables that each individual script uses.

$sourceSubId = "<source-subscription-id>" 
$destinationSubId = "<destination-subscription-id>" 
$sourceAccountRG = "<source-resource-group-name>"
$destinationAccountRG = "<destination-resource-group-name>"
$sourceAccount = "<cosmos-source-account-name>"
$destinationAccount = "<cosmos-destination-account-name>"
$jobName = ""
$sourceDatabase = ""
$sourceContainer = ""
$destinationDatabase = ""
$destinationContainer = ""

Assign read permission

Note

This step isn't required if you're copying data within the same Azure Cosmos DB account.

While copying data from one account to another account, you need to grant the source account's identity read-write access to the destination account. Follow these steps to assign the required permissions to the source account's identity.

Using system-managed identity

  1. Set source subscription context. 
    az account set --subscription $sourceSubId
  2. Add system identity on source account. 
    $identityOutput = az cosmosdb identity assign -n $sourceAccount -g $sourceAccountRG
$principalId = ($identityOutput | ConvertFrom-Json).principalId
  3. Set default identity on source account. 
    az cosmosdb update -n $sourceAccount -g $sourceAccountRG --default-identity="SystemAssignedIdentity"
  4. Set destination subscription context. 
    az account set --subscription $destinationSubId
  5. Add role assignment on destination account. 
    # Read-write access role
$roleDefinitionId = "00000000-0000-0000-0000-000000000002" 
az cosmosdb sql role assignment create --account-name $destinationAccount --resource-group $destinationAccountRG --role-definition-id $roleDefinitionId --scope "/" --principal-id $principalId
  6. Reset source subscription context. 
    az account set --subscription $sourceSubId

Using user-assigned managed identity

  1. Assign user-assigned managed identity variable: 
    $userAssignedManagedIdentityResourceId = "<CompleteResourceIdOfUserAssignedManagedIdentity>"
  2. Set source subscription context. 
    az account set --subscription $sourceSubId
  3. Add user-assigned managed identity on source account. 
    $identityOutput = az cosmosdb identity assign -n $sourceAccount -g $sourceAccountRG --identities $userAssignedManagedIdentityResourceId
$principalId = ($identityOutput | ConvertFrom-Json).userAssignedIdentities.$userAssignedManagedIdentityResourceId.principalId
  4. Set default identity on source account. 
    az cosmosdb update -n $sourceAccount -g $sourceAccountRG --default-identity=UserAssignedIdentity=$userAssignedManagedIdentityResourceId
  5. Set destination subscription context. 
    az account set --subscription $destinationSubId
  6. Add role assignment on destination account. 
    $roleDefinitionId = "00000000-0000-0000-0000-000000000002"  # Read-write access role
az cosmosdb sql role assignment create --account-name $destinationAccount --resource-group $destinationAccountRG --role-definition-id $roleDefinitionId --scope "/" --principal-id $principalId
  7. Reset source subscription context. 
    az account set --subscription $sourceSubId

Create copy job

az cosmosdb copy create `
    --resource-group $sourceAccountRG `
    --job-name $jobName `
    --dest-account $destinationAccount `
    --src-account $sourceAccount `
    --dest-nosql database=$destinationDatabase container=$destinationContainer `
    --src-nosql database=$sourceDatabase container=$sourceContainer
    --mode Online

Monitor progress

Monitor the progress using this command:

az cosmosdb copy show `
    --resource-group $sourceAccountRG `
    --account-name $sourceAccount `
    --job-name $jobName
  • Total count represents the total number of changes (total document + any new changes) in the source container at any given time.
  • Processed count represents the total number of events coming from source container’s change feed that were processed by the copy job.

Complete copy job

  1. When the processed count becomes greater than or equal to the total count, turn off any updates on the source container and wait for 5-10 minutes to flush any remaining changes.
  2. Run the completion API to finish the copy job and free compute resources, this also writes the remaining changes (if any) to the destination container.
az cosmosdb copy complete `
    --resource-group $sourceAccountRG `
    --account-name $sourceAccount `
    --job-name $jobName
  1. Update the client applications to start using the new (destination) container if needed.

Set shell variables

First, set all of the variables that each individual script uses.

$sourceAccountRG = "<source-resource-group-name>"
$sourceAccount = "<cosmos-source-account-name>"
$destinationAccount = "<cosmos-destination-account-name>"
$jobName = ""
$sourceDatabase = ""
$sourceCollection = ""
$destinationDatabase = ""
$destinationCollection = ""

Create copy job

Create a job to copy a collection within an Azure Cosmos DB API for MongoDB account:

az cosmosdb copy create `
    --resource-group $sourceAccountRG `
    --job-name $jobName `
    --dest-account $destinationAccount `
    --src-account $sourceAccount `
    --dest-mongo database=$destinationDatabase collection=$destinationCollection `
    --src-mongo database=$sourceDatabase collection=$sourceCollection

Note

--job-name should be unique for each job within an account.

Set shell variables

First, set all of the variables that each individual script uses.

$sourceAccountRG = "<source-resource-group-name>"
$sourceAccount = "<cosmos-source-account-name>"
$destinationAccount = "<cosmos-destination-account-name>"
$jobName = ""
$sourceKeySpace = ""
$sourceTable = ""
$destinationKeySpace = ""
$destinationTable = ""

Create copy job

Create job to copy a table within an Azure Cosmos DB for Apache Cassandra account:

az cosmosdb copy create `
    --resource-group $sourceAccountRG `
    --job-name $jobName `
    --dest-account $destinationAccount `
    --src-account $sourceAccount `
    --dest-cassandra keyspace=$destinationKeySpace table=$destinationTable `
    --src-cassandra keyspace=$sourceKeySpace table=$sourceTable

Note

--job-name should be unique for each job within an account.

Managing copy jobs

Monitor the progress of a copy job

View the progress and status of a copy job:

az cosmosdb copy show `
    --resource-group $sourceAccountRG `
    --account-name $sourceAccount `
    --job-name $jobName

List all the copy jobs created in an account

To list all the copy jobs created in an account:

az cosmosdb copy list `
    --resource-group $sourceAccountRG `
    --account-name $sourceAccount

Pause a copy job

In order to pause an ongoing copy job, you can use the command:

az cosmosdb copy pause `
    --resource-group $sourceAccountRG `
    --account-name $sourceAccount `
    --job-name $jobName

Resume a copy job

In order to resume an ongoing copy job, you can use this command:

az cosmosdb copy resume `
    --resource-group $sourceAccountRG `
    --account-name $sourceAccount `
    --job-name $jobName

Cancel a copy job

In order to cancel an ongoing copy job, you can use this command:

az cosmosdb copy cancel `
    --resource-group $sourceAccountRG `
    --account-name $sourceAccount `
    --job-name $jobName

Get support for copy issues

For issues related to a copy job, raise a New Support Request from the Azure portal. Set the Problem Type as Data Migration and Problem subtype as Container copy.

Next step

  • For more information about container copy jobs, see Copy jobs.
  • Last updated on