Need a different partition key in Azure Cosmos DB? Pick the right approach

Originally posted on https://devblogs.microsoft.com/cosmosdb/need-a-different-partition-key-in-azure-cosmos-db-pick-the-right-approach/

Once you create a container, its partition key is fixed at creation, and you can’t change it in place. However, if your original key starts causing problems like cross-partition queries or hot partitions, you need to consider your options for changing it. This post explains the mechanics of changing a partition key, and the tradeoffs between the options.

First, think about the intent of your change:

• Re-partition the data. You want your items (and future writes) to live under a different key. This means moving to a new container and there are three ways, from least to most effort: Change partition key feature, container copy jobs, or a do-it-yourself migration.
• Query by a different key without moving. Your writes are fine where they are, and you only have read patterns that fan out across partitions. A Global Secondary Index adds a separate, read-only, automatically-synchronized container with a different partition key and its own data model, a projection of your items (all properties, or just a subset). The source container stays unchanged.

Note that every “change the partition key” option is actually a move: get your data into a new container that has the new key, then point your app at it. The exception is a Global Secondary Index, which adds a different key for read patterns while the source container stays unchanged.

Let’s walk through each path and its tradeoffs.

Re-partition the data

The three options below all land your data in a new container with the new key. But, they differ in how much Azure Cosmos DB does for you, whether you can keep writing to the source while the copy runs, and how much control you have over the cutover.

Option 1: the portal Change partition key feature (least effort)

The portal feature is the managed path. In the Azure portal, open Data Explorer, pick your container, go to Scale & Settings → Partition Keys, and select Change.

The portal creates or selects a destination container in the same database, then copies your data into it using container copy jobs (Option 2 covers those, since they’re the same engine). If you let the portal create the destination container for you, all configurations except the partition key and unique keys are replicated to the destination, so you only restate those two. You can run the copy online or offline (the next section explains what that means). In online mode you finish the job by selecting Complete, and once the copy is done you start using the new container with the new key and optionally delete the old one.

Option 2: container copy jobs directly (same engine, scriptable)

Container copy jobs are the engine behind the portal capability. You can also create and manage them yourself with Azure CLI (via the cosmosdb-preview extension), which is what you want for scripting, automation, or finer control. The workflow is: create the target container with the partition key (and throughput, unique keys, etc.) you want, create the copy job, monitor it, and cut over. When the source and destination are in different accounts, you first give the destination account’s identity read access to the source container. Same-account copies don’t need this step.

Container copy has two modes: online, where writes continue during the copy, and offline, where you stop writes before the job starts.

Online mode keeps writes flowing during the copy and ends with a short Complete-then-cutover, but every source write is charged double RUs while online copy is enabled, and it needs continuous backup, all-versions-and-deletes change feed, and the account-level capability switched on first. Offline skips those prerequisites at the cost of a write freeze on the source until you cut over. The Container copy jobs documentation walks through the prerequisites and cutover for both modes.

Both modes run on a best-effort basis (there’s no guaranteed completion time), and jobs run one at a time in the write region. Two things are worth planning for: a copy doesn’t carry TTL across, so an item that hasn’t expired restarts its countdown in the destination, and you’ll want the target provisioned at roughly twice the source throughput to keep up.

Option 3: roll your own migration (most control, most work)

If you need transforms during the move, your container exceeds the container-copy limits, or you just want full ownership of the process, create the new container yourself and move the data with your own pipeline. The large-scale migration guidance covers the building blocks:

• Bulk ingestion to load the new container fast: the .NET v3 SDK has it built in, and the bulk executor library covers apps still on .NET SDK 2.x.
• Azure Data Factory for smaller datasets (configure source and sink, no code), or the Spark connector if you already work in Spark.
• Change feed to catch writes that land on the source after your bulk load and replay them into the new container until you cut over. Its default latest-version mode skips deletes and TTL expirations, so use all-versions-and-deletes (which needs continuous backup) to handle delete operations.

Pre-create the destination with enough RU/s and turn off indexing during the load to hold down write cost. You own incremental sync, error handling, cutover, and any reshaping along the way.

Query by a different key, without moving (global secondary index)

If your only pain is cross-partition queries (reads that fan out because your filter property isn’t the partition key), you may not need to move at all. For that case, use global secondary indexes.

A global secondary index (GSI) is a separate, read-only container keyed by a different partition key, with its own data model. That model is a projection query over your items (SELECT * for all properties, or just the subset you query), and it stays automatically synchronized with the source container. You keep writing to your existing container with its existing key, and queries whose filter matches the GSI’s partition key (the specific patterns you built the GSI for) can become single-partition queries against the GSI instead of cross-partition queries on the source. A GSI helps the patterns it’s keyed for, not every cross-partition query.

GSIs fit when changing your existing partition key would be disruptive, and you have multiple query patterns that no single key can satisfy. A GSI needs continuous backups on the account, stays eventually consistent regardless of your consistency level, and runs on autoscale with its own storage and RU costs. Once one exists, replace and delete on the source cost 50–100% more RU (creates aren’t affected), and global secondary indexes cover the rest.

A GSI fixes reads. It doesn’t change where your writes land, so it won’t relieve a write-side hot partition on the source. If your problem is write distribution, or you need a different primary key for the data, you’re back to the data re-partitioning option above and you have to move.

To wrap up

If your writes need a different key, you end up moving the data, and the only question left is how much of the copy you hand to Azure Cosmos DB. If only your reads fan out, a GSI fixes that without changing where your writes land. The table below puts all four side by side.

Decision table

Learn more

• Partitioning in Azure Cosmos DB, Change the partition key (portal feature)
• Container copy jobs, Create and manage container copy jobs (CLI)
• Global secondary indexes, Hierarchical partition keys
• Migrate large amounts of data, Bulk executor library overview
• Change feed modes