Migrating a Deployment between namespaces without downtime

Why you can't change a namespace on the fly, which fields to clean up in an export, and what order to migrate objects in so the service doesn't go down.

Why this matters

An app outgrows dev and moves to staging, teams split a shared namespace into two, a cluster switches to a "namespace per team" scheme. In all these cases, the objects have to move, and clients shouldn't notice the move. The commands for this are simple; people get the order of steps wrong, or forget something that needs to move.

Why migration isn't mv

In Kubernetes, the namespace is stored directly in metadata.namespace on every object. You can't change it on the fly: the namespace is part of the resource's path in the API (/apis/apps/v1/namespaces/dev/deployments/web), and to the apiserver, deployment/web in dev and deployment/web in staging are two different objects.

So there's exactly one working approach: export, clean up, import into the new namespace, verify, delete the old one. In exactly that order, because reordering any of these steps opens a window of downtime.

Start with an inventory

kubectl get all is misleading: all isn't "every object in the namespace", it's a category of ten types: Pod, Service, ReplicationController, Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, CronJob, and HorizontalPodAutoscaler. ConfigMap, Secret, Ingress, PVC, ServiceAccount, Role, RoleBinding, NetworkPolicy, and PodDisruptionBudget aren't part of it.

The cluster itself can show what's in that category, and api-resources gives the full list of what can live in a namespace at all:

kubectl api-resources --categories=all
kubectl api-resources --namespaced=true -o name

Taking inventory of a specific namespace:

kubectl get all -n dev
kubectl get cm,secret,ingress,pvc,sa,role,rolebinding,netpol,pdb -n dev

Before you migrate, write down exactly what's moving. The manifests themselves give you a hint: envFrom, volumes, serviceAccountName, and imagePullSecrets in the Pod template reference objects that aren't in get all. A missed ConfigMap only turns up as a Pod error in the new namespace.

Extra fields in the export

kubectl get deploy web -n dev -o yaml returns everything, including fields the apiserver fills in itself:

• metadata.resourceVersion: the object's version, for optimistic locking;
• metadata.uid: a unique ID, generated by the apiserver;
• metadata.creationTimestamp: creation time;
• metadata.generation: a counter of spec edits;
• status: the current state, not part of the desired one;
• the kubectl.kubernetes.io/last-applied-configuration annotation: a copy of the last apply, with the old namespace inside its JSON;
• spec.clusterIP and spec.clusterIPs on a Service: an address from the cluster's pool;
• metadata.namespace: we rewrite this to the new one.

Most of these fields don't get in the way when you create a new object. kubectl apply and kubectl create reset resourceVersion on their own, and the apiserver sets its own uid, creationTimestamp, generation, and status. It's still worth cleaning them up: a manifest with no junk in it can go into git and be applied again later. A stale resourceVersion in a file applied on top of an object that's already changed causes a conflict:

Error from server (Conflict): error when applying patch:
...
Operation cannot be fulfilled on configmaps "web-config": the object has been modified; please apply your changes to the latest version and try again

Two fields always break the migration. The first is clusterIP: the address is already taken by the Service in the old namespace, and the apiserver refuses.

The Service "web" is invalid: spec.clusterIPs: Invalid value: ["10.96.193.8"]: failed to allocate IP 10.96.193.8: provided IP is already allocated

Both fields need to go, clusterIP and clusterIPs. If you only remove the second one, you get a different error: must be empty when clusterIP is not specified.

The second is metadata.namespace. If you don't rewrite it and apply the file without -n, the manifest goes right back to dev and updates the old object in place. With -n staging, kubectl refuses on its own:

error: the namespace from the provided object "dev" does not match the namespace "staging". You must pass '--namespace=dev' to perform this operation.

Cleaning up is easiest with yq (kubectl's --export flag has been gone for a long time):

kubectl get deploy web -n dev -o yaml \
  | yq 'del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp, .metadata.generation, .metadata.annotations."kubectl.kubernetes.io/last-applied-configuration", .status)' \
  | yq '.metadata.namespace = "staging"' \
  > web.yaml

This needs yq from mikefarah, version 4 (yq --version prints mikefarah). There's another program with the same name, a wrapper around jq, with different syntax, and it won't understand this expression.

The kubectl neat plugin does this cleanup in one command, but it doesn't ship with kubectl and isn't available everywhere.

There's no point exporting a Pod. A ReplicaSet creates the Pod, and a Deployment owns the ReplicaSet; the link is held through metadata.ownerReferences. You migrate the Deployment, and Pods appear in the new namespace on their own.

Migration order without downtime

The main rule: bring things up in the new namespace first, remove the old ones second. Otherwise the service is down in the window between deleting and applying.

dev                  staging
 cm, secret, sa ─1─>  cm, secret, sa
 deploy         ─2─>  deploy   (wait for rollout)
 svc            ─3─>  svc      (wait for addresses)
 clients        ─4─>  new DNS name
 delete dev     <─5─  after clients switch over

1. Create the target namespace: kubectl create ns staging.
2. Migrate the dependencies: ConfigMap, Secret, ServiceAccount. Without them Pods won't start, or won't be created at all.
3. Migrate the Deployment.
4. Wait for it to be ready: kubectl rollout status deploy/web -n staging --timeout=120s.
5. Apply the Service in the new namespace, without clusterIP and clusterIPs.
6. Check that the Pods passed readiness and the Service is returning addresses.
7. Switch clients to the new name and make sure they're calling it.
8. Only now delete the objects in the old namespace.

If traffic comes in through an Ingress, step 7 creates a new Ingress in staging and switches the route over. An Ingress can only reference a Service inside its own namespace, so pointing it at "someone else's Service" won't work: the whole Ingress object has to move. Until the switch, the old Deployment keeps serving requests.

The DNS name changes with the namespace

Inside the cluster, the Service is reachable as web.dev.svc.cluster.local. After the move, the name becomes web.staging.svc.cluster.local. Clients that call the full name with the old namespace, or the short web from inside dev itself, won't switch over on their own. Their configuration has to be updated, or they'll start getting resolution errors once the old Service is deleted.

Updating a client's configuration also takes time. If the address lives in a ConfigMap mounted as a volume, kubelet updates the file in the Pod not right away but at its next sync: per the docs, the delay can run up to kubelet's sync period (one minute by default) plus its cache delay. In our run, 4 seconds was enough, but you can't count on that. If the address comes from a ConfigMap through environment variables, the Pod won't see it at all until it restarts. You can delete the old objects once clients are already calling the new address, not right after you edit their ConfigMap.

Checking before you delete

After kubectl apply into the new namespace, make sure the Pods are actually ready, not just started:

kubectl rollout status deploy/web -n staging --timeout=120s
kubectl get pods -n staging -l app=web
kubectl get endpointslice -n staging -l kubernetes.io/service-name=web

The old v1 Endpoints object is deprecated as of 1.33. kubectl get endpoints still works, but it prints a warning:

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice

The point of this check: a Service only routes traffic to ready Pods. An empty address list in the EndpointSlice means readiness hasn't passed yet, and requests will get refused. You can delete the old objects once real IPs show up in the EndpointSlice and clients have switched over.

Where dependencies live

• PVCs don't move between namespaces. A PersistentVolume is bound to one specific PVC through spec.claimRef, and one PV can't bind to two PVCs at once. The standard path: set persistentVolumeReclaimPolicy: Retain on the PV, delete the old PVC, clear the PV's claimRef, and create a PVC with that PV's volumeName in the new namespace. The other path: back up and restore the data. With a stateless service whose only state is a ConfigMap, this doesn't come up at all.
• An imagePullSecret migrates by hand. The default ServiceAccount in the new namespace has no imagePullSecrets field. If the image comes from a private registry, you export the Secret itself, of type kubernetes.io/dockerconfigjson, and reference it from spec.template.spec.imagePullSecrets or from the ServiceAccount.
• The ServiceAccount that serviceAccountName references has to migrate before the Deployment. Without it, the ReplicaSet can't create a single Pod.
• A RoleBinding ties permissions to a subject inside a namespace. The new namespace has no permissions at all until the Role and RoleBinding land there too.
• A NetworkPolicy selects Pods by namespace and labels. Rules written for dev won't work in staging.
• There's no need to migrate ResourceQuota and LimitRange: those are namespace policy, not part of the application.

What commonly breaks

• Left spec.clusterIP on the Service. Apply fails with provided IP is already allocated. Remove both fields, clusterIP and clusterIPs.
• Didn't migrate the ConfigMap or Secret. The Pod in the new namespace goes into CreateContainerConfigError, and Events show Error: configmap "web-config" not found.
• Didn't migrate the ServiceAccount. Pods don't appear at all, kubectl get pods is empty. The cause is in the ReplicaSet's Events: FailedCreate with error looking up service account staging/web: serviceaccount "web" not found. After you create the ServiceAccount, the ReplicaSet retries on its own; in our run, Pods showed up after 25 seconds.
• Deleted the old Deployment before checking the new one. A downtime window opens up between the deletion and readiness. The order is reversed: EndpointSlice and switched clients first, then delete.
• Forgot about the DNS name with the old namespace in a neighboring service's configuration. Everything looks green, and calls to web.dev stop resolving.
• Deleting several types in one command in the form kubectl delete deploy web cm web-config svc web. kubectl reads it as a Deployment with the names web, cm, web-config, svc. List different types as type/name.

The full sequence as commands

# What's actually in the source namespace
kubectl get all -n dev
kubectl get cm,secret,sa,ingress -n dev

# Target namespace
kubectl create ns staging

# Clean up the export and swap the namespace
CLEAN='del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp, .metadata.generation, .metadata.annotations."kubectl.kubernetes.io/last-applied-configuration", .status) | .metadata.namespace = "staging"'

kubectl get cm web-config -n dev -o yaml | yq "$CLEAN" | kubectl apply -f -
kubectl get deploy web -n dev -o yaml | yq "$CLEAN" | kubectl apply -f -

# Wait for readiness
kubectl rollout status deploy/web -n staging --timeout=120s

# Service without clusterIP and clusterIPs
kubectl get svc web -n dev -o yaml \
  | yq "$CLEAN | del(.spec.clusterIP, .spec.clusterIPs)" \
  | kubectl apply -f -

# Check that the Service found ready Pods
kubectl get endpointslice -n staging -l kubernetes.io/service-name=web

# Switch clients to web.staging and wait until they go there

# Only now remove the old objects
kubectl delete -n dev deploy/web cm/web-config svc/web

In short

• The namespace is part of an object's identity, so migration means export and recreate, not rename.
• kubectl get all only shows the all category; check the full list with kubectl api-resources --namespaced=true.
• Clean out of the export: uid, resourceVersion, creationTimestamp, generation, status, last-applied, and always clusterIP and clusterIPs on a Service.
• Order: dependencies, Deployment, wait for rollout, Service, switch clients, and only then delete the old objects.
• The readiness criterion: a non-empty EndpointSlice on the new Service, and clients already calling the new name.
• PVC, imagePullSecret, ServiceAccount, RoleBinding, and NetworkPolicy migrate separately; a service's DNS name changes along with its namespace.

What you'll need for the tasks

• kubectl config current-context: to make sure you're working in the lab cluster.
• kubectl get <type> <name> -n <ns> -o yaml and yq to clean up and change the namespace, kubectl apply -f -.
• kubectl rollout status, kubectl get endpointslice -l kubernetes.io/service-name=<name>.
• kubectl describe rs, kubectl get events to find what didn't make it over.
• kubectl logs -f to watch the client, kubectl edit cm or kubectl patch cm to switch it over.
• kubectl delete -n <ns> type/name type/name.