Upgrade, rotate, and remove

After Remote Operator is installed, its upkeep is rotating its connection key, upgrades and rollbacks, repairing a lost connection, and uninstalling. An installation is one Remote Operator container in one customer environment, registered with Alien.

Every change starts from that installation on the Remote Operator setup page (Manage installations in your project):

1. Open it.
2. Enter the same Kubernetes context and namespace.
3. Select the owning release type you installed with.

The page generates guarded commands that check the target before they change anything. Run those, not a bare helm or aws command. The Kubernetes commands stop unless the release is deployed, so check it first:

helm history operator-<installation-id> --namespace checkout --kube-context customer-1-staging

An installation keeps one identity for its whole life, and upgrades, rotations, and repairs keep all of it:

• its Alien registration
• its encryption key
• its identity storage: a PersistentVolumeClaim on Kubernetes, EFS on ECS

Never delete the identity volume, the credentials Secret, or the ECS registration secret to fix a problem.

Rotate the connection keyCopy Anchor Link

Each installation authenticates to Alien with its own deployment-scoped connection key. Rotation replaces that key and nothing else.

• Only workspace administrators can prepare, cancel, or download a rotation.
• Rotate on its own. Don't combine it with an image, operation, permission, or CRD change.
• The downloaded patch or JSON file holds a live key. Keep it outside version control and shared folders, chmod 600 it, and delete it when the rotation completes.

Each rotation has a revision number. The Operator ignores credentials older than the revision it has.

KubernetesCopy Anchor Link

For a setup-owned Secret, the command:

1. Checks that the release is deployed.
2. Checks that the Secret holds exactly collector-token, encryption-key, and sync-token, and that the encryption key's SHA-256 matches remoteOperator.existingSecret.encryptionKeySha256.
3. Patches only sync-token.
4. Runs helm upgrade --reuse-values --set remoteOperator.syncTokenRevision=<revision> --atomic --wait --timeout 5m, so the Operator restarts with the new key.

For a Terraform-owned Secret, the command:

1. Writes remote_operator_sync_token and remote_operator_sync_token_revision to .alien-remote-operator.auto.tfvars.json.
2. Runs terraform plan.
3. Applies it only if the plan changes nothing but kubernetes_secret_v1.remote_operator_credentials and helm_release.runtime.

Run it in the module and workspace that installed the release. In a Git checkout, the tfvars file must be ignored by Git.

When the Operator reports the new key, the status changes to Replacement active. Run a read-only diagnostic to confirm.

Amazon ECSCopy Anchor Link

The command checks the revision, expiry, AWS account, and the stack outputs AccountId, Region, EnvironmentName, and ClusterArn. Then it:

• writes a new version of the registration secret with the new syncToken and the same encryptionKey
• runs a CloudFormation change set that changes only RegistrationSecretVersionId
• waits for the ECS service to be stable

If another stack update is running, the command stops. Wait for it, then rerun the same command.

If a rotation is interruptedCopy Anchor Link

• If the apply command stopped after it changed the Secret or secret version, run the same command again. Don't prepare another revision.
• To stop, select Cancel pending rotation, download the rollback patch (Kubernetes) or rollback value (ECS), and run the apply command with that file.

When permissions changeCopy Anchor Link

When the enabled operations need different permissions than the installed ones, setup and the installation page show Permissions need an update, with the added and removed AWS, Google Cloud, and Kubernetes rules. Existing installations keep their current access until you apply the update.

• On Kubernetes, download the regenerated chart and run the upgrade. If the Operator has AWS or Google Cloud access, also apply the updated cloud access configuration.
• On ECS, regenerate and review the CloudFormation template, then run Apply reviewed update. The stack replaces the task role's grants.

Then select I re-applied this setup. Stop new operations while you change permissions, and let running ones finish.

Upgrade a dedicated releaseCopy Anchor Link

The commands are under Upgrade or restore this operator once the installation has registered.

1. Rename the current remote-operator folder to remote-operator-previous. The restore command needs it.
2. Select the operations you want and wait for the new Operator image to build.
3. Download every chart file into a new remote-operator folder, and review the image and Role changes with the cluster administrator.
4. Run Upgrade this operator. It stops unless the release is deployed, then runs helm upgrade ./remote-operator --reset-values with the release's current values and --atomic --wait --timeout 5m.

Restore previous operator chart upgrades back to remote-operator-previous with the current values, so a key rotated since then stays in place. First restore the operation selection you used with that chart. The command refuses a saved chart without values.schema.json.

Releases installed from the v0.2 chart kept credentials in Helm values. Before their first upgrade, run Migrate dedicated operator from v0.2 to v0.3 once with the v0.3 chart. It copies the live credentials into the setup-owned Secret and leaves only sanitized history.

Upgrade a product releaseCopy Anchor Link

When Alien builds your product chart, the Operator is part of that chart, so every upgrade, restore, and removal is an upgrade of the whole product release. Helm runs its hooks and may reconcile application workloads, so plan a maintenance window.

The chart carries the annotation alien.dev/remote-operator-lifecycle: "v2", and the generated commands check it with helm show chart. The chart refuses to enable the Operator on the first helm install: install the product with remoteOperator.enabled=false, then run Enable operator in the deployed product release.

Record the exact chart reference (repository-qualified name or OCI URL), the chart version, and the release name. Rotation and restore need the reference, and helm history doesn't keep it.

• Upgrade product release runs helm upgrade <release> <chart> --version <version> --reuse-values --atomic --wait --timeout 5m.
• Restore previous product version runs the same upgrade with the previous version. Restore that version's operation selection first.

The chart expects Helm's Secret storage backend. With HELM_DRIVER=configmap, also set remoteOperator.helmHistoryBackend=configmap and runtime.cleanup.onUninstall.helmHistoryBackend=configmap. The SQL and memory backends are rejected.

Remove only the OperatorCopy Anchor Link

Run Remove the Remote Operator from this product release from setup. It upgrades with remoteOperator.enabled=false and remoteOperator.confirmRemoval=<release>; the chart refuses removal unless confirmRemoval is the exact release name.

The removal deletes the Operator Deployment, ServiceAccount, and RBAC, and keeps the identity PersistentVolumeClaim, its identity ConfigMaps, and the credentials Secret. The chart notes print their exact names.

Keep confirmRemoval on later upgrades (--reuse-values does this). Restore the Remote Operator brings it back with the same identity. To retire it for good, retire its registration, then delete the kept PersistentVolumeClaim and ConfigMaps.

Recover a stuck Helm releaseCopy Anchor Link

Use recovery only when no Helm process is running. If your terminal disconnected, check helm history first; a pending operation may still be running. Don't delete Helm release Secrets, the identity volume, or the shared CRD to clear a lock.

1. Under Upgrade or restore this operator, enter a Recovery Helm revision: the current pending revision if its identity step already completed, otherwise the last successful revision from helm history.
2. Run Recover failed or stopped Helm operation.

Recovery checks the chart, credentials, resource ownership, and shared CRD against that revision before it clears the pending operation.

Update or roll back on ECSCopy Anchor Link

To upgrade, regenerate the template on the setup page, save the new JSON over the old file, and run Apply reviewed update. After the latest heartbeat reports the new image, confirm the template is the deployed revision.

• A registered installation's update asks for no token. It reuses the current registration secret version and never writes to it.
• The account, Region, cluster, and stack must stay the same.

Alien keeps the previously accepted image as the rollback target. To roll back:

1. Restore the operation selection you used with the previous image. The rollback uses the operations selected now.
2. Under Rollback template, select Prepare rollback to the previously accepted image and download remote-operator.previous.json.
3. Run Rollback.
4. When the heartbeat reports the old image, select Accept rolled-back image as desired.

Credentials and EFS storage are never rolled back.

Repair a connectionCopy Anchor Link

An installation is connected while its heartbeats are less than 5 minutes old.

Open the installation under Deployments. The Repair connection section opens on its own when the installation is stale or still connecting.

• Kubernetes. Inspect Operator prints the Helm status, the Operator Deployment and pods, the last hour of logs, and recent events. Restart Operator runs kubectl rollout restart on the Operator Deployment and waits up to 5 minutes. The Deployment uses the Recreate strategy, so the old pod stops before the new one starts and the identity volume stays attached. Run both with the cluster's kube context; Alien doesn't record it.
• Amazon ECS. Repair task runs aws ecs update-service --force-new-deployment on the stack's service and waits until it is stable. If the new task fails too, read the log group in the stack output LogGroupName.

If the logs show a rejected or expired credential, rotate the connection key.

Before the first registrationCopy Anchor Link

Until the Operator registers, setup can issue new values for the same installation: Generate replacement values (Kubernetes) or Generate replacement registration token (ECS). Both disappear after registration; a registered installation never gets new bootstrap values.

If an earlier Kubernetes install attempt already created the credentials Secret, don't apply the replacement files as a whole: they contain a new encryption key. Instead:

1. Patch only sync-token in the existing Secret.
2. Update that field in your saved operator-credentials.yaml.
3. Check it with kubectl diff --filename operator-credentials.yaml.
4. Rerun the same install command.

On ECS, rerun the deploy command and paste the new token. It keeps the stored encryption key.

UninstallCopy Anchor Link

Uninstalling takes two steps: remove the Operator from the environment, then retire its registration in Alien. Do the second only after the first succeeds. A retired installation can't be installed again from its old values.

Then retire the registration on the setup page: type the release name and select the button that starts Uninstall succeeded (Kubernetes) or AWS teardown succeeded (ECS) and revokes the Alien registration. This deletes the deployment record and its API key at once.

Delete the kept identity volume, ConfigMaps, Secret, or EFS storage only when the installation is retired for good. Remove a Terraform-owned Secret through Terraform, or the next apply recreates it. Also revoke any cloud identity or service credential you created outside the release or stack.

NextCopy Anchor Link

• Add to an existing product
• Troubleshooting