Troubleshooting

Troubleshooting maps each symptom you see on a deployment to what it means, who has to fix it, and what to run. Every investigation starts with these commands:

alien deployments get customer-1/us-east-1
alien deployments events customer-1/us-east-1
alien logs --deployment customer-1/us-east-1 --level error --since 1h

In the dashboard, the deployment page shows the same status plus a Needs attention list. Each item says who acts: For your team or For the deployment owner (the customer's admin). Debug a deployment explains each command's output.

Running but not healthyCopy Anchor Link

Running means the deployment applied its target release. It doesn't mean every replica works, or that your service answers. Check health separately:

• Unhealthy replicas: the release applied but the code fails. Read the logs, then fix forward or roll back.
• The endpoint fails: status doesn't call your endpoint. Call it from where your product calls it, then search alien logs --deployment <deployment> --system --level error for proxy, image pull, or disk errors.
• Release not confirmed: alien deployments wait --for ready checks that the current release equals the desired one. A healthy replica from the old release is not a successful rollout.
• Not reporting: the status shown is the last report. See Not reporting.

Running an older releaseCopy Anchor Link

The deployment keeps serving its current release while a newer one doesn't land. Compare Current release and Desired release in alien deployments get, then read its Deployment updates table:

Failed initial setup or updateCopy Anchor Link

A step failed. During an update, the previous release keeps serving. Read Failed resources in alien deployments get:

1. Fix the first failure. Resources marked Resource was not deployed because resource '<id>' failed were interrupted by another failure. Fix the resource under Failed resources; the interrupted ones deploy on retry.
2. Look past wrappers. Polling timed out after N attempts …, a failed health check, and CrashLoopBackOff only say the resource stopped making progress. Search alien logs --level error and alien logs --system --level error for the first startup error.
3. Decide who fixes it. Code or configuration errors are yours: fix them and release again. Cloud errors such as AccessDenied, an invalid principal, or a quota limit are in the customer's account: the admin fixes them.
4. Retry. Retry from the deployment page (Retry) or with alien deployments retry <deployment>, then alien deployments wait <deployment> --for ready.

A new release fixes code, not the environment. Disk, quota, and permission failures come back until someone changes the environment.

When the admin used the CLI and the first initial setup failed, alien deployments retry prints the setup command to rerun. Initial setup runs with the admin's credentials, so it resumes only when they rerun it against the same account, project, or subscription. See Retry a failed initial setup.

Held releaseCopy Anchor Link

The deployment shows Setup update needed and the rollout row shows Waits for owner. It keeps running its current release.

The release changes what the customer's initial setup owns: it adds, removes, or changes a frozen resource, or needs permissions the management identity (the cloud role Alien uses for ongoing management) doesn't have.

In alien deployments get, the Deployment updates table shows reasons such as Cannot add frozen resources during update, Frozen resource '<id>' was modified, or Permission profile '<name>' was modified, and Action required says Apply the updated setup to continue this update.

The customer's admin reruns initial setup with their own credentials. The held release then continues on its own. See Held releases for the ways to rerun it.

Retrying doesn't help, because the deployment is Running:

• alien deployments retry refuses with Deployment '<id>' is in status 'running' and cannot be retried.
• alien deployments wait --for ready fails right away because the deployment settled on its older release.

Update not confirmedCopy Anchor Link

The update was accepted, but its rollout wasn't confirmed in time. It may still finish and restart replicas, so treat it as in progress: don't start migrations or load tests against it yet.

Select Check again on the deployment page, or run alien deployments retry <deployment>. A newer release or a pin replaces the pending target.

Stuck in progressCopy Anchor Link

The deployment stays in Pending, Update pending, Initial setup, Provisioning, or Updating longer than usual. The top-level status can look calm while one resource retries the same error.

1. Read alien deployments events <deployment> for a step that repeats.
2. Search alien logs --deployment <deployment> --system --since 2h for an error that repeats.
3. Read alien deployments get <deployment> --json and look at each resource's status and error. A resource can be retrying a cloud permission or quota error.
4. If the error is in the customer's account, the admin fixes it. Alien keeps retrying and continues once it can.

retry only applies to failed statuses. For any other status it refuses with cannot be retried.

When nothing above explains the wait, contact Alien support with the deployment ID, the time window, the events output, and your CLI version (alien --version).

Not reportingCopy Anchor Link

The deployment's last heartbeat, the periodic report it sends to Alien, is more than 5 minutes old. The status shown is its last report. Workloads keep running while Alien is unreachable.

1. On a deployment that runs on machines, run alien deployments machines <deployment>. Fresh per-machine heartbeats mean the hosts are up and connected; the problem is in reporting, not an outage.
2. Run alien deployments resources <deployment> and alien logs --deployment <deployment> --system --since 30m.
3. When nothing reports, check the environment's outbound HTTPS. The environment or its network is down, or a firewall change blocks it. The customer's admin fixes that.

Treat stale inventory as a reporting problem first, not proof that a resource is down.

Waiting for secretsCopy Anchor Link

Waiting for secrets means a required secret that the customer provides is missing. Workloads start once the value is present.

These secrets never pass through Alien. The customer writes them into their own secret store, and the deployment page shows where: AWS Systems Manager Parameter Store, Google Secret Manager, Azure Key Vault, or a Kubernetes Secret. See Inputs and secrets.

If the missing input is one you provide, the page shows Deployment input needed with Edit inputs.

Customer key unavailableCopy Anchor Link

The customer revoked or disabled the key used for customer-managed keys, or Alien lost access to it. The Encrypt/Decrypt API returns KEY_UNAVAILABLE (HTTP 503) within minutes of the change.

Alien keeps the resources. When the customer restores access to the key, the deployment recovers on its own without data loss.

Logs are missingCopy Anchor Link

Check these in order:

• Logs aren't collected: the deployment's telemetry setting is Off.
• Logs aren't connected yet: the deployment hasn't connected to a manager, the Alien server that collects its logs.
• Not reporting: a disconnected deployment sends nothing new. See Not reporting.
• Platform logs are hidden: image pulls, mounts, and credential errors appear only with --system.
• The window is too short: alien logs searches the last hour by default. Widen it with --since or --from.
• Remote Operator on Kubernetes collects logs only from Pods that match the label scope chosen in setup.

Teardown requiredCopy Anchor Link

Teardown required means Alien removed what it manages. The resources the initial setup created belong to the customer, so the owner deletes them the same way they set it up, for example by deleting the CloudFormation stack.

See Remove a deployment for what the customer sees.

When removal doesn't finish:

• The stack deletion failed. Check the stack in the cloud console. DELETE_FAILED names the resource that blocked it, often a non-empty bucket, an attached volume, or a network still in use. The owner removes the blocker and deletes the stack again.
• The stack was mid-update. Delete the deployment first with alien deployments delete <deployment>, then delete the stack.
• The resources are already gone, but the record stays. Remove only Alien's record with alien deployments delete <deployment> --force <deployment-name>, or Forget record in the delete dialog. The name is the confirmation, and --force can't be combined with --yes. Do the same for a deployment that never created anything.

Never force-delete a deployment whose resources still exist: they keep running in the customer's account with nothing managing them.

After removal, some resources stay on purpose:

• frozen resources, such as a bucket with the customer's data
• snapshots kept on deletion
• anything initial setup didn't create, such as an existing VPC
• on Azure, key vaults in their soft-delete retention period

Check the account by name prefix or tags before you call removal complete.

CLI errorsCopy Anchor Link

Remote OperatorCopy Anchor Link

When an operation fails, check its dependencies in this order and fix the first one that fails:

1. The operation is selected in Alien.
2. It's present in the generated Operator image.
3. Kubernetes RBAC or the cloud identity allows it.
4. The target is reachable over the network.
5. The target accepts the credential.

Don't grant broad cluster access as a shortcut.

Install and upgrade commandsCopy Anchor Link

The generated commands and the chart check the target before they change anything. When a check fails, they print one of these messages and stop.

If an ECS update fails, check the CloudFormation stack events. Wait for any running update or rollback to finish, fix the cause, and rerun Apply reviewed update or Rollback.

NextCopy Anchor Link

• Debug a deployment
• Deployments and groups
• Roll out releases