Troubleshooting

This page contains helpful commands, common problems and safe working practices for troubleshooting a running Plethos instance.

Helpful commands

Terminal

# All VClusterClaim
kubectl get vcc -n plethos

# Status and events of a single VClusterClaim
kubectl describe vcc <claim-name> -n plethos

# All Kubernetes resources running in a vCluster namespace
kubectl get all -n vcluster-<claim-name>

# The ingresses created for a vCluster (should return 2 rows, one for the vCluster API and one for the exposed applications)
kubectl get ingress -n vcluster-<claim-name>

# Controller and portal logs
kubectl logs deployment/plethos-controller -n plethos
kubectl logs deployment/plethos-portal -n plethos

Resource names are derived from your release name, adjust them if yours isn’t plethos.

Common problems

Symptom	Likely cause and what to check
Claim stuck in Provisioning	The Helm install Job can’t finish, for example because the vCluster image or chart can’t be pulled. Run kubectl get jobs,pods -n <vcluster-namespace> and read the Job’s logs.
Claim stuck in Pending or Sleeping...	Either the controller is down or the vCluster pod won’t scale up/down. kubectl describe pod in the vCluster’s namespace shows why.
Claim in Error	The claim’s status names the reason: kubectl get vcc <name> -n plethos -o jsonpath='{.status.message}'. One example is a claim whose version does not exist in supportedVclusterVersions.
Claim stuck in Deleting	The controller isn’t running (crashed, or scaled down earlier), or the uninstall Job keeps failing. Start the controller or read the Job’s logs in the vCluster’s namespace.
Cluster shows Sleeping but workloads still run	A known race: the control plane recreated workloads while scaling down. They are removed on the next reconciliation (every 3 hours), or you can delete the leftover pods by hand.
Cluster went to sleep or was deleted unexpectedly	Expiry. Check the claim’s renewed-at annotation and creation date. kubectl get jobs -n plethos lists the expiry job runs and their logs show which steps ran and whether the warning emails were sent.
Portal pod won’t start	Usually the portal Secret doesn’t exist yet or lacks a key. kubectl describe pod shows the missing Secret. Create it and restart the Deployment.
Login fails, or redirects and email links point to localhost	portal.baseUrl is still the default, or the redirect URI registered at your identity provider doesn’t match it. Also check portal.oidc.* and the client secret in the Secret.
No emails arrive	portal.smtp.host and port are still the defaults (empty and 1025), or the credentials are missing from the Secret. The portal logs show the SMTP error.
vCluster API unreachable or shows the wrong certificate	The wildcard DNS record for the API domain is missing, or the ingress controller wasn’t started with SSL passthrough. Check the ingress controller’s arguments. Also, controller.ingressNamespace must match the namespace it really runs in and with hostNetwork you need ingressControllerIpRanges.
Exposed app returns 404 or 503	The owner hasn’t deployed the http Service, or it is in the wrong namespace or on the wrong port. Inside the vCluster, kubectl get svc http -n default must show port 80. Also check if the ingress exists via kubectl get ingress -n vcluster-<claim-name>.

Stopping the controller during troubleshooting

Sometimes it is safer to stop the controller during troubleshooting if you make changes to VClusterClaims by hand. While you are changing things, the controller can react to a half-finished edit, undo your changes or modify vClusters in ways you didn’t expect. Stopping it first gives you full control and it picks everything up when you start it again.

Terminal

# stop
kubectl scale deployment plethos-controller -n plethos --replicas=0

# start again (use your controller.replicas value, default 1)
kubectl scale deployment plethos-controller -n plethos --replicas=1

Stop the controller before you:

• edit managed resources by hand, such as a vCluster StatefulSet, an Ingress or a NetworkPolicy.
• recreate a VClusterClaim, for example to change a value that is immutable. Deleting a claim normally tears its vCluster down, but with the controller stopped nothing is removed. The old claim stays in Deleting until you remove its finalizer, after which you can create the new one with the same name.
• delete or clean up resources by hand, such as a vCluster namespace or the finalizer of a claim that is somehow stuck. The controller might recreate resources while you are still removing them. To remove a cluster properly, delete its claim instead.
• change many claims at once, for example when restoring them from a backup. Apply everything first, then start the controller so it sees the complete state.

While it is stopped, no clusters are provisioned and claim status stops updating, but nothing breaks.

The expiry job CronJob keeps running while the controller is stopped. If you want to pause/continue it use the following commands:

Terminal

# suspend
kubectl patch cronjob plethos-inactivity-check -n plethos -p '{"spec":{"suspend":true}}'

# continue
kubectl scale deployment plethos-controller -n plethos --replicas=1