Troubleshooting operations

Each section starts from what you see. Logs of the services are in the journal: journalctl -u kyvernon-controller and journalctl -u kyvernon-agent. The controller writes JSON lines by default (format in [logging]).

Controller not starting

Cause. The unit does not start without /etc/kyvernon/controller.toml. A value outside its range, an unknown key or a missing credential file stops the service at start.

Remedy.

1. Check the configuration: sudo kyvernon-controller --config /etc/kyvernon/controller.toml check-config. The message names the key.
2. Read the last lines of the journal: journalctl -u kyvernon-controller -n 50.
3. Compare the file with the skeleton /usr/share/kyvernon/controller.toml.example after an upgrade. The keys of the settings are in Configuration.

Controller not ready

Cause. Liveness and readiness are different. The controller is alive when it serves requests, and ready when its store has a leader and it has applied its state.

Remedy.

1. Ask both: kyvernon system health and kyvernon system ready. ready exits with code 1 when the controller is not ready. The same checks are GET /healthz and GET /readyz.
2. When it stays not ready after a start, read the journal for the message about the store.
3. kyvernon system metrics prints the controller metrics in Prometheus text format (GET /metrics). They need a token unless anonymous = true in [metrics].

Host not connecting

Cause. The agent registers with the join token and verifies the controller by its certificate. A host that shows Disconnected after the agent started is usually one of these: the join token is wrong or missing, the certificate in controller-ca.pem is not the controller's bootstrap certificate, server_name is not one of the names in [tls] san of the controller, endpoint or the gRPC port of the controller is not reachable from the host, the name of the host differs from the name added in the cluster, or an administrator disconnected the host.

Remedy.

1. Read the journal on the host: sudo journalctl -u kyvernon-agent.service -n 50.
2. Compare endpoint, ca_file, server_name and hostname in /etc/kyvernon/agent.toml with Install from an offline depot, step 3.
3. Check that /etc/kyvernon/join-token exists and equals the token on the controller.
4. A host that lost its certificate, for example after a reinstallation or by removing /var/lib/kyvernon/agent/identity, is refused under the same name. Add it again with --force: Hosts.
5. Check whether an administrator disconnected the host, and reconnect it with Reconnect.

Host not responding

Cause. The controller shows Not responding when no heartbeat arrives from a host that was connected: the agent stopped, the host is down, or the network between them is interrupted. Virtual machines on the host are not stopped by a stopped agent.

Remedy.

1. Check the machine and the agent: systemctl status kyvernon-agent.service on the host.
2. Check the network between the host and the gRPC address of the controller.
3. Start the agent again and check that the host shows Connected. Until then, operations that need the host fail, for example the console with error.console.host_disconnected.

Power-on refused

Cause. The licence or the memory of the host does not allow it. The message id names which.

Remedy.

1. error.licence.limit_running_vms: Licence.
2. error.host.insufficient_memory, error.vm.exceeds_host_cpu or error.vm.exceeds_host_memory: Capacity.
3. error.host_in_maintenance: choose another host or leave maintenance mode.

Backup session failing

Cause. A session ends as ABORTED or EXPIRED, or a request is refused with a backup.* key.

Remedy. The reason of the session is a message key. Back up a virtual machine lists the keys a client must handle and what to do for each.

Data for support

When you contact support, pass on the message id of the error, its correlationId (also in the header x-request-id of the response), the time, and the output of kyvernon about. The correlation id appears in the logs and in the audit records of the request.

Related

• Reference of error codes
• Configuration