Configuration

Files, directories and the controller.toml keys for capacity, tasks, memory and storage of the controller.

Where things are

What	Where
Binaries	`/usr/bin/kyvernon-controller`, `kyvernon-agent`, `kyvernon-vmm`, `kyvernon`
Configuration	`/etc/kyvernon/controller.toml`, `agent.toml`, `join-token`, `admin-password`, `controller-ca.pem`
Skeletons	`/usr/share/kyvernon/controller.toml.example`, `agent.toml.example`
State	`/var/lib/kyvernon/controller`, `/var/lib/kyvernon/agent`, images under `/var/lib/kyvernon/images`
Logs	`journalctl -u kyvernon-controller`, `journalctl -u kyvernon-agent`
Manual pages	`man kyvernon-controller`, `man kyvernon-agent`, `man kyvernon-vmm`, `man kyvernon`

Capacity

Capacity settings in controller.toml (section [vm_ops]; the defaults suit most installations):

Key	Default	Meaning
`max_vcpus_per_vm`	`0`	Most vCPUs of one virtual machine. `0` means the logical CPUs of its host.
`host_memory_reserve_mib`	`2048`	Memory kept for the host itself. A virtual machine cannot be larger than its host's memory minus this.
`memory_overcommit_ratio`	`1.0`	Power-on is refused when the memory of the running virtual machines of a host would exceed its memory (after the reserve) times this. `1.0` means no overcommit.
`max_thin_disk_gib`	`0`	Largest thin disk. `0` means the capacity of its datastore. Thick disks must always fit in the free space.

Tasks

Task settings in controller.toml (section [tasks]; the defaults suit most installations):

Key	Default	Meaning
`workers`	`8`	Tasks that work in the controller at the same time.
`waiting_tasks`	`256`	Tasks that may be in progress on top of `workers` because they only wait: for a guest to shut down, for a host to copy a disk or migrate a virtual machine, for another task on the same virtual machine. They hold up no other task while they wait. With `0` a waiting task keeps its worker, and eight slow shutdowns hold up every other task.
`queue_capacity`	`1024`	Tasks that may wait to start. A request beyond it is refused with `RESOURCE_EXHAUSTED`; the caller tries again.
`sweep_interval`	`1m`	How often tasks that nobody runs any more are looked for. `0s` turns the periodic check off.
`orphan_timeout`	`15m`	A queued task that could not be started for this long fails.
`max_run_time`	`1d`	A task that runs longer than this is cancelled and fails. `0s` means no limit.
`scrub_secrets`	`true`	When the controller becomes leader, finished tasks whose recorded parameters still hold guest user data or passwords (records from before those were redacted) are rewritten without them. The log says how many.

At most workers + waiting_tasks tasks are in progress at a time; the next one waits in the queue.

Backup

Settings of the backup API in controller.toml (section [backup]; see Back up a virtual machine and Restore a virtual machine):

Key	Default	Meaning
`state_dir`	`backup` under the state directory	Where sessions and their change ids are kept.
`remote_exports`	`false`	Serve the disk exports to other machines over TCP with TLS 1.3, instead of on the host only.
`agent_timeout`	`10m`	How long the controller waits for a host in a backup call.
`verify_schedule`	not set	Nightly restore verification: a time of day in UTC (`02:30`) or an interval (`6h`). Not set: no job.
`verify_repository`	not set	Directory with the stored backups that the verification restores.
`verify_network`	not set	The quarantine network, by name or id.
`verify_timeout`	`300s`	Time allowed for one verification.
`verify_destroy`	`true`	Destroy the restored virtual machine after its verdict.
`verify_pool`	empty	Datastore for the restored disks. Empty: the first usable one.
`controller_target`	`controller-backups` next to `data_dir`	Folder for the controller state archives, for example `/var/lib/kyvernon/controller-backups`. See Back up the controller.
`controller_schedule`	not set	When the controller archives its state: a time of day in UTC (`02:30`) or an interval (`12h`). Not set: only on request.
`controller_retention`	`7`	Archives of the controller state that are kept.

The repository and the verification settings can also be changed in the console, and the backup settings of the console apply on every controller within about 10 seconds. A value that was never changed there is read from the keys above.

Memory and storage

Memory and storage of the controller (the defaults suit most installations; the service refuses to start with a value outside the stated range):

Key	Default	Meaning
`[node] change_feed_max_bytes`	`67108864` (64 MiB)	Memory for the change history that API clients catch up from. A client further behind lists again. `0`: no byte limit; otherwise at least 65536.
`[node] snapshot_log_bytes`	`33554432` (32 MiB)	The controller writes a snapshot of its store when the log since the last one exceeds this (or the size of the last snapshot, when that is larger), and every 5,000 changes. Bounds the size of `inventory.redb`. `0`: every 5,000 changes only; otherwise at least 1048576.
`[node] store_cache_bytes`	`67108864` (64 MiB)	Page cache of `inventory.redb`, memory of the controller process. At least 4194304.
`[node] history_on_disk`	`true`	Keep tasks, events and audit records in `inventory.redb` with only an index in memory. `false`: keep them in memory and inline in the snapshot, as before (see the upgrade note below).
`[node] history_flush_bytes`	`16777216` (16 MiB)	A snapshot is also written when this much history text changed since the last one; it bounds the history held in memory. `0`: off; otherwise at least 1048576.
`[node] keep_pre_history_copy`	`true`	Copy the store to `inventory.redb.before-history-on-disk` before the start that moves history into it (once). Remove the copy when satisfied.
`[retention] history_max_bytes`	`4294967296` (4 GiB)	Most JSON text of tasks and events kept in the store. Above it the oldest events and finished tasks are removed first, before their age (30 days for tasks, 90 for events) or count limit. Running tasks and audit records are not removed by it. `0`: no byte limit; otherwise at least 1048576.
`[retention] audit_warn_bytes`	`2147483648` (2 GiB)	Above this much audit record text a warning is logged and an `AuditHistoryLargeEvent` is emitted once per crossing (an alarm can trigger on it). Audit records are never deleted by size; they leave by `audit_age` and `audit_max`. The gauge is `kyvernon_history_text_bytes{kind="audit"}`. `0`: off.
`[metrics] cache_bytes`	`67108864` (64 MiB)	Page cache of `metrics.redb`, memory of the controller process. At least 4194304.
`[metrics] deleted_entity_retention`	`"7d"`	How long performance charts of a deleted virtual machine (or a removed host or datastore) are kept before they are removed. `"0s"` to `"366d"`. Usage reports are not affected (`ledger_retention_days`).