Kyvernon

Reference For: administrators, developers

Release notes

What changed in each version of Kyvernon, newest first.

Unreleased

Added

  • Distributed switches and port groups through the native API, CLI and console: POST /v1/distributed-switches, PUT/DELETE /v1/distributed-switches/{id}, POST /v1/distributed-portgroups, PUT/DELETE /v1/distributed-portgroups/{id}; kyvernon network dvs-create|dvs-update|dvs-delete|portgroup-create|portgroup-update|portgroup-delete (docs/network/DESIGN.md §6.4).
  • Host uplinks of distributed switches: uplinkNames and hostUplinks (per member host the physical NIC behind each uplink), shown by the vSphere facade as config.uplinkPortPolicy and config.host[].config.backing.pnicSpec. VLAN-backed port groups get an OVN localnet port with their VLAN; the agent puts the NICs into the switch's bridge on each host and sets ovn-bridge-mappings. Agent setting [network] datapath_type for hosts on the userspace datapath.
  • Addresses on port groups: ipv4 (subnet, gateway, DNS, range, DHCP), ipv6 (/64 prefix, DHCPv6) and staticAddresses; every NIC gets a fixed address (from the range, or the one fixed for its MAC) and its EUI-64 IPv6 address, kept in the port group's leases. OVN hands them out by DHCPv4 and DHCPv6, port security admits exactly them, and firewall groups match segment vNICs by these addresses.
  • OVN logical networks in production (docs/network/DESIGN.md §6.4), off until [ovn] enabled = true in controller.toml: distributed port groups become OVN segments (overlay with a VNI or VLAN 0, VLAN-backed otherwise) and VM NICs on them logical ports <vm>-nic<key> with MAC port security, bound to the VM's host and, during a live migration, to the destination as well (the migration waits until the destination reports them bound). The controller builds the network model from the inventory, realises it in the OVN Northbound database (unix socket on the controller host, TLS when remote), corrects other writers' changes, resynchronises after a reconnection and every five minutes, and reports the realization at GET /v1/network/realization (System.Read) with events NetworkRealizationFailedEvent, NetworkRealizationRestoredEvent, NetworkNorthboundLostEvent, NetworkNorthboundConnectedEvent.
  • [ovn] of controller.toml: northbound, southbound (read only, rule counters), host_southbound (must be ssl:), domain, resync_interval, cookie_interval, [ovn.pki] (issue, central_names), [ovn.tls]. The OVN TLS material is issued from the host CA into <data_dir>/tls/ovn (central.* for the databases, controller.* for the controller, ca.pem), renewed 30 days before expiry; ovn-controller uses the agent's host certificate (CN = host id = chassis name, Southbound RBAC role ovn-controller).
  • The desired-state stream carries each host's network slice (DesiredState.network_json) and the heartbeat its realization (network_report_json): the agent sets system-id, ovn-remote, ovn-encap-type and ovn-encap-ip from the controller's settings and the host's join data, plugs the taps of its logical ports into br-int with external_ids:iface-id, and reports each port's ovn-installed. [network] ovn_controller_log of agent.toml (default /var/log/ovn/ovn-controller.log).
  • Distributed firewall on segment vNICs reported like host bridges: rule counters from br-int's flow statistics (the controller sends the logical-flow cookie map), log records and monitor entries from ovn-controller's ACL log, and a flush of a port's conntrack zone when its mode moves from monitor to enforce (DFW-HOSTBRIDGE.md §11.1). Rule statistics report supported: true for segment-only rules when the controller reads the Southbound database.
  • deploy/server1/setup-ovn-central.sh, setup-ovn-host.sh and ovn-doctor.sh (read-only checks); none of them runs from setup-host.sh. tests/lab/ovn-e2e.sh runs the end-to-end test on a private OVN.
  • BIOS firmware on Kyvernon VMM (protocol 1.4, BootConfig::Bios): a VM with firmware = bios boots SeaBIOS from the host's seabios package ([kyvernon_vmm] firmware_bios, default /usr/share/seabios/bios-256k.bin). The image ends at 4 GiB and its last 128 KiB are copied below 1 MiB (again on every in-place reset); the machine is described through fw_cfg as for UEFI; no SMM. With a console the display adapter gets vgabios-bochs-display.bin (firmware_bios_vga_rom) as its option ROM, and BIOS text output is mirrored to the serial log through COM1 (the firmware debug port then stays out of the log, which would otherwise show each line twice). boot.menu shows the BIOS boot menu prompt. Low RAM of a BIOS VM ends at 2.75 GiB (SeaBIOS places its ECAM window at 0xB0000000) and the 32-bit PCI window is 0xC0000000 to 0xFEC00000. Design: docs/hypervisor/FIRMWARE.md §11.
  • kyvernon vm create --firmware bios|efi and kyvernon vm reconfigure --firmware bios|efi replace the firmware of the JSON file.
  • vSphere API: bootOptions.efiSecureBootEnabled in VirtualMachineConfigSpec (CreateVM, ReconfigVM, CloneVM, ImportVApp) sets Secure Boot; config.bootOptions.efiSecureBootEnabled reports it.
  • NSX Policy API subset over the distributed firewall (docs/network/DFW-HOSTBRIDGE.md §6.4, step C12; docs/api-compatibility/NSX_POLICY.md): [facades.nsx] serves domains (default), groups (tag conditions, AND/OR, nested expressions, addresses, group and network paths, VM and vNIC external ids), services, security policies and rules (also through the hierarchical PATCH /policy/api/v1/infra for these objects), VM tags, VIFs, search, statistics, realization status and projects (tenants) on a listener of its own, with HTTP Basic or POST /api/session/create. Everything is a native firewall record with the native validation, privileges, tenant scoping, events and audit; NSX ids, tags and the client's own fields are kept in fw_nsx_binding records written in the same transaction. Features outside the subset answer with an NSX error that names the field (error.nsx.unsupported). Verified with the Terraform NSX-T provider 3.12.2 on the lab host.
  • NSX Policy API subset, follow-up: a hierarchical PATCH /infra body is validated as a whole and committed in one inventory transaction, all or nothing, and may create objects that refer to each other (transactions can name records they create, Transaction::reference); every NSX write goes through the same batch (fwapi::batch). Display names follow NSX: not unique, up to 255 characters; the native record gets a derived unique name (web_servers [dup-b]), and error.nsx.display_name_in_use is gone. Address ranges in groups and rules, source_ports in services, MACAddressExpression in groups; identity groups are refused with the reason. Rule writes through the NSX paths raise the rule events. Tag categories and tags created for NSX VM tags are removed with their last use unless a firewall group names them.
  • Firewall model: source port ranges on TCP, UDP and SCTP services (sourcePorts in the REST and gRPC bodies, tcp:443@1024-65535 in the CLI), compiled by both the OVN and the Open vSwitch backends and the reference evaluator; groups take macs (every vNIC with one of these MAC addresses; --mac in the CLI).
  • Built-in catalogue of 37 named services (SSH, HTTPS, DNS-UDP, ICMP_Echo_Request and the rest), read-only, usable as {"protocol": "named", "id": "SSH"} in native rules and as /infra/services/SSH through the NSX subset.
  • Provider rule templates: PUT, GET and DELETE /v1/firewall/templates/{id} (gRPC PutTemplate, GetTemplate, DeleteTemplate; Kyvernon.Firewall.ManageProvider), applied like the built-in ones. Event FirewallTemplateChangedEvent.
  • Firewall console (docs/network/DFW-HOSTBRIDGE.md §7, step C13): the Firewall section (/firewall) with policies in four category bands (pending edits, review with validation issues, publish with If-Match and ?wait=realized, drag and keyboard ordering), the rule editor with a simulate preview, groups with members and redacted shared groups, services, statistics, log records, rollout, realization per host, emergency bypass with a banner, settings, default profiles and provider-managed tag categories; a Firewall tab on every VM (effective rules per network adapter, mode switch with confirm timer and countdown, open port, templates, profile, monitor report with an acceptable proposal); all strings in English, Dutch, French and German.
  • Firewall emergency bypass (docs/network/DFW-HOSTBRIDGE.md §3.9): GET/POST /v1/firewall/bypass, DELETE /v1/firewall/bypass/{id} (Kyvernon.Firewall.Bypass): an audited Emergency allow rule for every vNIC of one host, with a reason and an end (60 s to 24 h); it ends by itself. Events FirewallBypassEnabledEvent, FirewallBypassExpiredEvent. CLI kyvernon firewall bypass list.
  • Firewall objects follow VMs and tenants: deleting a VM removes its own policy and group and its static group memberships in the same transaction; destroying a closed tenant removes its groups, services and policies. Clone takes copyTags and copyFirewall (same tenant); without them the clone's vNICs start like new ones.
  • enforceNew in the firewall settings is applied: while it is on, every vNIC added to a VM starts in mode enforce.
  • description on firewall groups, services and policies.
  • Roles holding Kyvernon.Firewall.ManageProvider or Kyvernon.Firewall.Bypass (Administrator among them) cannot be granted inside a tenant's folders and pools (error.identity.provider_privilege_in_tenant).
  • Firewall plans larger than one gRPC message are sent in parts to agents that advertise firewall-parts (up to 64 MiB).
  • Firewall counters, log records and monitor entries survive a controller restart (<data_dir>/firewall-telemetry.json, a bounded rolling window).
  • CLI kyvernon firewall (docs/network/DFW-HOSTBRIDGE.md §6.2, step C10): groups (with members), services, policies, rules (list, show, add, update, delete, move), effective, open-port, templates, mode (with --confirm-within and mode confirm), monitor, stats, simulate, realization, logs, profiles, settings, tag-categories, vnic-profile and bypass. Writes take --if-match <version> and --wait (with --wait-timeout); an update with flags reads the object and writes it back with the version it read. Tables for every list and view, -o json (same as --json) for the raw response; firewall errors print their error.firewall.* key and the objects they name.
  • Firewall rollout (docs/network/DFW-HOSTBRIDGE.md §4.4, step C9): every vNIC stays in mode off until it is changed (no upgrade step, nothing changes for running VMs). POST /v1/vms/{id}/firewall/mode (task, Kyvernon.Firewall.ManageProvider) sets off, monitor or enforce; with confirmWithin (30 s to 24 h) the previous mode comes back unless POST /v1/vms/{id}/firewall/mode/confirm keeps the new one. The pending revert is stored on the VM record, so it survives a controller restart. Events VmFirewallModeChangedEvent, VmFirewallModeRevertedEvent. GET /v1/vms/{id}/firewall/monitor lists what the rules would have dropped in mode monitor over monitorReportDays and proposes a template and rules.
  • The agent firewalls no vNIC of a host whose ovn-controller zone range overlaps the firewall's (ct-zone-range precondition); the vNICs keep forwarding as in mode off and the realization reports the error.
  • Firewall defaults (docs/network/DFW-HOSTBRIDGE.md §4, step C8): built-in, read-only records created when the controller starts: the baseline of profile public-default (sys:baseline-public: SSH and ICMP echo inbound), the provider SMTP rule (sys:smtp: outbound TCP 25 dropped per family for the vNICs whose pool and VM switch block it; the SMTP API and event are unchanged) and tenant isolation on private host-bridge networks (sys:tenant-isolation: pass from the same tenant, drop from the other tenants of the network). Memberships and isolation rules are computed from the inventory, not stored.
  • POST /v1/vms/{id}/firewall/open-port, GET /v1/firewall/templates, POST /v1/firewall/templates/{id}/apply (task; ssh-only, web, mail-server) fill the VM's own policy; PUT /v1/vms/{id}/nics/{key}/firewall sets a vNIC's profile (tenant users: selectable profiles only). Events FirewallTemplateAppliedEvent, VmFirewallProfileChangedEvent.
  • Distributed firewall API (docs/network/DFW-HOSTBRIDGE.md §6.1, step C7): /v1/firewall/groups (with members), /services, /policies with /rules (add, change, delete, move), /policies/{id}/stats, /rules/{number}/stats, /profiles, /settings, /tag-categories, /realization, /simulate, /logs, and GET /v1/vms/{id}/firewall, /v1/vms/{id}/nics/{key}/firewall, /v1/vms/{id}/firewall/logs; the same over gRPC (kyvernon.v1.FirewallService). Writes take If-Match and ?wait=realized&timeoutSeconds=, run the shared validation and refuse only the errors they add (error.firewall.validation with the net.validation.fw_* key), record an event (FirewallGroupCreatedEvent and the rest of §6.6) and an audit record. Tenant objects belong to the tenant record; tenant users work in their own tenant and see shared provider groups by name only. Rule numbers come from one domain counter.
  • Privileges Kyvernon.Firewall.Read, ManageGroups, ManagePolicies, SetProfile, ManageProvider, ReadLogs, Bypass; the tenant administrator role gets the five tenant ones, the tenant user role Read.
  • Tenant quotas fwGroups (default 100), fwPolicies (50), fwRules (500), fwGroupMembers (1,000); null means the default.
  • Tag categories have providerManaged: only such categories count in provider firewall groups, and tenant users cannot attach or detach their tags (Automation API tagging).
  • Agents ship firewall log records, rule counter deltas, monitor entries and host preconditions in the heartbeat (fw_report_json, bounded; fw_report_full in the answer). The control plane keeps them in memory for the counters, logs and monitor reports, and raises FirewallRealizationFailedEvent, FirewallFlowLimitEvent and FirewallRealizationRestoredEvent from the hosts' firewall errors.
  • The agent reports ct-zone-range as a host precondition when ovn-controller runs on the host and its conntrack zone range overlaps the firewall's zones ([network] ct_zone_base, default 32768). Setting Open_vSwitch external_ids:ct-zone-range=1-32767 is an owner action before the first vNIC of such a host leaves mode off; the agent changes nothing on the host.
  • History on disk. Tasks, events and audit records are stored in two tables of the controller's store file (inventory.redb) and written in the snapshot commit. Memory holds an index row per task and event, plus the bodies of records changed since the last snapshot and of open tasks. The snapshot blob is schema 2 (inventory records only). Design: docs/control-plane/HISTORY-ON-DISK.md.
  • Controller configuration: [node] history_on_disk (default true), [node] history_flush_bytes (default 16 MiB; a snapshot is also written when this much history text changed), [node] keep_pre_history_copy (default true), [retention] audit_warn_bytes (default 2 GiB, 0: off).
  • AuditHistoryLargeEvent, emitted once per crossing of audit_warn_bytes. Audit records are never deleted by size.
  • Metrics kyvernon_history_records{kind}, kyvernon_history_text_bytes{kind}, kyvernon_history_memory_records, kyvernon_history_memory_bytes, kyvernon_history_dirty_bytes, kyvernon_history_index_bytes, kyvernon_history_flush_records_total, kyvernon_history_flush_seconds, kyvernon_history_disk_reads_total{op}, kyvernon_history_scan_rows_total, kyvernon_history_index_unparsed_total (docs/control-plane/METRICS.md).
  • Console: public addresses on a server's Networks tab. Assign an address (primary with an IPv6 block, an extra address, or a specific IPv4 address) and release one or all addresses behind a confirmation that names the quarantine. The new-virtual-machine wizard can add a second network card with a public address and writes its network configuration. Described in Public addresses.
  • IP pools: GET /v1/ip-pools/{id}/available?limit=N (CLI kyvernon ip-pools available <id>) lists the IPv4 addresses that can be allocated now, lowest first (default 256, at most 4096), with the number of free IPv4 addresses and free IPv6 blocks. It uses the allocator's own list, so an allocation without an address takes the first listed one; a listed address that someone else took first fails with error.ip_pool.address_in_use. A pool's usage gains ipv6Quarantined and ipv6Free.
  • Console: the assign-address dialog and the wizard's second network card choose the IPv4 address from a list (Automatic (first free) and the pool's free addresses) instead of a text field, show the free count next to each pool and do not offer a full pool. When the chosen address was given to someone else in the meantime the dialog says so and lists again. The IP pools page has a Free column.

Changed

  • Segment vNICs moved from monitor to enforce: the agent flushes the port's conntrack zone once ovn-controller reports the enforcing flows installed, not when the slice arrives. The controller raises NB_Global.nb_cfg after each realization and sends it in the slice (realized); the agent waits for that value in br-int's external_ids:ovn-nb-cfg (at most 60 s). On the kernel datapath an early flush did not hold: the next packet of an established connection took it up again.
  • OVF and OVA import keeps the firmware of the package: BIOS stays BIOS, UEFI stays UEFI, and only a package that states no firmware gets UEFI (warning import.ovf.firmware_default). Secure Boot from the package (bootOptions.efiSecureBootEnabled) is kept with UEFI and dropped with BIOS (warning import.ovf.secure_boot_bios). The import plan (POST /v1/ovf/import-spec) gains secureBoot. Before, every package was imported as UEFI with the warning import.ovf.firmware_uefi, and a guest installed with BIOS on an MBR disk usually did not boot.
  • A virtual TPM needs UEFI firmware: the inventory refuses vtpm with firmware = bios (config.vtpm), as vSphere does, and so do the VM model and the agent.
  • The kyvernon-agent package depends on seabios.
  • The native import lease (POST /v1/resource-pools/{id}/import-vapp) gives a package network adapter without a MAC address a generated one; it failed with invalid MAC address before.
  • Tenant isolation rule numbers of the firewall are allocated densely per (network, owner) pair; they no longer depend on network and tenant numbers and cannot collide.
  • API: a Deleted change of a task or an event (GET /v1/inventory/changes, gRPC watch, the change feed of kyvernon-inventory) carries a stub body: the record's metadata with a nil uuid, and data with only entity (and user for a task). It carried the last full body before. Pruned audit records publish no per-record change.
  • Events and audit records are append-only in the state machine; Update of either and Delete of an audit record fail with FAILED_PRECONDITION (history_record_immutable). History records cannot have a parent or a name.
  • [retention] history_max_bytes defaults to 4 GiB (it was 128 MiB). It now bounds the history text on disk, not in memory.
  • Upgrade: the first start migrates the store with a forced snapshot and keeps a copy of the old store as inventory.redb.before-history-on-disk. To go back to a previous version, set history_on_disk = false and start once; the snapshot is rewritten in the previous format. See Upgrade.

Security

  • Guest passwords no longer leave the browser in plain text. The VM wizard and Edit guest access send a SHA-512 crypt hash ($6$) as the cloud-config password, which cloud-init sets with chpasswd -e; SSH password login and the password and confirm fields work as before. User data with a plain-text password is refused. A hashed password cannot be shown on the VM summary any more.
  • Task records no longer hold secrets. Parameters and results are redacted when a task is submitted and when it finishes (guestinfo.userdata, guestinfo.vendordata, unattend, and keys naming a password, secret, token or CHAP), for create, reconfigure, clone, customize and every other task; the handler gets the real values from memory. A task whose secrets were lost by a controller restart or fail-over fails with error.task.secrets_lost. Raw task reads redact the same fields.
  • [tasks] scrub_secrets (default true): when a controller becomes leader it rewrites finished tasks that still hold such secrets and logs how many. Raft log segments, older snapshots, inventory.redb.before-history-on-disk.* copies and backups are not cleaned (docs/control-plane/SECURITY.md section 4b).

Version 0.1.