How to do proper key rotation?

[isoos]

Is the following process a valid key rotation? E.g. once a month or every quarter:

• Generate new CA key (and store alongside the previous CA keys).
• Concatenate all valid CA key certificates into a single ca.crt file.
• Sign each host's public key with the new CA key and store the signed certificate alongside the existing host certificates.
• Concatenate all valid host certificates into a single host.crt file.
• Copy the new ca.crt and host.crt to the host and update/restart the process.

I assume this will work even if I would update certain hosts only much later (as they are not automated), but everything could use up to ~12 CA certificate and should figure out how to communicate with each other, am I right? (Of course frequencies and expiry dates could be customized, but that's not the scope of this question).

[johnmaguire]

@isoos Close, but not quite.

1. Generate a new CA key.
2. Add the new CA certificate to the pki.ca bundle of all hosts in the network. Leave the existing CA in-place for now. Reload or restart Nebula.
3. Sign host certificates with the new CA key, update pki.cert and pki.key and reload/restart Nebula on each host. You cannot run with multiple host certs. The new cert will be trusted by other hosts because you added the CA to the trust bundle in the last step.
4. Remove the old CA from pki.ca and reload/restart Nebula on each host.

[isoos]

You cannot run with multiple host certs.

@johnmaguire:
I'm a bit confused, because apparently nebula starts with multiple host certificates:

nebula-cert ca -name test -out-crt c1.crt -out-key c1.key
nebula-cert ca -name test -out-crt c2.crt -out-key c2.key
nebula-cert keygen -out-key host.key -out-pub host.pub
nebula-cert sign -ca-crt c1.crt -ca-key c1.key -in-pub host.pub -ip 192.168.100.1/24 -name host -out-crt host.c1.crt
nebula-cert sign -ca-crt c2.crt -ca-key c2.key -in-pub host.pub -ip 192.168.100.1/24 -name host -out-crt host.c2.crt
cat c1.crt c2.crt >ca.crt
cat host.c1.crt host.c2.crt >host.crt

with the following host.yml:

pki:
  ca: ca.crt
  cert: host.crt
  key: host.key

listen:
  host: 0.0.0.0
  port: 0

firewall:
  outbound:
    # Allow all outbound traffic from this node
    - port: any
      proto: any
      host: any
  inbound:
    # Allow all outbound traffic from this node
    - port: any
      proto: any
      host: any

config testing and after that running works:

nebula -config host.yml

INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:outgoing endPort:0 groups:[] host:any ip: localIp: proto:0 startPort:0]"
INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:incoming endPort:0 groups:[] host:any ip: localIp: proto:0 startPort:0]"
INFO[0000] Firewall started                              firewallHashes="SHA:498215dec4e5687a2353f51c10838c113bd1af35ef72b8e8c9f536986ada5417,FNV:2782948616"
INFO[0000] listening on 0.0.0.0:0
INFO[0000] Main HostMap created                          network=192.168.100.1/24 preferredRanges="[]"
INFO[0000] punchy disabled
WARN[0000] No lighthouse.hosts configured, this host will only be able to initiate tunnels with static_host_map entries
INFO[0000] Loaded send_recv_error config                 sendRecvError=always
INFO[0000] Nebula interface is active                    boringcrypto=false build=1.9.4 interface=tun0 network=192.168.100.1/24 udpAddr="0.0.0.0:36420"

[johnmaguire]

@isoos Only the first host certificate is used. Others are ignored:

nebula/pki.go

Lines 185 to 204 in 3e6c755

	pubPathOrPEM := c.GetString("pki.cert", "")
	if pubPathOrPEM == "" {
	return nil, errors.New("no pki.cert path or PEM data provided")
	}
	if strings.Contains(pubPathOrPEM, "-----BEGIN") {
	rawCert = []byte(pubPathOrPEM)
	pubPathOrPEM = "<inline>"
	} else {
	rawCert, err = os.ReadFile(pubPathOrPEM)
	if err != nil {
	return nil, fmt.Errorf("unable to read pki.cert file %s: %s", pubPathOrPEM, err)
	}
	}
	nebulaCert, _, err := cert.UnmarshalCertificateFromPEM(rawCert)
	if err != nil {
	return nil, fmt.Errorf("error while unmarshaling pki.cert %s: %s", pubPathOrPEM, err)
	}

This line ignores the rest return value: nebulaCert, _, err := cert.UnmarshalCertificateFromPEM(rawCert)

I would suggest that we file a ticket to make this a warning, however the next release of Nebula will introduce a v2 format of the certificate, at which point it will be valid to have multiple host certs here for backwards compatibility (a v1 cert and a v2 cert): https://github.com/slackhq/nebula/blob/cert-v2/pki.go#L277-L291

Because of this, it's unlikely that the warning would actually make it into a release. That said, the steps provided in #1292 (comment) are the correct way to perform a CA rotation.

[isoos]

@johnmaguire: Thanks for the further details!

I think nebula shouldn't hide the fact that it ignores parts of its input files. It is better to fail to start when input is not valid or the input parameters are in conflict, but at minimum it should warn about it. Especially since it has a specific subcommand for verifying the config, at least that should fail.

[johnmaguire]

@isoos I agree! The next release of Nebula will allow for multiple certs (v1 and v2), and will warn for other misconfigurations (i.e. two v1s or two v2s): https://github.com/slackhq/nebula/blob/cert-v2/pki.go#L297-L306

[johnmaguire]

Canonical source on rotation here: https://nebula.defined.net/docs/guides/rotating-certificate-authority/