"key values mismatch" error after certificate renew

[ikachurin]

Hi Smallstep team! Here is my flow:

1. Getting cert and key with acme provisioner
2. Installing systemd renew service using step renew --daemon command
3. After 16 hours cert is renewed:

<service_ip_here> certificate renewed, next in 15h53m25s

4. Service that uses this cert and key stopping to work with error:

certificate routines:X509_check_private_key:key values mismatch

5. I am trying to troubleshoot with openssl s_server -cert /path/to/service.crt -key /path/to/service.key.
   Getting same error

6. Manually trying to renew with step renew command, getting this error:

error renewing certificate: client POST https://<ca_ip_here>:9000/renew failed: remote error: tls: error decrypting message

So I concluded that either cert or key is broken and it is impossible to renew it and I left this problem for one day

7. Next morning I checked systemd renew service. It continues to renew the cert without errors, but service and openssl command still getting key values mismatch error

I'm at a complete loss as to what is causing this error. It is present on several of my servers with different services

Extra info:

• I am using third-party Ansible Smallstep module to bootstrap the hosts and get acme certs
• My CA in the official docker container
• My services are docker containers too and certs are present as read-only volumes in it

I searched all over the Internet, but I couldn't find the answer. I hope you can help me
Thanks in advance!

[hslatman]

Hey @ikachurin, the error seems to indicate that there's a mismatch between the private key (and its corresponding public key) vs. the public key in the certificate.

You mention that the certificates are in read-only volumes. Do you mean those are the leaf certificates issued by the CA? When they're renewed, are they actually overwritten in that case?

You also mention that you issue the initial certificate using ACME, and then use step ca renew --daemon for renewal. What ACME client are you using to get the initial certificate?

[ikachurin]

Hi @hslatman, I'm sorry I didn't respond sooner.

I found the source of the problem: during the tests I deleted certificates and requested them again, but step ca renew --daemon kept working

Apparently it caches the .key inside the process and when it comes time to renew, step ca doesn't recheck the .key and immediately renews the certificate with the cached one.
I find this behavior to be non-obvious

I think step ca renew --daemon is worth checking the cached and current .key before actually renewing the certificate

I couldn't find a similar issue on your Github. Should I start one?