Help me understand SSH certificate workflow

[danb35]

I'm having a little trouble understanding how to make SSH certificates work for me. Right now I'm trying to use this at home, and doubtless setting up this infrastructure is overkill to the nth degree, but it sounded like an interesting project, so...

User certificates

Let me address user certificates first. I have a number of devices I need to log in to, some as root and some as dan. On the servers, it looks like I need to copy the user CA cert somewhere, and then add this to the sshd config:

# Path to the CA public key for verifying user certificates
TrustedUserCAKeys /etc/ssh/ssh_user_key.pub

Easy enough, and now that SSH daemon will trust any user certificates issued by the CA. So now I need to get a user certificate.

I understand the next step would be to run step ssh login dan, which (once the OIDC authorization completes--I'm having a little trouble getting my LemonLDAP::NG installation to cooperate, but that's likely an issue for another discussion) will issue a certificate for dan, good for 16 hours or so. If I now ssh dan@host, that cert should be presented, and as long as the host trusts the CA and dan has an account there, I'm in (does that also work if I just ssh host, since my login at the system I use most often is also dan?). But what about root? Do I do a separate step ssh login root to get that cert? Or can I get both names on a single cert in a straightforward way? I understand that two names on one cert would require an admin user, and let's take it as given that I'm giving that user's credentials for the OIDC login.

Host certificates

Usage of these is, I think, quite a bit simpler--add the relevant line to ~/ssh/.known_hosts on any clients, and the client will trust hosts who present a cert signed by the CA. The necessary edits to the sshd config seem straightforward enough as well. What I'm not quite clear about here is getting and renewing the host cert.

I have an assortment of bare metal servers and virtual machines, the latter running under Proxmox (nothing on AWS/Azure/GCP). Initial cert issuance could be interactive, but given the short lifetimes, I'd want renewal to be automated. How would I go about this?

[mmalone]

Hi @danb35!

Sounds like you've got the right high level mental model. Hope you've figured out OIDC! Lemme try to answer your questions.

does that also work if I just ssh host, since my login at the system I use most often is also dan?

Yes. We're not doing anything special here, really. The OpenSSH client assumes your remote username is the same as the local username by default, so if they match you don't need to specify one. If they don't match you can also use ssh_config to specify your username for a specific host, or a group of hosts (if you're curious, man ssh_config is pretty good / you'll want to check out the Host, Match exec and User directives).

But what about root? Do I do a separate step ssh login root to get that cert? Or can I get both names on a single cert in a straightforward way?

Ok, first... you didn't ask, and really it's none of my business, but my personal recommendation is that you probably shouldn't be SSHing as root directly. Generally, it's "best practice" to SSH as a normal user and then use sudo to switch to root. There are reasons to SSH directly as root, though, so I'll assume that really is what you want to do. If anyone finds this via Google or something, however, consider using sudo. Note that some distributions configure OpenSSH by default to not allow SSH as root. If you're using one of those configurations you'll have to change that default.

You could get a separate certificate for root. But that'd probably be annoying.

SSH certificates contain a list of principals. The default behavior of the OpenSSH server is: you're authorized to SSH as user foo if you have user foo in your SSH principal list. With that in mind, you have two other options:

1. You can ask step-ca to issue you a certificate with two principals. This should be as simple as step ssh certificate [email protected] dan --principal dan --principal root.
2. You can change the default authorization policy to allow certificates with principal dan to SSH in as user root. To do this you'll want to use either AuthorizedPrincipalsFile or an AuthorizedPrincipalsCommand in your server's sshd_config (check out man sshd_config for details on these config directives). This can be a little confusing if you're new to it, so if you go this route and get lost lemme know and I can probably dig up an example.

Getting / renewing host certs

To get an SSH host cert you can use most of the step-ca provisioners (ACME is the only one that doesn't work, I think). The simplest option is probably to use a JWK provisioned with a password. You could also generate one-time tokens using the JWK provisioner. If you're on the cloud you could also use instance identity docs to authenticate to the CA and get a cert.

To renew you can run step ssh renew --force /path/to/ssh_host_key-cert.pub /path/to/ssh_host_key. Obviously you'll want to automate this. The easiest solution is probably to use systemd. A really basic example might use a systemd service definition like:

[Unit]
Description=Step SSH certificate renewer

[Service]
ExecStart=/usr/bin/step ssh renew --force /etc/ssh/ssh_host_ecdsa_key-cert.pub /etc/ssh/ssh_host_ecdsa_key

Paired with a systemd timer to fire the service periodically, like:

[Unit]
Description=Step SSH renewer timer

[Timer]
OnBootSec=60
OnUnitActiveSec=28800
AccuracySec=1

[Install]
WantedBy=multi-user.target

If you want to tweak this configuration we've got detailed instructions in docs on automated renewal for x509 certificates. Renewal for SSH certificates is gonna be almost identical, but you'll use step ssh renew instead of step ca renew.

If you're going this deep, you should probably check out our SSH product, too. It handles a lot of this for you, and it's free forever unless you connect an external identity provider (if you don't connect an identity provider we act as the identity provider for your single user account, so this works fine as long as it's just for you). Even if you don't end up using it, it may be worth setting it up and running through configuration on your client and on a VM somewhere. We handle a lot of this stuff in the product workflows, so it may help you wrap your head around everything (and you can probably reverse engineer some stuff ;).

Lemme know if you have any follow-up questions. Good luck!

[tashian]

I'm glad it's working so well for you!

All of these things are necessary for issuing a certificate. Here are three options for simplifying things:

• Set env variables somewhere instead of passing flags (see passing flags as environment variables)
• Use the step config file (see configuration file)
• Or use a shell alias

@mmalone @maraino this use case makes me wonder if defaults.json should support defaults by subcommand, or by provisioner?

[tashian]

Also @danb35 given your journey with this, if you have any suggestions for ways we could improve the docs, we're all ears. :D

I think one of the challenges of the step toolchain is its versatility.
Would a better overview/illustration of a typical SSH workflow (w/user and host certs) have been helpful?

[danb35]

if you have any suggestions for ways we could improve the docs, we're all ears.

That's a good question. I'm not sure I have as well-thought-out an answer as I'd like to, but I expect at least part of the issue is that I'm approaching this as someone not thoroughly familiar with the software, who sees a cool thing it can do and wants to adapt it to his own environment. IOW, I'm fairly task-focused. It seems the bulk of the docs are more of a "command reference" sort, with a lot of the "how-to" material being in blog posts.

FWIW, in the vein of "how-to", I've written up a few things. First, on the setup for user/host SSH certificates:
https://www.familybrown.org/dokuwiki/doku.php?id=advanced:ssh_certificates
Converting the Raspberry Pi-based "tiny CA" to handle SSH certificates:
https://www.familybrown.org/dokuwiki/doku.php?id=advanced:ssh_conversion

And finally, the matter that interested me about SSH certs in the first place--I saw the blog post and video about using Google's SSO to authenticate for them, and wanted to self-host that. This page describes installation of LemonLDAP::NG to provide OIDC (as well as other protocols), as well as configuration for a variety of relying applications including Step. The link below goes to the section talking about OIDC; the Step configuration is after Proxmox.
https://wiki.nethserver.org/doku.php?id=userguide:llng#openid_connect

[logopk]

Hi @mmalone

To renew you can run step ssh renew --force /path/to/ssh_host_key-cert.pub /path/to/ssh_host_key. Obviously you'll want to automate this. The easiest solution is probably to use systemd. A really basic example might use a systemd service definition like:

[Unit]
Description=Step SSH certificate renewer

[Service]
ExecStart=/usr/bin/step ssh renew --force /etc/ssh/ssh_host_ecdsa_key-cert.pub /etc/ssh/ssh_host_ecdsa_key

is it correct that there is no such thing as --expires-in 4h for ssh certs as in x509 renew?

Then any cron or systemd unit has to force renewal of the host (ssh client) cert? Correct? With the consequence that we get many unnecessary new host certs?

Shouldn't --expires-in 4h be possible?

Thanks for looking into this and your very detailed explanation how to get host certs running.

Peter

[maraino]

@logopk, I'd suggest you add a new issue about adding --expires-in to step ssh renew, something that behaves similarly to step ca renew:

      --expires-in=duration
          The amount of time remaining before certificate expiration, at which
          point a renewal should be attempted. The certificate renewal will not
          be performed if the time to expiration is greater than the --expires-in
          value. A random jitter (duration/20) will be added to avoid multiple
          services hitting the renew endpoint at the same time. The duration is a
          sequence of decimal numbers, each with optional fraction and a unit
          suffix, such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns",
          "us" (or "µs"), "ms", "s", "m", "h".