Like many of us, I have been using SSH since the beginning of learning how to learn Linux, or some other UNIX-like OS. My usage has moved from password logins, to SSH keys, and now I have picked up on SSH certificates. This is an introduction to using SSH certificates with OpenSSH.

Before I get to my pros/cons, what the heck are SSH certificates? Think of getting an SSL/TLS certificate. First, I would generate my private/public RSA keypair, then a trusted certificate authority confirms my identity, being the owner of the website I want to enable HTTPS on, then they give me a signed public certificate to use alongside my private key, the new public key, and the intermediate certificate. That is similar to how SSH certificates work, except I am the certificate authority (CA).

The way it is done, is I generate two unique SSH keypairs:

  1. To sign my servers' host keys

  2. To sign my users' personal keys

I then tell the servers to trust the user keys I sign and vice versa. So, why should I do this instead of using only a password, or dropping my public key in the authorized_keys file on my server? Well here’s the reasons I found that make this worthwhile:

  • When I create new keys, I don’t have to password login to then add my new SSH key

  • With the trust relationship in place, I don’t have to accept host keys on a new connection

    Note
    		 This basically remedies the TOFU situation (Trust on First Use). But not 100% of the time if you don’t have direct console access to the server. But it will solve the problem for users logging in after the host key signing has been finished.

  • If a host dies, instead of installing all users' public keys, I just sign the host keys and the clients won’t complain. It is already trusted.

  • It makes key expiration policies much more feasible and enforcable.

  • This can be automated pretty easily

And the cons:

  • It’s not as well known as passwords or plain old keys, so there is a learning curve for policy/use changes

  • You probably want 2 dedicated, and preferably offline hosts for key signing

  • Digging into the weeds with principals can be hard to track to get right

  • If you plan on expiring keys, you will want processes in place to manage and sign new keys for other users

Does it sound more complicated than just dumping a key on a server or using a password? Yeah probably. But the scope of certificates go beyond just signing in. It is a system for good security practices. And I argue, does add some convenience in some occasions.

Getting Started

Now that that is out of the way and some of the what’s and why’s are in the open, lets drill down into the application.

First, consider having 2 separate hosts for this job. But if you are just doing this for home use, a single host that isn’t internet accessible should do. But I find it nicer to have 2 virtual machines running that I can turn on/off as I need. From here, I will act on the assumption you have 2 servers.

Host Keys

On host A, create the keypair. If you want to get extra fancy, password protect these keys. That way, if someone somehow got one of the keypairs, they would still need the password. So lets generate the keys

# ssh-keygen -t ed25519 -a100 -f /root/.ssh/hostkey
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase): SuperSpecialAwesomeOptionalPassword
Enter same passphrase again: SuperSpecialAwesomeOptionalPassword
Your identification has been saved in hostkey
Your public key has been saved in hostkey.pub
The key fingerprint is:
SHA256:WaOws18CKpgntRnFSgmJeKWD9utZqjqN8cmaLNtyw10 root@hostkeys
The key's randomart image is:
+--[ED25519 256]--+
|+. ..            |
|+o.+             |
|.o= o .   o      |
|...+   o + .     |
|  +.  + S        |
|.+ +..E+         |
|+B=+.o. . .      |
|*=X.=  . o       |
|BOo=    .        |
+----[SHA256]-----+

Neat. That’s one down. A quick rundown of the flags I specified

  • -t ed25519 sets the key type to ed25519. The default before OpenSSH 9.5 was RSA. So if you are running 9.5 or newer, you don’t need this

  • -a 100 - rounds of KDF used. Makes the private key more brute-force resistant. Useful only if you are password protecting your key

  • -f - tells ssh-keygen where to create the key

User Keys

Now, same as before but on the other host, lets generate the user keys

ssh-keygen -t ed25519 -a100 -f /root/.ssh/userkey
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase): SuperSpecialOtherOptionalPassword
Enter same passphrase again: SuperSpecialOtherOptionalPassword
Your identification has been saved in userkey
Your public key has been saved in userkey.pub
The key fingerprint is:
SHA256:NP4fKqdvbILpwfOhKx9u3voefyKR4+/ugTTjnsj5NY4 root@userkeys
The key's randomart image is:
+--[ED25519 256]--+
|                 |
|                 |
|        o        |
|       o .       |
|        S.       |
|     . o+=       |
|      =+=+= .    |
|    .o+@=O*=..   |
|     *%*E#Xo.    |
+----[SHA256]-----+

Signing Host Keys

Onward to signing our first keys. Let’s start with the host keys.

On each host when you first enable and run the sshd server, it will create some keys for itself. Usually RSA, ED25519, and ECDSA. I only use ED25519, so I will sign only those here in this example. In some cases, I still sign the RSA keys if some clients are doing RSAkeys. These are the default public keys that are to be found on a host:

/etc/ssh/ssh_host_ecdsa_key.pub
/etc/ssh/ssh_host_ed25519_key.pub
/etc/ssh/ssh_host_rsa_key.pub

Copy these to the host that is doing the key signing. Now, this is perhaps where the TOFU I talked about earlier gets defeated. You need to sign into that computer once to get the public host keys. This is a predicament with cloud servers that you do not have console access to. This is only a problem once. After this, all subsequent logins to that server can be trusted. Here’s how the key signing will go:

ssh-keygen -s /root/.ssh/hostkey -I example.com -h -n example.com ssh_host_ed25519_key.pub

Breakdown:

  • -s - the signing key

  • -I - the key identity. Can be used for key revocation

  • -h - specifies that this is a host certificate, not a user certificate

  • ssh_host_ed25519_key.pub - the public key being signed

This generates a ssh_host_ed25519_key-cert.pub file. This needs to be copied back to /etc/ssh on the host we got it from.

Cool. Now what’s next is we need to tell sshd to accept and use this certificate. Let’s add these lines to the /etc/ssh/sshd_config file

HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
TrustedUserCAKeys /etc/ssh/userkey.pub
RevokedKeys /etc/ssh/revoked_keys

  • HostCertificate - the certificate that we just got back from signing the public key

  • TrustedUserCAKeys - this is the /root/.ssh/userkey.pub file created on the User Keys server

  • RevokedKeys - currently, just an empty file. But if any user keys ever need to be revoked for some reason, you’d add the public key here

There’s more that can be done, but I’ll reference it later as bonus material.

Next, run sshd -t which will let you know if a configuration is wrong or not. If all is good, run rcctl reload sshd or service sshd reload or systemctl ssh reload or whatever init system command you need to do.

Signing User Keys

The server is done, so now what? We just told the server that we want to trust keys signed by the Host Certificate Authority. Now our user keys need to be signed, and tell our computer to trust the host certificate.

First generate ssh keys

ssh-keygen -t ed25519 -a100 -f ~/.ssh/mycoolkey

Same as before, optional password, the -t and -a are optional to what you want. I want to get the ~/.ssh/mycoolkey.pub file to the USER key signing server this time. Once there, this is the command I want to run

ssh-keygen -s /root/.ssh/userkey -I me@example.com -n bob -V +180d mycoolkey.pub

The flags:

  • -s - the key used for signing

  • -I - the key ID. This can be useful for tracking logins. You could make this mylaptop or myphone or mycomputer and correlate each key to each sign in

  • -n - this is the permitted username, or principal (more on this later). If my user is bob, I should make this be bob for now

  • -V +180d - this sets an expiration of 180 days after creation. If you do not specify this, it will be valid forever

Now I have a shiny mycoolkey-cert.pub file. This will go in my ~/.ssh directory alongside my other keys.

With the file in place…​I still cannot sign in to the server. Why? I will get a prompt asking me whether or not I trust the destination server. Kind of defeats the purpose. What to do now? This is one of my favorite parts. Instead of having a billion entries in my ~/.ssh/known_hosts file, I can instead litter these with @cert-authority lines! This actually gets much cleaner when you have many servers using the same host key. For example, I have about 70 servers that I signed with the same host key and can use a SINGLE line in my known_hosts file, instead of multiple (one per host key, which will probably be 3, then if I signed in with the IP and then later with the hostname, that can be 4 lines per host!). Ashamedly, I have gotten my known_hosts file over 700 lines from all my sign ins and servers that I nixed and never cleaned out my known_hosts file. So what do I add to my known_hosts:

@cert-authority *.example ssh-ed25519 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILw3E3Pa1CvvdCDQ8EkMpjl9kD72Sxbms9R1QCU8/ni6 root@example.com

To break down this line:

  • @cert-authority - Indicates the public key of the accepted certificate authority

  • *.example.com - any example.com host that is signed with the aforementioned host key

  • everything else - this is the contents of hostkey.pub from the Host Keys section

Signing In

So far, we

  1. Created a user key signing keypair

  2. Created a host key signing keypair

  3. Signed a server’s host keys and told it to use that certificate, and trust the user keys we sign

  4. Signed the user keys, and told our ssh client to trust the host keys that we signed

If all went right, you should be able to sign into your server and NOT get the yes/no prompt to trust that host’s keys. If I signed into that server before, I remove the previous trust entries from my known_hosts file. I do this by making a backup copy, in case I totally dorked stuff. Don’t want to get locked out! If you are getting a yes/now prompt, something went wrong and steps taken should be revisited. From here, you can just go with this. But I want to explore key revocation as well as principals a little.

Key Revocation

There are 2 kinds of key revocation to consider:

  1. Host key revocation

  2. User key revocation

Host Key Revocation

This is useful for SSH clients, if you want to enforce not trusting host keys. Say, if a host’s keys were compromised and you want to ensure your users will not get tricked into logging into a server with the compromised keys. You can add this info to /etc/ssh/ssh_config, or even your own ~/.config file. You would update your Host * section to have this:

Host *
	RevokedHostKeys		/etc/ssh/revoked_hostkeys

or

Host *
	RevokedHostKeys		~/.ssh/revoked_hostkeys

In this file, add the public keys of the host keys you want to revoke. Your ssh client will look to see if the pubkey is in your revoked hostkeys file and cancel the login if the keys are revoked.

User Key Revocation

This is very similar to host key revocation, just the other way around. Earlier, I created a config like in the sshd_config file

RevokedKeys /etc/ssh/revoked_keys

Like before, if a user’s keypair instead gets compromised, you can add the public key to this RevokedKeys file and any login attempts with this keypair will get denied.

Principals

I don’t want to get too deep into principals, but what makes it handy is you can do stuff like say user1 can sign into host1.example.com but not host2.example.com. By default, user1 can sign into any host that has their key signed by the user CA. We can modify user1’s principal to be something like "host1only"

ssh-keygen -s /root/.ssh/userkey -I user1@example.com -n host1only -V +180d mycoolkey.pub

To make this work, a couple tweaks need to be made to our sshd_config file. Namely:

AuthorizedPrincipalsFile /etc/ssh/principals/%u

Next, make the file /etc/ssh/principals/user1 on host1.example.com and have it only contain host1only. Although user1 may exist on host2.example.com, this setup will mandate that user1 has their keypair principal be host1only. This can be useful, for example, if you have a backups user on all your hosts but only want the backups user to be allowed to SSH into only the backup server.

Conclusion

This sounds like a lot, but definitely give it a try if you have a bunch of servers you log into! This is definitely more useful on larger scale deployments, but even in my home of a handful of servers I find it nice to be able to just create a key, sign it, and I can sign into all my hosts with ease, no needing to copy around keys or copying public keys to all my authorized_keys files.

#100DaysToOffload #Post2