BTW: I describe slides that are mostly text under them - so no alt text there. If there’s an image or its a section header I’ve added alt text.

I’ve done a number of things with Yubikeys at Threat Stack. Here’s a talk about that.

So, some brief background on myself - i’m a director of SRE at F5 - they bought Threat Stack back in October ‘21. You can follow me on twitter if you like, but I’m not much of a tech poster these days.

The product I help support does many things, but one of the things it does the most of is convert data from the Linux Audit Framework (if you’ve used auditd, or seen the raw logs from it - you recognize the background) and turn it into a nice searchable log that you can alert on. There’s a very old shot of the Threat Stack UI on this slide.

So one of my favorite vendor assessment questions is “How do you manage access credentials including authorization keys in your organization”

And, generally the answer to this is somewhere between “haha what” and “no rules just right” - its a hard issue to tackle at scale, and often there are more important things to get to.

So in today’s talk, we’ll talk a bit about LDAP, summarize PKI, and talk a lot about Yubikeys

So for background on our infrastructure, we have a lot of EC2, with the control plane deployed using Terraform, and the actual machine config managed by Cinc.

One of the reasons we moved to LDAP was because: if Cinc happened to be off on a server (which happens sometimes) - then user accounts dont get created instantly across the network. They also dont get deprovisioned, which is probably more imporrtant

A secondary issue: users can modify their own authorized_keys file. This means they could add keys I’d rahter them not add, like maybe the one on their home desktop that has four different versions of Bonzi Buddy running.

LDAP still works well for Linux authentication though. Your SSO provider may be able to help you run some LDAP if that works for your threat model.

“When you read a security paper, there’s often a sentence near the beginning that says “assume that a public key cryptosystem exists” […] as if establishing a scalable key infrastructure is a weekend project, akin to organizing a walk-in closet or taming a chinchilla” - James Mickens in This World Of Ours

I love this quote, because it’s true. And I’m absolutely going to do that. Sorry.

So yes, setting up a PKI is kind of a pain. It can be super powerful, though!

If you’re looking to use it with your VPN or with Mutual TLS you’re going to have to set one up. SSH keys can work without a PKI.

There are a few ways one could setup up a PKI. I wrote a guide forever ago. A better option is probably something like Smallstep or Hashicorp Vault. You could use CloudHSM – that’s probably ideal for production deployments.

Just make sure to make an intermediate CA.

Okay, so here’s what you came to this talk for.

Alright, you have some form factors. Full USB-A, full USB-C, a mini USB-A, and a mini USB-C. The mini ones just live in the USB port. Not pictured: the one that has a lightning AND usb-c connector on it. But basically, the Yubikey comes in a handfull of shapes sizes and connectors.

And well, it does security things. It supports many different modes, called applets. You can do U2F/FIDO2, One-Time Password, Personal Identity Validation, and OpenGPG.

You may have noticed Personal Identity Validation (PIV) on the last slide. Yep, Yubikeys are smart cards.

If you’re wondering about the specifics of smart cards - they’re everywhere, and wouldnt you know it, they run Java. You see them in your SIM cards, government IDs, laundry cards, and credit cards.

There’s a CCID driver that is how your OS talks to the smart card. The messages you send to the card itself are called APDUs.

Red Balloon Security did a great CTF that had some smart card hacking, if you’re into that kind of thing.

If you’re wondering how smart cards came to be such a thing, this slide is for you. Slide text verbatim since it’s detailed

• 2004: Homeland Security Presidential Directive #12 (HSPD-12)
  • Summary: “Federal identity is a mess: every agency has different ID cards and different standards for issuing them, this is probably bad for Preventing Terrorism™”
  • DHS to Department of Commerce: plz fix
• A year later (2005) NIST FIPS 201, Personal Identity Verification (PIV) of Federal Employees and Contractors was released (revised in 2013 as well)
• It’s 87 pages of How to run an Identity Management System including things like:
  • Background Checks Are Good And You Should Probably Perform Them
  • It’s A Good Idea To Verify a Passport And Something Else Before Issuing an ID
  • Please Don’t Sticker Up Your ID Like You Do With Your Laptop
    • Not a joke! There is wording about covering the ID with stickers.
  • The part we care about is section 4.2: “PIV Card Logical Characteristics”

The Yubikey implements section 4.2: all the keys you could want. You have places to store keys for:

• PIV Authentication (slot 9a on the Yubikey)
  • This is what we’re going to talk about
  • Authenticates card/cardholder; one successful PIN entry can be used multiple times
  • Use for: SSh, TLS Client Auth, VPN
• Digital Signature (9c)
  • Key and certificate use fro document file or executable signing
  • PIN must be provided every time to ensure physical presence
• Key Management (9d)
  • File and email encryption; one successful PIN entry can be used mutliple times
  • macOS uses this for its smart card integration and maybe some other stuff
• Card Authentication (9e)
  • If the physical security system can use yubikeys, youd store the certificate here.
• Attestation (f9)
  • Is this device a valid Yubikey? Find out with the magic of PKI!

Set card identifiers, block the “reset PIN” functionality (incorrect 3 times)

yubico-piv-tool -a set-chuid -a set-ccc
yubico-piv-tool -a unblock-pin -P 000000 -N 000000
yubico-piv-tool -a unblock-pin -P 000000 -N 000000
yubico-piv-tool -a unblock-pin -P 000000 -N 000000

Ask the card to generate a key:

yubico-piv-tool -a generate -s 9a -a RSA2048 -o pubkey

Ask the card to generate a Certificate Signing Request:

yubico-piv-tool -s 9a \
-S '/C=US/ST=Massachusetts/L=Somerville/O=Hackerman, Inc./CN=patc/' \
-a verify -a request -i pubkey -o csr

Great things about what we did here:

• Key never leaves the device
• Certificate is signed by a (hopefully) secure PKI
• Probably difficult to extract the private key from the device unless you want to go and rent a Scanning Electron Microscope or something.
• Similar process for 9d, but you’d generate the key and keep it in escrow (maybe if you’re a company), and write the private key and certificate to the key.

Now lets get the signed certificate on the key:

yubico-piv-tool -a import-certificate -s 9a -i RECEIVED PUBKEY.pem

Changing the management key is probably a good idea, too:

yubico-piv-tool -a set-mgm-key -n $(cat /dev/urandom | tr -dc '0-9' | fold -W48 | head -n1)

Great, so you’re good to go! haha. Right. This is a “draw the rest of the owl” joke

So, it’s a fair question to ask if this particular kind of thing scales. I mean, kinda puts a lot of burden on whoever can sign the certificates. And your users need to be able to use the CLI right?

One way we solved this is by writing a script that gets put into a management tool (JAMF in our case) and the user initiates step 1 which will reset the key and the CSR. Then the files appear in a folder on the desktop, and they send them to the right team, and the right team returns the certificate to the user and instructs them to run step 2.

As you can see from this diagram, if you have your smart card connected, the keys go into the keychain. And if it isnt, the keys disappear. It’s just that simple!

The important thing to get out of this is that the toolkit that makes all this work is called CryptoTokenKit.

And if you’re interested in knowing if the operating system sees your provisioned Yubikey, you can run sc_auth identities. The output looks something like:

SmartCard: org.opensc-project.mac.opensctoken.OpenSCTokenApp.OpenSCToken:87d7410b816e765510bef0d8b83a3379
Unpaired identities:
1B124004FFA52A15E5A4438AD064E843F3776433	Certificate for PIV Authentication (cable)

If you see your certificate in the list, it’s experiencing the mortifying ordeal of being known.

Now, if you want to use this with SSH, you’ll use a PKCS11 connector (that only works with SSH): /usr/lib/ssh-keychain.dylib.

Here’s the console output:

$ ssh-add -e /usr/lib/ssh-keychain.dylib # Disconnect existing cards, error OK
Could not remove card "/usr/lib/ssh-keychain.dylib": agent refused operation

$ ssh-add -s /usr/lib/ssh-keychain.dylib
Enter passphrase for PKCS#11: ******** # 8 digit PIN here
Card added: /usr/lib/ssh-keychain.dylib

$ ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC99FzUnVXEGuFVe7C8sJVpluBklkq/6l31KxOaGuz+iNNc7CLSrL+4daDTmH5bsUzwL3GvKBxTrmkpVsFDCIbd1GcGwqXjdJv2SuNZEUjRoTmP7IR7hgwYW94EO1TYOnmEdNXwPPe3hdrNEiIZN06n4IdhW34fGUWZnxq6nuYc4O2X9dPKWMm9uL5FwvztgN0UgKTiyvCrjMAdGihHP1CPpdL8B5pz1fVypZJKPVmj+hw3lpjgzySmuardMLV6yyLOp+CrGc177me/lB0MfMboEFGQLyGcGGrJ3tlgSH0pIQe1k+BOnBfOMpSQFEIea49JPekCL5BvpwEjOLgP1Wqx

Your server may need libengine-pkcs11-openssl which is part of libp11 now.

This was broken in macOS 11.0-11.3 and you can read more about it on my blog.

Here’s what it looks like for Client TLS - the browser will ask you to pick a certificate that matches the CA the server sends down, then ask for your 8 digit PIN.

You’ll want to download OpenSC. PKCS #11 is the API that lets you talk to these hardware security devices, and is in no way related to Mambo #5. There’s a whole lot of cursed stuff about PKCS11, like any security Thing, but that’s way outside the scope of this talk.

You can use OpenSC with Viscosity on macOS and Windows to use your Yubikey with OpenVPN. Server side, you’ll need to make a folder of trusted CAs and run c_rehash on it, and ensure capath /path/to/folder is set in your OpenVPN server config

If you use the system PIVToken provider (built into macOS) along with a third party PKCS11 provider, both of them will wrestle control of the key. OpenSC does ship a token adapter, which solves this problem. If you want to automate this in JAMF, you could:

1. Bundle /Applications/Utilities/OpenSCTokenApp.appinto a package
2. Run the following as part of a post-install script:

   security smartcards token -d com.apple.CryptoTokenKit.pivtoken
   defaults write /Library/Preferences/com.apple.security.smartcard UserPairing -bool false
   pluginkit -a /Applications/Utilities/OpenSCTokenApp.app/Contents/PlugIns/OpenSCToken.appex
   

If you want to use this on Linux:

• Ensure pcsc-lite, libp11 and OpenSC is installed
• Point applications that support PKCS11 (openvpn, ssh) to the appropriate path for the onepin-opensc-pkcs11.so library
  • ssh-agent does have a path allowlist (ref. CVE-2016-10009)
  • Update the allowlist path to wherever your PKCS11 library is
  • Symlinks need not apply
• Gentoo was an easy slide title joke, but these things were all available with emerge and it worked!

• github.com/buptcza/WinCryptSSHAgent worked with WSL2
• Windows CryptoAPI will want a PKI signed certificate for this to work.
  • I didn’t have luck seeing a self-signed certificate in the certificate manager
• You can use this with Bitlocker for drives connected after boot. It doesnt work in the pre-boot environment.

Okay, Yubikeys are pretty cool. OS support has come a long way! This used to be pretty complicated to use and is now within reach of sufficiently advanced folks.

Questions? Comments? Well, you could ask me on Twitter.