Hey! Listen! This looks like it’s been fixed in macOS 11.4! You can read the rest of the post for a fun deep-dive though.

Using a Yubikey for user identity is great. Your Yubikey generates a private key that never leaves the device, outputs a CSR that your IT org can sign, and boom you’ve got a pretty solid story around user identity. For a while macOS has shipped an SSH agent that (I think?) accesses the Yubikey’s certificates through the Keychain and/or CryptoTokenKit API. It is helpfully named /usr/lib/ssh-keychain.dylib.

I had successfully tested that my Yubikey worked with my VPN, and that in-browser mutual TLS auth worked as expected. It seemed like a slam dunk to give SSH the old college try, until…

$ ssh-add -s /usr/lib/ssh-keychain.dylib
Enter passphrase for PKCS#11:
Could not add card "/usr/lib/ssh-keychain.dylib": agent refused operation

Oh, uh… ok… why’s that?

$ ssh-agent -d
...
debug1: process_message: socket 1 (fd=4) type 20
failed PKCS#11 add of "/usr/lib/ssh-keychain.dylib": realpath: No such file or directory

Fascinating. So, the library is not in /usr/lib (or anywhere in /usr). Yet, you could run man ssh-keychain and read more about using this PKCS11 Provider. I found this amusing and tweeted something to the effect of “haha, macos 11 kills /usr/lib/ssh-keychain.dylib but keeps the developer documentation. at least i can read about what used to work!”

I posted about this in a few places, and fortunately mendel chimed in to say “some libraries live only in the cache in macOS 11, but not in the filesystem, might it be that?” and linked me to two blog posts:

It turns out a thing that changed in macOS Big Sur is that all the system libraries live in a cache located in /System/Library/dyld/dyld_shared_cache_x86_64. When an application calls dlopen(), libc goes and pulls the library from the cache instead of the file system. This is interesting from a security perspective, but: is my SSH provider hiding in that cache too?

I ended up doing the steps in the second post to build dyld_shared_cache_util, which was a lot of fun as a first time Xcode user1. But, fortunately the effort paid off because I did find ssh-keychain.dylib in the cache. The existence of the man page (and the bug) now made way more sense.

In OpenSSH 7.5 and later, OpenSSH checks that whatever PKCS11 libraries you pass it are on an allowlist to mitigate CVE-2016-10009. If the library or it’s path is on the allowlist, ssh-agent passes the library to ssh-pkcs11-helper which actually calls libc’s dlopen(). But, since the library doesn’t exist in /usr/lib in the first place, my guess is that it never makes it that far.

Hopefully Apple has a fix for this in an upcoming release, since I preferred relying on the OS for token access. In the meantime, dropping OpenSC’s OnePIN PKCS11 library into /usr/local/lib works fine.

  1. It was not fun. ↩︎