Here at Threat Stack we really like Yubikeys — and they’re a critical part of our security program. Many folks know Yubikeys for their ability to generate one-time codes for use as a second factor. Did you also know you can store certificates on them and use them in your operating system? I’ve written about using the Personal Identity Verification applet on the Yubikey in the past, but now I’d like to take that one step further and use it to identify yourself to a web application. We’ll cover how to do this with a Mac OS X Mojave client — which works nicely with the OpenSC library and an HAProxy reverse proxy.

What is Client Authentication?

Likely, you are already aware of TLS (sometimes called SSL) — this is the technology your computer uses to securely communicate with websites. But TLS supports client authentication as well. Client authentication is a means to provide an identity for yourself — after you verify that the server you’re talking to has the right DNS and a certificate you trust. It’s a handy way to limit access to servers to individuals or devices without having to entirely rely on firewalls.

Getting Started

To begin, you’ll need an already existing public key infrastructure. We know this isn’t a small weekend project, so you may want to check out another article about building your own PKI. This article will still be here when you need it. Crucially, the security of your system will depend on you collecting your hardware certificates after employees leave, and maintaining certificate revocation lists. All of that is outside the scope of this guide, but the article above will help you get started.

You should have a client running macOS Mojave, and a recent version of HAProxy (we tested against 1.8) that is compiled with OpenSSL support.

Configuring the Server

You’ll want to make a few files in your HAProxy configuration directory (besides haproxy.cfg):

  • /etc/haproxy/server.pem has your server TLS information. It should be in the order of public certificate, private key, and then any intermediate certificates.
  • /etc/haproxy/verify.pem has your client CA TLS public certificate, along with the appropriate root certificate. So if you issue keys from “Contoso Users CA-1” then you’ll put that along with the “Contoso Root CA”.
  • /etc/haproxy/crl.pem has your CRL — for all CAs. If “Contoso Users CA-1” is an intermediate CA, make sure you have the CRL for “Contoso Root CA” in this file

In your frontend balancers, add these parameters to your bind line:

bind 10.0.0.1:443 ssl crt /etc/haproxy/server.pem ca-file /etc/haproxy/verify.pem crl-file /etc/haproxy/crl.pem verify required
These options set up the mutual TLS authentication:
  • ssl enables TLS for this port.
  • crt defines the server TLS private key/certificate/certificate chain.
  • ca-file defines the client CA verifier.
  • crl-file defines where HAProxy can find information on certificate revocations.
  • verify required tells HAProxy that it must require a certificate from a client before the request moves on.

Other headers you can add to incoming requests include:

http-request set-header X-SSL                       %[ssl_fc]
http-request set-header X-SSL-Client-Verify         %[ssl_c_verify]
http-request set-header X-SSL-Client-SHA1           %{+Q}[ssl_c_sha1]
http-request set-header X-SSL-Client-DN             %{+Q}[ssl_c_s_dn]
http-request set-header X-SSL-Client-CN             %{+Q}[ssl_c_s_dn(cn)]
http-request set-header X-SSL-Issuer                %{+Q}[ssl_c_i_dn]
http-request set-header X-SSL-Client-Not-Before     %{+Q}[ssl_c_notbefore]
http-request set-header X-SSL-Client-Serial         %{+Q}[ssl_c_serial,hex]
http-request set-header X-SSL-Client-Version        %{+Q}[ssl_c_version]

These headers get passed to your application and can be used to infer certain information about the client’s certificate, like the CN, DN, the output from openssl verify — all helpful for debugging. We’ll talk more about that later.

Your backend configuration can remain the same — these changes only affect the frontend.

Configuring the Client

macOS Mojave ships with support to read the certificates off recent Yubikeys, making it easy to roll this change out to your users. To ensure that’s the case, check the Keychain Access application when you connect a Yubikey configured with certificates. The token should appear as a separate keychain.

Trying It Out

If your HAProxy is running, and you’ve got your Yubikey showing up in Keychain Access, try to go to your webserver! Your browser should prompt you for a certificate that matches the CA that signed it, and let you through. Here’s an example of that prompt:

If it’s your first time using the Yubikey during this session, Chrome will ask you to enter your PIN:

If all goes well, you should get to where you were going — in this case, a test page I set up to output X-SSL-Client-CN and X-SSL-Client-DN:

This is a simple Sinatra application that prints out some headers:

require 'bundler' ; Bundler.require
get '/' do
  client_cn = request.env['HTTP_X_SSL_CLIENT_CN']
  client_dn = request.env['HTTP_X_SSL_CLIENT_DN']
  "Hello #{client_cn} (#{client_dn})"
end
get '/ping' do
  "Hello"
end

Debugging Issues

Sometimes this doesn’t work the first time around — and it can be frustrating to debug something new. One way to begin debugging is to turn off client CA verification on the server side, and then look at the output of the X-SSL-Client-Verify header. Be warned: This will let anyone who presents a certificate through to your application. It’s important in this case to ensure that proper firewalls are in place to limit that access while you debug.

To begin, add ca-ignore-err all to your HAProxy frontend bind line, and make sure you log that header to the HAProxy log with:

http-request capture req.hdr(X-SSL-Client-Verify) len 2

This will capture the output of the OpenSSL verify functions that HAProxy is calling. You can use these to figure out what exactly OpenSSL is having trouble processing to guide your further debugging. Depending on your threat model, ca-ignore-err can allow you to ignore specific OpenSSL error codes. Alternatively, you could log out the headers on the application side as well; outputting request.env with the Ruby application above can help get that information out of headers as well.

Using OpenSSL’s s_client

Unsure if you’re running into a browser or other client issue? Want to get a ton of information about your TLS session? OpenSSL’s s_client is an extremely handy tool to debug TLS connections — and can also help you debug client authentication. One problem with OpenSSL is that it does not have native support for PKCS11 — and the OpenSC libraries are too low level for OpenSSL to just use them. Fortunately, the OpenSC folks made libp11 — a higher level library that can be used with OpenSSL to add PKCS11 support in.

To begin, you’ll need to install GnuTLS, libp11, a recent version of OpenSSL. The yubico-piv-tool is handy, as well. All of these are available in Homebrew. GnuTLS is important because it ships with a handy utility called p11tool — which is the first thing we’ll use to find our RFC7512-compliant PKCS11 URI — an identifier for the key we’ll want to use off the smart card.

Running p11tool --provider=/Library/OpenSC/lib/opensc-pkcs11.so --login --list-privkeys will output the URI of the object on the token you’ll need for future steps. The output will look something like:

Token 'cable' with URL 'pkcs11:model=PKCS%2315%20emulated;manufacturer=piv_II;serial=12341234a12341b;token=cable' requires user PIN
Enter PIN:
Object 0:
URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=piv_II;serial=12341234a12341b;token=cable;id=%01;object=PIV%20AUTH%20key;type=private
Type: Private key (RSA-2048)
Label: PIV AUTH key
Flags: CKA_WRAP/UNWRAP; CKA_PRIVATE; CKA_NEVER_EXTRACTABLE; CKA_SENSITIVE;
ID: 01

Next, export the certificate on your device. OpenSSL’s engine support works with private keys, but does not pull a certificate from a key as well. We’ll use yubico-piv-tool -a read-certificate -s 9a > mycert.pem for this purpose.

Finally, you can enter the OpenSSL console. Make sure to point to your newer version of OpenSSL that supports engines and you can run:

$ /usr/local/Cellar/openssl@1.1/1.1.1c/bin/openssl
OpenSSL> engine dynamic -pre SO_PATH:/usr/local/Cellar/libp11/0.4.10/lib/engines-1.1/pkcs11.dylib -pre ID:pkcs11 -pre LIST_ADD:1 -pre LOAD -pre MODULE_PATH:/Library/OpenSC/lib/opensc-pkcs11.so
(dynamic) Dynamic engine loading support
[Success]: SO_PATH:/usr/local/Cellar/libp11/0.4.10/lib/engines-1.1/pkcs11.dylib
[Success]: ID:pkcs11
[Success]: LIST_ADD:1
[Success]: LOAD
[Success]: MODULE_PATH:/Library/OpenSC/lib/opensc-pkcs11.so
Loaded: (pkcs11) pkcs11 engine
OpenSSL> s_client -engine pkcs11 -keyform engine -key "pkcs11:model=PKCS%2315%20emulated;manufacturer=piv_II;serial=12341234a12341b;token=cable;id=%01;object=PIV%20AUTH%20key;type=private" -cert mycert.pem -connect neato.tls.zone:443 -servername neato.tls.zone -state -debug
engine "pkcs11" set.
CONNECTED(00000005)
(Lots of output removed - all good for debug however)
---
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 4D87C5D67DBB35A72EC56A3963AB9E455865739BFA1B142643D9616E483C071E
Session-ID-ctx:
Master-Key: DD107AAEB642FDCEFA54A27E112AE6176D004AB4D78B81237863E2D3B08FE018CCD015D35942FF91DA21F3E8565000B6
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1568128169
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
GET / HTTP/1.1
Host: neato.tls.zone
Connection: close
HTTP/1.1 200 OK
Content-Type: text/html;charset=utf-8
Content-Length: 75
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Server: WEBrick/1.4.2 (Ruby/2.6.4/2019-08-28)
Date: Tue, 10 Sep 2019 15:31:50 GMT
Connection: close
Strict-Transport-Security: max-age=16000000; includeSubDomains; preload;
Hello cable (/C=US/ST=Massachusetts/L=Boston/O=Threat Stack, Inc./CN=cable)
SSL3 alert read⚠️close notify
closed
SSL3 alert write⚠️close notify
OpenSSL>

If there are issues negotiating the TLS connection, OpenSSL’s s_client will tell you what happened, and your HAProxy logs will be helpful here as well.

Putting It Together

Client authentication over TLS has a lot of prerequisites — and you may not have them today. If you do happen to be leveraging PKI and hardware keys for other purposes like VPN or SSH key management, this is a nice way to give access to web resources without forcing the use of a VPN.

This was a fairly technical deep dive, but if it’s interesting to you, we’re always happy to talk to customers who are doing interesting security things!

This was originally posted on the Threat Stack blog.