RabbitMQ, Erlang, TLS Validation, and Your LDAP Server

This is on RabbitMQ 3.8.11 with Erlang OTP 23.2.3. I was running OpenLDAP 2.4.48 without any load balancing.

Recently I had the pleasure of upgrading RabbitMQ from an older version to something more recent. On the plus side: all the super important bits worked fine. Data comes in, data goes out - AMQP continued to be extremely AMQP. But there was one annoyance: the web management UI wouldn’t accept my LDAP-backed login.

It’s not often that folks need to log into a web interface for a queuing system. If one’s doing it, it’s probably because there’s some sort of incident going on. A service account could have been used, but getting those credentials might be “2am challenging” in a world where every system in my infrastructure ties in with LDAP somehow and has named user access. So, uh: how do?

Issue 1: Sweet Cipher Suites

The initial issue was that RabbitMQ wouldn’t connect to LDAP server at all. Logs showed:

error,{tls_alert,{handshake_failure,"TLS client: In state hello received SERVER ALERT: Fatal - Handshake Failure\n"}}}

Yet, openssl s_client had no issue connecting to this server, and with that I mistakenly thought this was a client issue. Yep, I know it says SERVER ALERT in big ol’ capital letters, but it worked on RabbitMQ 3.6.x on Erlang 19.1 and now it doesn’t.

It broke because Erlang 21 changed the default cipher suites that are enabled for all TLS connections, and none of the RSA ones made the cut. This ejabberd issue was similar to what I was seeing. Anyways, this is an easy fix once you know what it is. To have OpenLDAP advertise cipher suites that will work with Erlang 21+:

1. Generate a dhparams file
2. Tell OpenLDAP to use it (Set olcTLSDHParamFile)
3. Restart and continue to live life

After that, the connection worked! Except now there’s a new error…

Issue 2: Wildcard Certificate Verification

After I got that issue solved, I still couldn’t log in - but at least I did get a new error message:

{error,{tls_alert,{handshake_failure,"TLS client: In state certify at ssl_handshake.erl:1889 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}"}}}

At least it was definitely a CLIENT issue this time (those caps are the hint). Running openssl x509 on the certificate presented it seemed like the Subject Alternative Names were correct. It was a wildcard certificate, and the hostname matched the wildcard and FQDN.

Turns out, later versions of Erlang require SNI information on connections, which is fine - except if that certificate is a wildcard certificate. The folks who work on Erlang made a decision to not validate wildcard certificates unless you specifically allow it. You can do this by specifying a matching function in a connection’s ssl_options block. That function is:

{customize_hostname_check,[{match_fun, public_key:pkix_verify_hostname_match_fun(https)}]}

Nice, good to go! Except: as you might imagine, referencing a function in a configuration file won’t work. On RabbitMQ 3.8.11 and earlier, you could just turn SNI off - toss {server_name_indication,disable} into the ssl_options block and move on. I don’t love this, but it could be OK depending on your threat model and how your LDAP servers are load balanced¹.

Good news, though: thanks to Michael Klishin’s work, RabbitMQ 3.8.12 will have a couple new settings:

• auth_ldap.ssl_options.hostname_verification can be set to wildcard or none. In my opinion, wildcard is probably fine even if not using a wildcard certificate, but none is the default. In the advanced config, this parameter is rabbitmq_auth_backend_ldap.ssl_hostname_verification.
• auth_ldap.ssl_options.sni can be used to set the hostname to use for SNI. It should match your server name. This could prove to be a challenge if you set multiple LDAP server names in your configuration file. Anyways, if you’re using the advanced config, rabbitmq_auth_backend_ldap.ssl_options.server_name_indication is the option you’re looking for.

Conclusion

Another day, another rabbit hole, but this was a fun dive into a language and system I hadn’t explored very much of. I owe Michael a huge thanks for helping dig into the issue and the super quick patch for this. Hopefully, this is helpful to anyone else encountering this issue - it’s a particularly challenging thing to search for!