In the past I was a bitesnap user. It was an okay(?) enough app to capture what I eat without focusing too much on the nutritional facts label. But getting the data out of the app has two options: JSON or CSV. Ultimately I’d like to be able to build a nice-ish styled report from the app, but it seems like the folks who build it are busy working on their main bite.ai product.

“I can probably get a layout that works well enough for ‘print to PDF’ with some javascript and a single HTML file” was my first thought. But then I realized that I am a child when it comes to the front-end JS ecosystem. It’s not that I couldn’t figure it out, but it was going to take a bit longer to make that magic happen. So I did what people do when they need a drill but instead have a wacky-waving inflatable arm-flailing tube thing: I wrote a lambda function in golang, deployed it using lambda + api gateway and lived to tell the tale.

This didn’t actually seem like it would have been that challenging, except between outputting a HTML template that worked well enough and then figuring out how to get it to play nice with an HTTP API Gateway was a bit of a challenge!

What Challenge?

My app worked OK when I ran fmt.Printf on a template - but when that HTML was returned through the Lambda handler I saw a bunch of unicode codepoints (think \u003c and \u003e everywhere). Turns out the API Gateway Proxy event types give more flexibility on what my function would return. At one point I got my HTML out of the app, but none of the other bits. Progress!

OK, so… output. I had been testing my API with curl -d@bitesnap-data.json and not curl -F 'data=@bitesnap-data.json – which, the latter would have better emulated a web form. This lead me down a quite the journey of multipart mime/form processing in Go. An extremely helpful block of code from Adam Drake’s Serverless with Lambda, API Gateway, and Go saved me a ton of time. Recreating the incoming request parameters inside of an http.Request{} struct made it easier to use the built-in Golang functions for parsing multipart forms.

Lastly: S3 still doesn’t do HTTPS on the edge! I tried to proxy the S3 bucket with the static pages in it but it’d only work if I didnt verify TLS end-to-end. That might be OK-enough for this use case, except the only way the API gateway would work with a fancy hostname and proxying is if I did verify TLS. So, I got forced into setting up some Cloudfront.

Anyways, it works now.

Is… Is it safe?

Probably? Like, the lambda only has a max execution time of 1 sec, and 128mb of RAM, an extremely low rate limit. Folks are welcome to set up their own rube goldberg machines since you can view the source on GitHub.

Recently I had the pleasure of upgrading RabbitMQ from an older version to something more recent. On the plus side: all the super important bits worked fine. Data comes in, data goes out - AMQP continued to be extremely AMQP. But there was one annoyance: the web management UI wouldn’t accept my LDAP-backed login.

It’s not often that folks need to log into a web interface for a queuing system. If one’s doing it, it’s probably because there’s some sort of incident going on. A service account could have been used, but getting those credentials might be “2am challenging” in a world where every system in my infrastructure ties in with LDAP somehow and has named user access. So, uh: how do?

Issue 1: Sweet Cipher Suites

The initial issue was that RabbitMQ wouldn’t connect to LDAP server at all. Logs showed:

error,{tls_alert,{handshake_failure,"TLS client: In state hello received SERVER ALERT: Fatal - Handshake Failure\n"}}}

Yet, openssl s_client had no issue connecting to this server, and with that I mistakenly thought this was a client issue. Yep, I know it says SERVER ALERT in big ol’ capital letters, but it worked on RabbitMQ 3.6.x on Erlang 19.1 and now it doesn’t.

It broke because Erlang 21 changed the default cipher suites that are enabled for all TLS connections, and none of the RSA ones made the cut. This ejabberd issue was similar to what I was seeing. Anyways, this is an easy fix once you know what it is. To have OpenLDAP advertise cipher suites that will work with Erlang 21+:

1. Generate a dhparams file
2. Tell OpenLDAP to use it (Set olcTLSDHParamFile)
3. Restart and continue to live life

After that, the connection worked! Except now there’s a new error…

Issue 2: Wildcard Certificate Verification

After I got that issue solved, I still couldn’t log in - but at least I did get a new error message:

{error,{tls_alert,{handshake_failure,"TLS client: In state certify at ssl_handshake.erl:1889 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}"}}}

At least it was definitely a CLIENT issue this time (those caps are the hint). Running openssl x509 on the certificate presented it seemed like the Subject Alternative Names were correct. It was a wildcard certificate, and the hostname matched the wildcard and FQDN.

Turns out, later versions of Erlang require SNI information on connections, which is fine - except if that certificate is a wildcard certificate. The folks who work on Erlang made a decision to not validate wildcard certificates unless you specifically allow it. You can do this by specifying a matching function in a connection’s ssl_options block. That function is:

{customize_hostname_check,[{match_fun, public_key:pkix_verify_hostname_match_fun(https)}]}

Nice, good to go! Except: as you might imagine, referencing a function in a configuration file won’t work. On RabbitMQ 3.8.11 and earlier, you could just turn SNI off - toss {server_name_indication,disable} into the ssl_options block and move on. I don’t love this, but it could be OK depending on your threat model and how your LDAP servers are load balanced¹.

Good news, though: thanks to Michael Klishin’s work, RabbitMQ 3.8.12 will have a couple new settings:

• auth_ldap.ssl_options.hostname_verification can be set to wildcard or none. In my opinion, wildcard is probably fine even if not using a wildcard certificate, but none is the default. In the advanced config, this parameter is rabbitmq_auth_backend_ldap.ssl_hostname_verification.
• auth_ldap.ssl_options.sni can be used to set the hostname to use for SNI. It should match your server name. This could prove to be a challenge if you set multiple LDAP server names in your configuration file. Anyways, if you’re using the advanced config, rabbitmq_auth_backend_ldap.ssl_options.server_name_indication is the option you’re looking for.

Conclusion

Another day, another rabbit hole, but this was a fun dive into a language and system I hadn’t explored very much of. I owe Michael a huge thanks for helping dig into the issue and the super quick patch for this. Hopefully, this is helpful to anyone else encountering this issue - it’s a particularly challenging thing to search for!

Using a Yubikey for user identity is great. Your Yubikey generates a private key that never leaves the device, outputs a CSR that your IT org can sign, and boom you’ve got a pretty solid story around user identity. For a while macOS has shipped an SSH agent that (I think?) accesses the Yubikey’s certificates through the Keychain and/or CryptoTokenKit API. It is helpfully named /usr/lib/ssh-keychain.dylib.

I had successfully tested that my Yubikey worked with my VPN, and that in-browser mutual TLS auth worked as expected. It seemed like a slam dunk to give SSH the old college try, until…

$ ssh-add -s /usr/lib/ssh-keychain.dylib
Enter passphrase for PKCS#11:
Could not add card "/usr/lib/ssh-keychain.dylib": agent refused operation

Oh, uh… ok… why’s that?

$ ssh-agent -d
...
debug1: process_message: socket 1 (fd=4) type 20
failed PKCS#11 add of "/usr/lib/ssh-keychain.dylib": realpath: No such file or directory

Fascinating. So, the library is not in /usr/lib (or anywhere in /usr). Yet, you could run man ssh-keychain and read more about using this PKCS11 Provider. I found this amusing and tweeted something to the effect of “haha, macos 11 kills /usr/lib/ssh-keychain.dylib but keeps the developer documentation. at least i can read about what used to work!”

I posted about this in a few places, and fortunately mendel chimed in to say “some libraries live only in the cache in macOS 11, but not in the filesystem, might it be that?” and linked me to two blog posts:

• Reverse Engineering macOS 11.0
• Extract the system libraries on macOS Big Sur

It turns out a thing that changed in macOS Big Sur is that all the system libraries live in a cache located in /System/Library/dyld/dyld_shared_cache_x86_64. When an application calls dlopen(), libc goes and pulls the library from the cache instead of the file system. This is interesting from a security perspective, but: is my SSH provider hiding in that cache too?

I ended up doing the steps in the second post to build dyld_shared_cache_util, which was a lot of fun as a first time Xcode user¹. But, fortunately the effort paid off because I did find ssh-keychain.dylib in the cache. The existence of the man page (and the bug) now made way more sense.

In OpenSSH 7.5 and later, OpenSSH checks that whatever PKCS11 libraries you pass it are on an allowlist to mitigate CVE-2016-10009. If the library or it’s path is on the allowlist, ssh-agent passes the library to ssh-pkcs11-helper which actually calls libc’s dlopen(). But, since the library doesn’t exist in /usr/lib in the first place, my guess is that it never makes it that far.

Hopefully Apple has a fix for this in an upcoming release, since I preferred relying on the OS for token access. In the meantime, dropping OpenSC’s OnePIN PKCS11 library into /usr/local/lib works fine.

It’s a neat tool, but I’m pretty sure I’m still the only user. That makes sense: some assembly is required, it requires Micro Snitch, and I don’t have a UI to go with it. But also: I didn’t have a package download for it; adventurous users had to download a Golang toolchain to build OnAir. Making someone download a build toolchain to use your app is mean in the year 2020 I’m told, so I wanted to fix that.

While reading about how the code signature and notarization processes work on macOS, somehow I came across Little Snitch’s Internet Access Policies and thought it’d be cool to include one. The tl;dr on IAPs are: you can compile your application with some added context about what calls it makes out to the Internet, and that context will show up when Little Snitch prompts you about a connection your application is making. It looks like this:

Little Snitch tells you that OnAir wants to connect to discovery.meethue.com on port 443. It also mentions what OnAir does (“OnAir changes the color of IP-based lighting when you are on a audio or video call”), a link to the Github page, and a specific statement about connections to the meethue.com domain. In the image above, it reads “OnAir uses the Hue Broker Server to discover Hue Bridge devices on your network. If denied, OnAir will not be able to find the Hue Bridge device on your network.”

Useful security information is useful. More applications should probably have useful information. Except, getting that useful information into a single binary when you’re not using Xcode is trickier than I anticipated. So how do? Enough of the recipe intro: here’s how to do the thing.

Building Your Personal Plist Pizza

An Info.plist file has attributes about the application. You’ll need to make one.

There are tools that can help with this like PlistEdit Pro. In this case “it’s just xml” so you’d be able to use what I threw together as a template for your project. PlistEdit Pro was nice to use, though.

If you’re wondering what some of the CoreFoundation (CF) keys and values are, Apple’s Core Foundation Keys page has the answer. Though, you probably only need the ones listed in the Code Signing Tasks doc.

We’re here to talk about IAPs, though: so take a look at the specification on Little Snitch’s Research Assistant developer documentation. To summarize, you’ll need to add a dictionary entry for each endpoint your application connects to. The text that shows up for the IAP is put into the Purpose and DenyConsequences fields.

You can also reference a string that is localized, which is a nice thing to do in general. The OnAir Info.plist does this - though only for English today².

The One Weird Linker Trick

OK, so you’ve made your XML. How can you build it into your application?

Generally, an Info.plist lives as inside of an app bundle. However, it is also possible for the data to live as a text segment in a Mach-O binary. We’re going to do that with the power of some linker flags.

1. Add something like -ldflags='-extldflags "-sectcreate __TEXT __info_plist $(shell pwd)/Info.plist"' to your go build command. I copied this from my Makefile, you may need to do something special to get the full path to the file.
2. Toss an import "C" into your application’s imports.

The import "C" thing is important, even if you’re not doing Cgo things in your application. If you don’t import it, the ldflags component will be ignored, and you won’t see your IAP.

You can confirm that the text segment made it in to the binary by running otool -s __TEXT __info_plist ./path/to/your/binary. Convert it to human-readable text by piping the output to xxd -r.

I talk about this in the context of a Golang application - but this is a linker flag. I imagine you can pass the sectcreate flag to whatever your build toolchain uses and you’ll get that __info_plist section.

License to Distribute

Little Snitch will only display an IAP for a signed application, so get your credit card out. Apple’s Developer Program costs $99 a year, but: you get unlimited certificates! This is an unverified claim, but I have at least two (2) certificates now, which feels like a lot for me.

After your account is approved (it took ~10 min for me on a Saturday morning), open up Xcode, go to Preferences, pop into the Accounts tab, and add your Apple ID. Then, click Manage Certificates... and generate you a Developer ID Application certificate, and possibly a Developer ID Installer certificate. These will be saved directly into your keychain, though you should export them with a secure password and put them somewhere safe.

I should note that the certificate generation step is different if you have an enterprise account; I think you’ll need to send a Certificate Signing Request (CSR) to the person who manages your account. I gleaned this from googling, so at least something has been written about it if you find yourself lost at this point. You’ll probably need the --acr-provider option with altool later.

Since you probably have MFA turned on for your Apple ID, you’ll need an “Application Password” which you can make over on the (Apple ID Management Portal](https://appleid.apple.com). Kinda funny that the developer tools don’t support native MFA, but security is hard. Anyways, once you’ve got that one-time password, get it into your keychain under the id appleid by running:

xcrun altool --store-password-in-keychain-item appleid -u my@cool.email -p asdf-asdf-asdf-asdf

Signing Your Binary

Run security find-identity -v and get the hash for your Developer ID Application certificate. Sign your binary with codesign -s DEV_APP_HASH_HERE -f -v --timestamp --options runtime BINARY_PATH_HERE. Congratulations: you made a signed binary on macOS.

At this point, if you delete the Little Snitch rules that exist for your program and then run it, you’ll probably see the words that you wrote in the Little Snitch alert dialog box. Congrats!

Notarizing Your Binary

That’s fine and good, but let’s say you want to distribute your application. Apple’s notarization process seems to be a very quick static analysis pass on your binary to make sure that you’re not distributing obvious malware. So I tried to zip up my binary and ship it off to Apple… without much luck. It worked once, at least?

So instead I built a package. I moved my signed binary into an empty folder creatively named package/ and made an installer with this pkgbuild:

pkgbuild --root package --identifier com.example.yourapp.dontusethis.makeyourown --version $(VERSION:v%=%) --install-location /usr/local/bin onair-$(VERSION)-raw.pkg

Signed the package with my developer installer key (grab that hash using security find-identity -v):

productsign -s DEV_INSTALL_HASH_HERE --timestamp onair-$(VERSION)-raw.pkg onair-$(VERSION).pkg

And submitted for notarization ³

xcrun altool --notarize-app --primary-bundle-id com.example.yourapp.dontusethis.makeyourown --username 'my@cool.email' --password "@keychain:appleid" --file onair-$(VERSION).pkg

Notarization takes a little bit of time: generally <5 min, but could be longer than an hour, too. Your request will get a UUID, which you can use to check on the status of notarization, or you can wait for the email. For the impatient, keep asking Apple for an update using this command:

xcrun altool --notarization-info UUID_HERE -u 'my@cool.email' --password "@keychain:appleid"

If it passes, then staple your notarization to your package (xcrun stapler staple onair-$(VERSION).pkg) and distribute. I just uplodaded my package as part of a Github release.

Embedded Data? In A Binary?

Well, that’s about it. If you’re building a tool for users who might use Little Snitch, this is a nice way to give additional context on what your app is connecting to. Maybe there are other things you can put in this header that get parsed by the OS too! 😜

Just one thing: before I dive into using this tool, it’s definitely not going to work for my needs if I can’t tie it in with an authentication provider. The documentation hints at supporting Open ID Connect (OIDC)/Proof Key for Code Exchange (PKCE), but… how? And with my Open ID Connect provider? I’ve never OIDC’ed before, what config values does Dispatch actually need? What do they look like? Why is the app ignoring the OIDC codepath for this even though I (thought I) set the right environment variables? Is what I’m trying to do impossible?

Well, It wasn’t impossible. It did, however, feel like pushing a boat up a mountain. I definitely lost at least an hour (it was more than an hour) because I put a trailing slash on a URL. This resulted in a CORS error, which was a very exciting, new, and fun way to pull my hair out. Thankfully my hair has regrown since.

OIDC Setup

Right, the first step is getting an OIDC connector set up.

You’ll want to set up a new OIDC connector. Perhaps you use an enterprisey authentication provider like OneLogin - which separates out connectors from the apps that use connectors. Go set up a connector, or ask the person who can set up connectors and apps in your enterprisey authentication provider to do that. You’ll need your base URL for your Dispatch setup to set up a connector. These values should do:

Then, create a new “app” using that connector. On the SSO page, set:

And then copy your Client ID, and the URL for the OIDC “Well Known Configuration”, which you’ll need later.

Patching Dispatch

Now for the code bits. We’re going to make a “custom” authentication provider and patch in an option for decoding JWTs. I say “custom” because it’s a pretty small change to the provided PKCE authentication provider.

Go clone the Dispatch repository for these next steps.

Making Your Custom Authentication Provider

I absolutely missed that big ol’ warning in the Dispatch documentation that VUE_APP settings are used during the build of the UI/frontend, so I kept trying to set these parameters as part of the backend’s environment. Don’t be like me! Instead, put these values in src/dispatch/static/dispatch/.env:

VUE_APP_DISPATCH_AUTHENTICATION_PROVIDER_SLUG=""
VUE_APP_DISPATCH_AUTHENTICATION_PROVIDER_PKCE_CLIENT_ID=(the client id from earlier, without these parenthesis)
VUE_APP_DISPATCH_AUTHENTICATION_PROVIDER_PKCE_OPEN_ID_CONNECT_URL=https://your-cool-org.onelogin.com/oidc/2

You’ll notice that VUE_APP_DISPATCH_AUTHENTICATION_PROVIDER_SLUG is empty - this allows the use of a custom authentication provider. Speaking of a custom authentication provider, you can drop this in src/dispatch/static/dispatch/src/auth/. It’s just about the same as the pkceAuthProvider.js with some more values added to the request that makes it to the backend, I think. You should diff the two though, since I am a stranger on the internet who is giving you authentication code.

One Weird JWT Decode Trick

You may also need to incorporate changes that are in my branch of Dispatch. I made a PR to get those into the main project; we’ll see where it goes. I added a log.debug() on the JWT decode as well, so if you run into JWT issues you can bump the log level and get some valuable information.

Docker Docker Docker

Once you’ve gotten the bits in place, build the container. Put it somewhere it can be used, etc.

App Runtime Config

Since our “custom auth provider” is still OIDC/PKCE, we can use the existing backend variables.

Set these parameters in the app config. DISPATCH_PKCE_DONT_VERIFY_AT_HASH is from the patch I mentioned in the other section.

DISPATCH_AUTHENTICATION_PROVIDER_SLUG: dispatch-auth-provider-pkce
DISPATCH_AUTHENTICATION_PROVIDER_PKCE_JWKS: https://myorg.onelogin.com/oidc/2/certs
DISPATCH_JWT_AUDIENCE: (the client id from earlier, without these parenthesis. But you don't have to specify this, and you may not need to.)
DISPATCH_PKCE_DONT_VERIFY_AT_HASH: "true"

Start your app. Go to the main page. You should be logged in! Wow!

Fin

Funny enough, I set this up as a way to get more familiar with some kubernetes infrastructure that I’ve been playing with. Mission accomplished: debugging an application in a brand new environment is definitely one way to learn the joys of debugging. But honestly, I would not have figured any of this out without this issue and this PR.

Happy Incidenting!

But, distributing yubico-piv-tool on the mac is a pain because it links against OpenSSL. You can get OpenSSL through Homebrew, but, maybe you don’t want to force everyone to install homebrew. And, maybe you need OpenSC to let other tools that speak PKCS11 use the Yubikey. Also, maybe you have Jamf and just want to give folks a one-click install. Many maybes here.

Anyways, if this is your world, this script will build a macOS package file that installs all the things into /usr/local/vpnkit and /Library/OpenSC.

The userdata script is a toml file. Save this as userdata-bottlerocket.tpl in your terraform folder:

[settings.kubernetes]
api-server = "${endpoint}"
cluster-certificate ="${cluster_auth_base64}"
cluster-name = "${cluster_name}"

Then, in the place you call the terraform-aws-eks module, fill in the AMI (or use the search parameters option) and reference the userdata file:

module "my-cluster" {
  cluster_version        = "1.17"
  ...
  worker_ami_name_filter = "bottlerocket-aws-k8s-1.17-x86_64-v*"
  worker_ami_owner_id    = "092701018921"
  worker_groups = [
    {
      ...
      userdata_template_file = file("userdata-bottlerocket.tpl")
      ...
    }
  ]
}

Recently, we noticed that every Sunday morning we’d encounter a backup in writing these raw events to one of our databases. Writes would stop for about 10 minutes, our processing queues would back up, and our on-call engineer for that week would get paged. Generally, this kind of thing would clear itself up with some light human interaction… until the following week. Ideally, you want on-call activities to be actionable and infrequent, so this was a fast ticket to engineer burnout. How’d we fix the event?

Weekly Events

It’s easy to get used to looking at the application code for bugs when you’re lucky enough to have access to it, as is the case for our internal services. But this was a red herring: in the end, the issue was related to lower-level system configuration. After we determined that the system wasn’t receiving any load spikes from customers, we started poking around the system itself, and that’s where we found a few weekly cron jobs that run by default.

One of these jobs will update the man page database, another one would check to see if there was a new distribution version available, and the third one ran a command called fstrim. This was initially unremarkable to us – we were running in a cloud environment where we’re not selecting our own disks or controllers, so surely something outside of the OS was handling this important function.

As you may guess, this post is here to tell you: the OS was handling this important function in a way that we did not expect – the fstrim weekly cron command!

A Brief History of Deleting Files

When you delete a file on an actual spinning hard drive, the file gets marked with an identifier that the position on the disk is available for use. Over time, the bits that make up your file get overwritten with the bits of your new files, regardless of their physical position on the disk. Sometimes you could run an optimization tool like a disk defragmenter to make sure that all the individual blocks that made up a file were in order. This would increase read performance, and keep users happy with their desktop systems. In the server space, people running specialized commercial database software could (and did) increase performance by putting frequently accessed data as close to the platter as possible.

Solid State Drives changed this dynamic. SSD controllers abstracted away much of the “physical drive” bits to be compatible with existing computers. SSD’s store files in “pages” vs. block locations on a spinning platter. And, pages need to be contiguous – so the SSD needs to reserve some space to be able to move files around when you create (or delete) files. To facilitate this action, a new hardware operation called TRIM was implemented. This operation was a way to let your operating system tell your SSD to free up space for a file that was no longer needed.

Implementing TRIM

The TRIM operation can be sent to the SSD with every delete (run continuously) or it can be run in batches. On a Linux system, you can use the mount option discard to run the TRIM operation continuously. Alternatively, you can use the fstrim command to run it all at once. We had not mounted our drive with the discard option, and had thought a controller might be handling this operation for us – on our directly attached storage provided to us by our cloud provider. We ran fstrim on a single database node, and writes stopped for a certain amount of time. We realized two things at this point:

1. CPU IO Wait spiked up as did drive writes.
2. There wasn’t any sort of more advanced storage controller between us and the drives – they were in fact (and as advertised) directly attached.
3. Running fstrim weekly was the source of our Sunday morning pain.

We deployed configuration to add that flag, and the issue was fixed. Right? Well, not quite – in fact, we posted a service availability note, because deploying that configuration caused delays in making raw events searchable. We reverted and processed the backlog at expected speeds, but we were disappointed: wasn’t continuous TRIM the solution to our Sunday morning wake up call? It was only part of the issue.

How to Store an Event

We leverage a search-optimized database for storing events for later analysis. This database supports storing data across multiple folders – however, we found that writes were not evenly balanced across the drives in that configuration, so we made a RAID 0 array of both drives and had the database use that. We chose RAID 0 because we built the system to replicate data across availability zones and different nodes. Disk errors on one or many nodes can be solved in software – when a node fails, a new node replaces it and any replicas are recreated on the new node.

When initially investigating the disk issue, I wanted to know if the drives actually supported the TRIM operation in the first place. Documentation said yes, but it’s always good to double check. When we ran lsblk -D, the linux command which prints information about a drive’s support for the TRIM operation, the output looked like this:

NAME        DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
nvme0n1            0        0B       0B         0
└─nvme0n1p1        0        0B       0B         0
nvme1n1          512      512B       2T         0
└─md0              0      512B      32K         0
nvme2n1          512      512B       2T         0
└─md0              0      512B      32K         0

We had implemented our RAID 0 array using Linux’s multi-device RAID driver, which creates a single logical device (/dev/md0 in this case) that can be formatted and accessed like a normal disk. We expected the root drive to not support the TRIM operation, since it is network attached block storage. Seeing this confirmed that we did have actual drives attached directly to the system.

In another part of our infrastructure, we had used LVM to set up a RAID 0 device. Interestingly to us the lsblk -D output showed:

NAME                     DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
nvme0n1                         0        0B       0B         0
└─nvme0n1p1                     0        0B       0B         0
nvme1n1                       512      512B       2T         0
└─vg--data-disk                 0      512B       2T         0
nvme2n1                       512      512B       2T         0
└─vg--data-disk                 0      512B       2T         0

While the drives support up to 2TB as part of a TRIM operation, the way in which we created the multi-device raid0 had limited the maximum size of a TRIM operation to 32KB. This made our performance degradation make a lot of sense: we saw many thousands of IO requests per second when the fstrim command would run, and our database would experience degradation while performing writes. This didn’t happen elsewhere in our infrastructure.

We brought the LVM provisioning code into this database’s provisioning system, tested our changes, and finally applied in production. Finally, the issue was resolved! Indeed, LVM sits on top of the multi-device RAID driver in Linux, so it’s possible we might have been able to address this issue without switching to LVM. Fortunately, one other benefit of this work is that it introduced some consistency in how storage is provisioned on hosts with multiple directly-attached storage devices.

Wrapping Up

The cloud abstracts a lot away from us in terms of hardware – most of the issues we encounter these days don’t involve optimizing the interaction between the OS and hardware. After all, cloud service providers do a good job of optimizing their hypervisors to be “out of the way,” and things function well enough that further optimization can result in diminishing returns. Every so often, however, an interesting behavior comes up and we like to write about it. If you find yourself in the same situation, remember to poke around the physical systems you’re running on. Beyond that, books like Brendan Gregg’s Systems Performance: Enterprise and the Cloud can help provide valuable methodologies and tools to help debug these kinds of problems.

This was originally posted on the Threat Stack blog.

Over the years I have noticed something in friends¹ who perform similar work: the skills that make one particularly effective at this role often are picked up outside of tech. Things like navigating unstable infrastructure, unstable personal relationships, or other unstable systems and situations. Whether tech or personal, there is overlap in quickly understanding how systems come together, reverse engineering, distilling clarity from complex code (or humans), communicating calmly during incidents/emergency situations. But, the development of those skills come at a cost to the person who had to learn and deploy these tools to create their own stability.

The “good news” is that these skills are often seen as ones “senior” engineers have, and under the right conditions mixed with technical knowledge turns out to be in demand. Robustness and resilience are things that People Are Very Interested In™ and anxiety (or trauma, or both) can be synthesized into a career. It is good that these things contribute to keeping a roof over one’s head in the economic system we currently exist in today. The flip side of this is that there seems to be an inevitable march towards getting burnt out, especially for those who juggle all these demands on the regular.

I am left with a lot of questions: In what ways are the systems we maintain mirrors of ourselves? The system designs we push for (or don’t push for) or the debug mechanisms we try (or don’t try) are collected over years of technical, familial, and social encounters. Is there a safe way to perform this work for a lifetime and not burn out? Self-introspection is challenging for many reasons. I’ve tried to start reflecting on what forces are motivating me when I take on extra work, or jump into an incident, or engage in glue work. After that, though: what’s next? I don’t have any answers.

A number of people reviewed this post before I published it. I am thankful for their edits, thoughts, and insights.

To start: An EMR lifecycle manager is extremely helpful here, and we were fortunate to have one that our architecture team wrote. Our lifecycle manager takes a JSON-based configuration and handles spinning up a cluster with a configuration that can be controlled via a collection of HTTP endpoints. But that API only works for internal machines that want EMR clusters. If we’re adhering to the principle of least privilege (and we are), giving access to that API would allow analysts to affect other clusters that exist to process customer data.

So how can we solve this problem? Surprisingly: ChatOps and AWS Lambda. We wrote a function that consumed events from the Slack Events API, and depending on the use of a certain phrase, it would either start, get the status of, or destroy a cluster used for data exploration.

This method fits our security requirements in a few ways, specifically:

• The function can only affect one specific cluster, as opposed to all of them.
• The analyst doesn’t have direct access to the customer data.
• Accidental destruction of the cluster doesn’t affect the delivery of the Threat Stack service.

Access to the notebooks analysts use is authenticated using SSO, so we’re able to maintain positive control over that as well. Ultimately, it lets the SOC analyze the data they need when they want, without a platform engineer having to submit the request — which is how it is done today.

While many examples for Slack bots are in Node.js and Python, much of our security tooling at Threat Stack is written in Go, so we naturally used that. This presented a small challenge in that existing examples were built around a Go application running on a server full time (leveraging net/http’s http.ListenAndServe). After a bit of experimentation, we got a sample function working, which is shared below.

Setting up Slack Before you get started, you’ll need some authentication information available from Slack. To do this, you’ll visit the Slack API center and list your applications — then, click “Create App” and name your Slack bot. At this point you can modify how the application will show up in your Slack instance. You can also specify how you want Slack to interact with your application. We’ll do that a bit later, but for now, copy the “Signing Secret,” and then go to “OAuth & Permissions” to give your bot the following OAuth scopes:

• channels:history
• chat:write & groups:history

Feel free to modify these scopes. For example, if you expect your bot to work over direct message versus public or private chats, adjust the permissions accordingly. You’ll still need the chat:write permission since that’s how the bot can communicate what it’s doing. After you update the OAuth scopes, you can click the “Install App to Workspace” button, which will generate a bot user access OAuth token. Copy that as well. Both the signing secret and access token (which generally starts with xoxb) will be useful to you later.

Building the Go integration The whole integration is available on the Threat Stack GitHub account. The code has comments, but we’ll describe it here at a high level, too. Our sample application relies on the AWS Lambda library, which has you pass in a function you want to run. The receiveSlackEvent function performs the work, with a handful of functions to improve readability of that main function. One of those functions makes it easier to set the log level if you’re working in a development environment: setLogLevel. Another function validates the message we receive from Slack: verifyReq. Verifying that the request comes from Slack is very important to ensure that an adversary didn’t craft a malicious request. There are some risks here, but given our threat model, we’re OK with them.

The sample application code leverages the slack-go library, which has a built-in parser for the Slack Events API messages. A variety of messages can come in through the Events API, but the ones we care about are challenge events and callback events. Challenge events are how Slack verifies that the URL you tell Slack to send these events to is ready for Slack API events — so you don’t send a bunch of API events to someone who doesn’t expect them. Callback events contain the events you’re listening for. Slack has a variety of events you can subscribe to, but the ones you care about for this sample application are:

• message.channel
• message.groups (if your bot will be in private rooms)

Once you receive a callback event, you parse the event that’s sent to you. The InnerEvent data contains the actual event that you’ve subscribed to above. This application specifically listens for a MessageEvent with the text “It’s showtime!” and responds with “Yes, it is!”

Deploying your code Running GOOS=linux GOARCH=amd64 go build && zip slackevent.zip slackevent will get you a zip archive that you can upload to AWS Lambda. We leveraged the beta HTTP API Gateway — which worked well enough for what we wanted to accomplish. Once everything’s uploaded to AWS and you have an API Gateway URL, head on back to the Slack application listing and click on your application. Then go to “Event Notifications” and enable it. Paste in your API Gateway URL, and then under “Subscribe to Bot Events,” add the scopes we listed above.

At this point, you should invite your new Slack bot into a channel and write some messages into the channel. For each message, you should see a Lambda invocation — and if you write “It’s showtime!” you should get a response. If you did: Congratulations! If you didn’t, start retracing step by step. Did the API Gateway get your request? Did the Lambda function invoke? Are there places you could add in fmt.Printf statements to see what went wrong? This was a big part of how we debugged this in our development environment.

This was originally posted on the Threat Stack blog.

What is Client Authentication?

Likely, you are already aware of TLS (sometimes called SSL) — this is the technology your computer uses to securely communicate with websites. But TLS supports client authentication as well. Client authentication is a means to provide an identity for yourself — after you verify that the server you’re talking to has the right DNS and a certificate you trust. It’s a handy way to limit access to servers to individuals or devices without having to entirely rely on firewalls.

Getting Started

To begin, you’ll need an already existing public key infrastructure. We know this isn’t a small weekend project, so you may want to check out another article about building your own PKI. This article will still be here when you need it. Crucially, the security of your system will depend on you collecting your hardware certificates after employees leave, and maintaining certificate revocation lists. All of that is outside the scope of this guide, but the article above will help you get started.

You should have a client running macOS Mojave, and a recent version of HAProxy (we tested against 1.8) that is compiled with OpenSSL support.

Configuring the Server

You’ll want to make a few files in your HAProxy configuration directory (besides haproxy.cfg):

• /etc/haproxy/server.pem has your server TLS information. It should be in the order of public certificate, private key, and then any intermediate certificates.
• /etc/haproxy/verify.pem has your client CA TLS public certificate, along with the appropriate root certificate. So if you issue keys from “Contoso Users CA-1” then you’ll put that along with the “Contoso Root CA”.
• /etc/haproxy/crl.pem has your CRL — for all CAs. If “Contoso Users CA-1” is an intermediate CA, make sure you have the CRL for “Contoso Root CA” in this file

In your frontend balancers, add these parameters to your bind line:

bind 10.0.0.1:443 ssl crt /etc/haproxy/server.pem ca-file /etc/haproxy/verify.pem crl-file /etc/haproxy/crl.pem verify required
These options set up the mutual TLS authentication:

• ssl enables TLS for this port.
• crt [path] defines the server TLS private key/certificate/certificate chain.
• ca-file [path] defines the client CA verifier.
• crl-file [path] defines where HAProxy can find information on certificate revocations.
• verify required tells HAProxy that it must require a certificate from a client before the request moves on.

Other headers you can add to incoming requests include:

http-request set-header X-SSL                       %[ssl_fc]
http-request set-header X-SSL-Client-Verify         %[ssl_c_verify]
http-request set-header X-SSL-Client-SHA1           %{+Q}[ssl_c_sha1]
http-request set-header X-SSL-Client-DN             %{+Q}[ssl_c_s_dn]
http-request set-header X-SSL-Client-CN             %{+Q}[ssl_c_s_dn(cn)]
http-request set-header X-SSL-Issuer                %{+Q}[ssl_c_i_dn]
http-request set-header X-SSL-Client-Not-Before     %{+Q}[ssl_c_notbefore]
http-request set-header X-SSL-Client-Serial         %{+Q}[ssl_c_serial,hex]
http-request set-header X-SSL-Client-Version        %{+Q}[ssl_c_version]

These headers get passed to your application and can be used to infer certain information about the client’s certificate, like the CN, DN, the output from openssl verify — all helpful for debugging. We’ll talk more about that later.

Your backend configuration can remain the same — these changes only affect the frontend.

Configuring the Client

macOS Mojave ships with support to read the certificates off recent Yubikeys, making it easy to roll this change out to your users. To ensure that’s the case, check the Keychain Access application when you connect a Yubikey configured with certificates. The token should appear as a separate keychain.

Trying It Out

If your HAProxy is running, and you’ve got your Yubikey showing up in Keychain Access, try to go to your webserver! Your browser should prompt you for a certificate that matches the CA that signed it, and let you through. Here’s an example of that prompt:

If it’s your first time using the Yubikey during this session, Chrome will ask you to enter your PIN:

If all goes well, you should get to where you were going — in this case, a test page I set up to output X-SSL-Client-CN and X-SSL-Client-DN:

This is a simple Sinatra application that prints out some headers:

require 'bundler' ; Bundler.require
get '/' do
  client_cn = request.env['HTTP_X_SSL_CLIENT_CN']
  client_dn = request.env['HTTP_X_SSL_CLIENT_DN']
  "Hello #{client_cn} (#{client_dn})"
end
get '/ping' do
  "Hello"
end

Debugging Issues

Sometimes this doesn’t work the first time around — and it can be frustrating to debug something new. One way to begin debugging is to turn off client CA verification on the server side, and then look at the output of the X-SSL-Client-Verify header. Be warned: This will let anyone who presents a certificate through to your application. It’s important in this case to ensure that proper firewalls are in place to limit that access while you debug.

To begin, add ca-ignore-err all to your HAProxy frontend bind line, and make sure you log that header to the HAProxy log with:

http-request capture req.hdr(X-SSL-Client-Verify) len 2

This will capture the output of the OpenSSL verify functions that HAProxy is calling. You can use these to figure out what exactly OpenSSL is having trouble processing to guide your further debugging. Depending on your threat model, ca-ignore-err can allow you to ignore specific OpenSSL error codes. Alternatively, you could log out the headers on the application side as well; outputting request.env with the Ruby application above can help get that information out of headers as well.

Using OpenSSL’s s_client

Unsure if you’re running into a browser or other client issue? Want to get a ton of information about your TLS session? OpenSSL’s s_client is an extremely handy tool to debug TLS connections — and can also help you debug client authentication. One problem with OpenSSL is that it does not have native support for PKCS11 — and the OpenSC libraries are too low level for OpenSSL to just use them. Fortunately, the OpenSC folks made libp11 — a higher level library that can be used with OpenSSL to add PKCS11 support in.

To begin, you’ll need to install GnuTLS, libp11, a recent version of OpenSSL. The yubico-piv-tool is handy, as well. All of these are available in Homebrew. GnuTLS is important because it ships with a handy utility called p11tool — which is the first thing we’ll use to find our RFC7512-compliant PKCS11 URI — an identifier for the key we’ll want to use off the smart card.

Running p11tool --provider=/Library/OpenSC/lib/opensc-pkcs11.so --login --list-privkeys will output the URI of the object on the token you’ll need for future steps. The output will look something like:

Token 'cable' with URL 'pkcs11:model=PKCS%2315%20emulated;manufacturer=piv_II;serial=12341234a12341b;token=cable' requires user PIN
Enter PIN:
Object 0:
URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=piv_II;serial=12341234a12341b;token=cable;id=%01;object=PIV%20AUTH%20key;type=private
Type: Private key (RSA-2048)
Label: PIV AUTH key
Flags: CKA_WRAP/UNWRAP; CKA_PRIVATE; CKA_NEVER_EXTRACTABLE; CKA_SENSITIVE;
ID: 01

Next, export the certificate on your device. OpenSSL’s engine support works with private keys, but does not pull a certificate from a key as well. We’ll use yubico-piv-tool -a read-certificate -s 9a > mycert.pem for this purpose.

Finally, you can enter the OpenSSL console. Make sure to point to your newer version of OpenSSL that supports engines and you can run:

$ /usr/local/Cellar/openssl@1.1/1.1.1c/bin/openssl
OpenSSL> engine dynamic -pre SO_PATH:/usr/local/Cellar/libp11/0.4.10/lib/engines-1.1/pkcs11.dylib -pre ID:pkcs11 -pre LIST_ADD:1 -pre LOAD -pre MODULE_PATH:/Library/OpenSC/lib/opensc-pkcs11.so
(dynamic) Dynamic engine loading support
[Success]: SO_PATH:/usr/local/Cellar/libp11/0.4.10/lib/engines-1.1/pkcs11.dylib
[Success]: ID:pkcs11
[Success]: LIST_ADD:1
[Success]: LOAD
[Success]: MODULE_PATH:/Library/OpenSC/lib/opensc-pkcs11.so
Loaded: (pkcs11) pkcs11 engine
OpenSSL> s_client -engine pkcs11 -keyform engine -key "pkcs11:model=PKCS%2315%20emulated;manufacturer=piv_II;serial=12341234a12341b;token=cable;id=%01;object=PIV%20AUTH%20key;type=private" -cert mycert.pem -connect neato.tls.zone:443 -servername neato.tls.zone -state -debug
engine "pkcs11" set.
CONNECTED(00000005)
(Lots of output removed - all good for debug however)
---
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 4D87C5D67DBB35A72EC56A3963AB9E455865739BFA1B142643D9616E483C071E
Session-ID-ctx:
Master-Key: DD107AAEB642FDCEFA54A27E112AE6176D004AB4D78B81237863E2D3B08FE018CCD015D35942FF91DA21F3E8565000B6
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1568128169
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
GET / HTTP/1.1
Host: neato.tls.zone
Connection: close
HTTP/1.1 200 OK
Content-Type: text/html;charset=utf-8
Content-Length: 75
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Server: WEBrick/1.4.2 (Ruby/2.6.4/2019-08-28)
Date: Tue, 10 Sep 2019 15:31:50 GMT
Connection: close
Strict-Transport-Security: max-age=16000000; includeSubDomains; preload;
Hello cable (/C=US/ST=Massachusetts/L=Boston/O=Threat Stack, Inc./CN=cable)
SSL3 alert read:warning:close notify
closed
SSL3 alert write:warning:close notify
OpenSSL>

If there are issues negotiating the TLS connection, OpenSSL’s s_client will tell you what happened, and your HAProxy logs will be helpful here as well.

Putting It Together

Client authentication over TLS has a lot of prerequisites — and you may not have them today. If you do happen to be leveraging PKI and hardware keys for other purposes like VPN or SSH key management, this is a nice way to give access to web resources without forcing the use of a VPN.

This was a fairly technical deep dive, but if it’s interesting to you, we’re always happy to talk to customers who are doing interesting security things!

This was originally posted on the Threat Stack blog.

An edge case around all of this is the ability for our developers to run arbitrary privileged commands in Production. At face value, this is scary! What if an insider leaves something behind? What if they cause more damage to the infrastructure in the process of running arbitrary privileged commands? But the reality is: When your infrastructure is processing tens of billions of events per day, you are absolutely going to find debugging issues that become impossible to debug in a development or QA environment. How can we ensure that engineers can run commands in Production while maintaining the cleanliness of our infrastructure?

At Threat Stack, our Platform Security team’s job is to figure out solutions to problems like this one. Our solution to this was developing a tool called Trash Taxi. Users who would potentially run a command like sudo -i or sudo bash are no longer allowed to run that command; instead they run sudo nt. Running this command will register the server as having had the “seal broken,” and on the next Trash Taxi pickup, the machine will be terminated.

There are a few safety controls on this kind of collection — the end user can specify certain “trash holidays” either by an EC2 Role or Type tag that you may use in your infrastructure. This allows you to track when your developers may need to execute commands on a sensitive database host — so you can go and have a conversation about why that may have needed to happen (or not). Some infrastructures will have hosts that are challenging to terminate — that’s okay! You can use this data to justify spending engineering time in the right places to improve those processes if necessary.

Trash Taxi Resources Trash Taxi was released at Black Hat Arsenal this year. If you’re interested in hearing more about it, I did an interview with Dark Reading that you can listen to here. More information, including documentation and configuration information, is available at https://trash.taxi.

This was originally posted on the Threat Stack blog.

One thing we’ve wanted to do is create an alert when folks are using a VPN to connect to one of our environments. In the event of a stolen laptop and stolen credentials, a user could be alerted to someone logging in with their credentials. With OpenVPN, performing actions on a client connect is possible using a client-connect script, so in the tradition of writing small Go applications to improve visibility, we did just that.

For the last few months our Slack bot VPN Notifier has been letting our engineers know when they connect into a Threat Stack environment. We’ve now done the work to open source the tool so that others can use and improve on it. We specifically mention improve, because our tool has limitations: The current version does extremely basic environment checking, and extremely basic alert suppression. Our hope is that we can collaborate with others who want to take this tool the extra mile.

Using VPNNotify Our README.md has full instructions on using the tool, along with requirements. In summary, you will:

1. Use a configuration management tool to place a configuration file (/etc/vpnnotify.json is the default) on your OpenVPN host.
2. Use a configuration management tool to distribute a VPNnotify package. (We use FPM to make our own.)
3. Set the client-connect option in your OpenVPN Server configuration to where you installed VPNNotify — it will read in the VPN/user details over environment variables that OpenVPN provides.

This was originally posted on the Threat Stack blog.

One piece of metadata that we want to control is the engineering team member’s SSH key, which is tied to their YubiKey. This asymmetric key is what authenticates a specific user to a specific server (authorization to access that server is handled with special PAM configuration). We don’t use passwords to log into machines, except in emergencies which require multiple people to be present (and even then, an SSH key is involved).

When we were using Chef to set up all user accounts, Chef would set up their home directory and place an authorized_keys file which contained the public component of the user’s private key. Now that we’re using LDAP, a question remained: How do we distribute these keys to the machine?

Fortunately, OpenSSH has a configuration option that allows the use of an external application on login that returns the appropriate keys for a user. This configuration option is called AuthorizedKeysCommand. A few open-source options exist for performing an LDAP lookup of SSH keys, but they rely on interpreters like Lua and Python to run. Lua and Python are great languages, but we felt more confident in putting an application into the mix that didn’t rely on an interpreter. So we created Authkeys, a small application written in Go that will perform a lookup and return the appropriate keys so that SSH can make an authentication decision.

Using Authkeys Our README.md has full instructions on using the tool, along with requirements. In summary:

• You’ll use a configuration management tool to place a configuration file (/etc/authkeys.json is the default) on each host containing information about your LDAP setup.
• You’ll also use a configuration management tool to distribute an Authkeys package. (We use FPM to make our own.)
• Then, you’ll set the AuthorizedKeysCommand option in your sshd_config to wherever path you installed Authkeys to, and you’re off and authenticating in no time.

This was originally posted on the Threat Stack blog.

This presents a security issue with regard to access control: should all staff have access to all servers all the time? In early start-up life this is unavoidable. But as an organization matures and grows, it becomes a bigger risk. Administrator and similarly scoped credential theft is a goldmine for attackers, so we wanted to improve our story around internal access control.

Unwrapping who needs access to what is always an evolving task, but we put in the work to figure out who goes where and why, and then created groups to control that access. Since we already use groups as a way to control who can log into specific machines, and we use PagerDuty to assign on-call rotations, it seemed like we could create a tool that would query PagerDuty and update our on-call group. So we did! And as a gift to you, we’ve open sourced it.

Deputize is the tool that we created and use to manage our on-call rotation. Deputize uses PagerDuty’s API to get the email addresses of the engineers on call, then compares that to the named users in the on-call group, and updates accordingly. When changes occur, we’ve added in integration to notify Slack and log to a tool like Graylog as well. This allows us to ensure that everyone who needs to access all hosts can (in an automated fashion), while keeping that access limited to the time they’re on-call.

Using Deputize

Deputize has a few prerequisites:

1. Deputize builds on PagerDuty, so you’ll need to be using PagerDuty to manage your on-call rotation.
2. We use LDAP to manage authentication and group access.
3. Secrets such as the Slack API key, the PagerDuty API key, and the password of the LDAP user authorized to make changes to the on-call group are stored in Vault.

We provide information on how to configure Deputize in the README.md file located with the Deputize source, but the summary of it is that you’ll provide a path to a JSON config file via an environment variable. You can run Deputize on a schedule in a variety of ways — we have Sensu run Deputize like a “check” on an interval, but you could also run the tool via cron as well. Then, sit back and watch Slack for your on-call schedule to change automatically.

This was originally posted on the Threat Stack blog.

The YubiKey is a security device that originally outputted a 44-character “one time password” that could be decoded and mathematically verified and used as a second factor for authentication. Over the last few years, improvements to the devices mean that they can also perform other important functions, such as storing:

• Identity, Signature, and Encryption Certificates
• U2F data for websites (GitHub and GMail, among others, support this)
• GPG Keys

If you’re looking to set this up on your own, read on to learn how this extra functionality helps your security game, and how you can configure services to use it.

What’s In An Identity?

The YubiKey 4 can store four types of certificates — authentication (identity), digital signature, key management (encryption), and card authentication. Each of these has a different use as prescribed by the National Institute of Technology FIPS 200-2 standard. That’s because YubiKey’s certificate support is implemented in a way that’s similar to the way smart card certificate support works.

The identity certificate can be used for authenticating to any service that supports another standard: PKCS11. PKCS11 describes an interface to be used (regardless of platform) to talk to cryptographic tokens such as smart cards, YubiKeys, or Hardware Security Modules (HSMs). Fortunately, software such as OpenVPN, SSH, and MacOS Sierra have support for talking to such devices — which means you can move your SSH key off the laptop and onto a physical device.

These identity certificates don’t need to be kept in escrow (unlike an encryption certificate, for example) since they aren’t storing data to be retrieved and decrypted later — they’re just making a point-in-time assertion that the certificate is present on the hardware device. We’re not going to cover making encryption certificates, but the main point to remember is to have a safe place to store such keys in escrow.

Managing Your Keys

To begin, you’ll need to create a PKI. While we won’t get into too much detail here, there are tools that make this easier than it was once upon a time. CFSSL is a handy tool to help create a PKI. While making a PKI isn’t necessary for SSH or MacOS support, it is important if you want to use your certificates with tools like OpenVPN that rely on a Certificate Authority to authenticate clients.

It’s worth thinking about the physical security of your CA — any compromise to your CA can result in an adversary generating valid certificates to access your infrastructure. Using a machine that restores from an original state each time it boots is a good first step, with state stored on a secure external drive like an IronKey. A Live CD with the tooling you need on it would work; guides exist for customizing an Ubuntu LiveCD.

These instructions for provisioning the key work on a Linux machine with the most recent version of the yubico-piv-tool application (1.4 or greater). On a Mac you can install this tool with brew install yubico-piv-tool.

To begin, we’re going to initialize the YubiKey with a set of IDs for the card itself. We’ll also block resetting the PIN. It’s easy enough to provision keys and reset them so that it makes more sense to reprovision the key in the event someone enters the PIN incorrectly a number of times:

yubico-piv-tool -a set-chuid -a set-ccc
yubico-piv-tool -a unblock-pin -P 000000 -N 000000
yubico-piv-tool -a unblock-pin -P 000000 -N 000000
yubico-piv-tool -a unblock-pin -P 000000 -N 000000

Next, we’ll tell the key to generate a new private key. The public key will be pasted to standard out. Keep that key (for now).

Caution: Following this procedure may overwrite your existing setup.

yubico-piv-tool -a generate -s 9a -A RSA2048 -o pubkey.temp

You’ll then want to generate a certificate signing request:

yubico-piv-tool -s 9a -S "/C=US/ST=STATE_NAME/L=CITY_NAME/O=ORG_NAME/CN=USER_NAME" -a verify -a request -i pubkey.temp -o username.csr

Now, we mentioned having a CSR. We’re going to sign this with our Users CA. It is possible to create a self-signed certificate as well, using a different set of yubico-piv-tool options — some documentation is available as well. The CA configuration that CFSSL provides out of the box has a client option that should work for this purpose.

cfssl sign -ca ca.pem -ca-key ca-key.pem -config ca-config.json -profile client username.csr | cfssljson -bare username

Add the certificate to the card:

yubico-piv-tool -a import-certificate -s 9a -i username.pem

Finally, set a management key for the card and change the PIN. The management key will be required to make future device changes unless you reset the device:

yubico-piv-tool -a change-pin

(The default pin is: 123456)

yubico-piv-tool -a set-mgm-key -n (48 character 0-9a-f string)

At this point, you will have a key that can be used to provide identity for SSH and/or MacOS Sierra.

Configuring MacOS Sierra This part is easy — pull the YubiKey out and reinsert it into the computer. MacOS will ask if you want to pair the key with your local account, and you’ll be good to go. You’ll notice that authentication boxes ask for the PIN you set earlier, rather than an account password. If you run into issues, the sc_auth command can be helpful.

Configuring SSH

SSH requires a PKCS11 provider — fortunately, when we installed the yubico-piv-tool we installed a provider as well. The easiest way to use SSH with a PKCS11 provider is to use an ssh-agent.

To get your SSH key, run:

ssh-keygen -D /usr/local/opt/yubico-piv-tool/lib/libykcs11.dylib

To add the identity to your SSH agent, run:

ssh-add -e /usr/local/opt/yubico-piv-tool/lib/libykcs11.dylib

Once you’ve done that, try connecting to another machine. You should be authenticated and ready to go.

Configuring OpenVPN

On the server side, you’ll need to point the OpenVPN server towards your CA and any applicable CRL, as you would do if you weren’t using tokens. This configuration doesn’t change.

OpenVPN does support using PKCS11 identities. We found that it works better on Macintosh and Windows clients by using the 32-bit OpenSC PKCS11 provider and a GUI client like Viscosity. The OpenSC libraries are available via homebrew, or directly from them.

Wrapping Up

Moving user credentials off the laptop and onto dedicated hardware provides you and your users with peace of mind. Since the token generates the actual private key, you can be confident that the key is not floating around your client endpoint. There’s a little more user education involved in the proper use of the security token, but we’ve found this to be a small cost for a big security win.

This was originally posted on the Threat Stack blog.

However, once your organization grows to a certain size, managing users through CM can be a hassle. For one thing, not all systems are going to rely on UNIX authentication (such as Jenkins, Grafana, etc.), so you’ll need to start configuring those separately and possibly outside of your CM platform. As you add monitoring and oversight into your network, this can get confusing quickly - and keeping passwords in sync would be difficult at best. On the security side, you may forget to disable an account after an employee has left. Or, onboarding can take longer than it should as you add accounts to a variety of systems.

Read on to find out what we learned deploying LDAP at Threat Stack - complete with a few bits of open source code!

Lesson #1: Go with what you know (if it makes sense)

One of the de facto standards for storing and querying directory data is LDAP, the Lightweight Directory Access Protocol - and conveniently, it can store passwords and handle authentication as well. Other protocols, such as Kerberos, may be better for this purpose, but deploying Kerberos is a larger and more complicated project. We felt that LDAP would allow us to achieve some quicker wins as long as we ensured a secure deployment with enforced TLS. We also have pretty simple requirements for identity management, and don’t need to integrate with anything that uses a proprietary authentication system, which simplifies our needs.

Having worked with 389DS and OpenLDAP in the past, we ended up selecting OpenLDAP. OpenLDAP seemed like it would be easier to automate deployment and maintenance tasks. We prefer to rely on open source tooling because, generally speaking, that means we can support, debug, and most importantly, automate it easier than we would a proprietary solution. The #openldap channel on the Freenode IRC network also has people who are knowledgeable and respond to questions generally - as long as you have demonstrated that you’ve gone through the documentation. This can be frustrating at times, since the OpenLDAP documentation can be lacking and inconsistent. For those looking to begin with a test deployment, the Ubuntu OpenLDAP Server Guide is a helpful place to begin your journey.

Lesson #2: Build your own OpenLDAP package

We initially started using the version of OpenLDAP that ships with Ubuntu. This allowed us to get going quickly, but presented issues. First, Ubuntu and Debian packages automatically configure the server and start it after installation. This makes managing its installation with Chef a little more difficult than it needs to be. More importantly though: the distros can lag behind in updates. Trying to get help from the OpenLDAP community while running a distribution-provided version of OpenLDAP is difficult at best.

This became a factor for us when we were trying to debug an interaction between our Chef Custom Resource for managing replication and OpenLDAP. Creating our own version of the OpenLDAP server that installed into /opt allows us to run the latest version of the server, while leaving the Ubuntu userspace alone - and capture useful debugging information to boot.

We created our own package by compiling OpenLDAP and using FPM, then handled post-install configuration with Chef. It’s helpful to look at how Ubuntu handles its post-install configuration by downloading the package source by running apt-get source slapd. Our Chef cookbook writes out the initial configuration to disk, then runs the appropriate slapadd command, then sets a node attribute saying the configuration has been done. This ensures the action is idempotent.

Lesson #3: Replication is a hard thing to get right the first time

The Ubuntu guide we referenced above is decent for getting your first deployment up and running. It has a few downsides, however, one of which includes the fact that writes to the directory must go to the producer server. This presents a problem: do you have a way of tagging which server in your infrastructure does writes and reads? Clever application of Chef roles could accomplish this, but this is not the future we were promised in 2016. Instead, we set up Multi-Master Replication (MMR) so we can send writes to either server and not have to add complexity to our setup.

Setting up MMR can be complicated, and the documentation and guides available online vary in quality. At a high level, you’ll need to enable the accesslog and syncprov overlay. The syncprov overlay is what actually performs replication - and it uses the accesslog as a signal to know when to perform replication and what to replicate. Then you’ll need to configure your olcSyncrepl entries on the LDAP database you want to replicate so that OpenLDAP knows where to send changes.

It’s best to look towards examples that work in production, and we found one in the Zimbra project’s zmldapenable-mmr script, specifically lines 145–191. This gives an example of attributes required to set up a new database, configure the new database to store accesslog information, and enable the target database (the LDAP root that your records are stored in) for replication. A key part of this is the setup of olcSyncrepl attributes on line 205. This is how your server knows what peers to replicate with.

One place we chose to deviate from Zimbra’s configuration was using certificates rather than passwords for replication. We wanted to use certificates to avoid having a password with replication permissions in plaintext on our host. To do this, you’ll need to generate a client certificate signed by a CA you trust - a good use for Vault or CFSSL.

Lesson #4: Managing LDAP with CM tooling is complicated

One thing that makes handling the configuration of OpenLDAP complicated is the fact that it stores its configuration in its own database (cue the “inception” noise here). This presents a problem: Modern CM tooling has lots of tooling around for writing and reading files and restarting or signaling services as necessary. This is a bit harder with OpenLDAP and Chef.

To make this easier for you, we’re open sourcing the providers (and helpers) we wrote to interface with LDAP. We use the Syncrepl resource we developed and Chef’s search functionality to figure out what olcSyncrepl entries each host needs to have. For every Chef run, we also get the list of existing ones and compare it to our Chef search results so we can remove ones that no longer need to be there. We also have a basic LDAPModify resource that functions like the execute provider to set attributes required after olcSyncrepl entries exist (i.e., olcMirrorMode). You’ll find some of this logic in our helper library.

When we originally deployed OpenLDAP with Chef, we just cleared all the olcSyncrepl entries and added new ones every Chef run. What we found was an interesting condition: sometimes the Chef run would hang when adding the olcSyncrepl entry. This only happened when Chef would add an entry for replicating with the local host - not what we needed. When we cleared that up, another bug appeared: clearing and readding syncrepl entries for hosts that OpenLDAP was still replicating with would result in having too many open file handles. The Syncrepl resource we open-sourced above handles everything nicely so you avoid these issues.

Lesson #5: Deploy carefully

Many of our internal services are running in AWS and take advantage of auto-scaling groups. This means we can test a deployment, and if it doesn’t work out, we can quickly replace the host into a known working state. That said, we still ran into issues along the way, and not all of our services are stateless. Screwing up authentication on a Cassandra host could mean many GB of data to resync, so it’s important to try deploying to a few hosts and slowly grow from there. Most of our issues revolved around allowing all of our hosts access to the LDAP security group. We also needed to make sure we configured our pam_access setup to allow for local service users (i.e., cassandra, elasticsearch, etc.). The existence of our (secure) emergency account proved to be useful for us as well.

Is centralized authentication for you?

Well, we can’t answer that specific question - though from a cloud infrastructure security standpoint, we do see a lot of benefit. More information on the code we released (including how to use it) can be found on GitHub. If you decide to deploy OpenLDAP, hopefully we’ll have saved you a few hours of frustration!

This was originally posted on the Threat Stack blog.

With teams becoming larger and more distributed, keeping the password in an envelope for one person to use probably won’t work in an emergency. And making the password easy enough to remember is definitely not secure in an age when your adversary can rent a password-cracking cluster for dollars per hour. We also want to trust our internal admins to do the right thing, but we should ensure that adequate technical controls are in place to enforce that. So what can we do?

How to Share a Secret

Back in 1979, Adi Shamir (the “S” in “RSA” for folks familiar with cryptography) published a paper titled “How to Share a Secret.” The problem Shamir was considering was presented in a combinatorial mathematics text:

Eleven scientists are working on a secret project. They wish to lock up the documents in a cabinet so that the cabinet can be opened if and only if six or more of the scientists are present. What is the smallest number of locks needed? What is the smallest number of keys to the locks each scientist must carry?

“A minimal solution uses 462 locks and 252 keys per scientist. These numbers are clearly impractical, and they become exponentially worse when the number of scientists increases,” Shamir wrote. Indeed, that is a large keyring. He would later go on and describe an algorithm called Secret Sharing that allows someone to break a secret into several “shares” that can be distributed amongst many people, such that when a certain number of shares (defined by the organization) are combined, they reveal a secret value and that if you don’t have enough of the shares, you won’t get any information.

We’ll save you the deep dive on the details - if you have an interest in polynomial mathematics, Shamir’s paper is short, and worth reading. What is exciting, though, is that we can use the solution to the “secret project” example to protect our Administrator Account. People have turned Shamir’s secret sharing algorithm from 1979 into libraries that you can use today to enforce that your developers or operations staff can only get your emergency account’s password if a quorum of them get together and agree to do so.

Sharing a Secret

Here’s a sample program written in Ruby that uses the sssa-ruby library we found on GitHub. In this example, we’re imagining that we have four staff members, and any two of them can get together and recreate the password.

#!/usr/bin/env ruby
require 'securerandom'
require 'sssa'
random_string = SecureRandom.hex
puts "* Random string is #{random_string}"
shares = SSSA::create(2, 4, random_string)
shares.each do |e| puts "* Share: #{e}" end
combine_shares = [shares[rand(0..1)], shares[rand(2..3)]]
puts "* Reconstructed string: #{SSSA::combine(combine_shares)}"

Our program generates a random string, separates it into four parts (these are the individual parts you’d distribute to your staff), and picks a random entry from the first two and the second two to combine them.

When we run this program, our random string is separated into four shares, and then reconstructed from a semi-random selection:

* Random string is 92b7d4bf87fd82916c2d234a3c1769ca
* Share: ZqjMrDRCN_hwgPrxe_zVLHkQ0QtZLKIwst6HlFu5HZ8=OEaUQ8CdEtFptF0Gm8_GzY2b5O1x6PV8DfWDZVyC-p4=
* Share: JcscmxQaO-wi7_5HvxoWIynYf9jik5jrQkGxC4jkAIg=u9Yy5iZe5NKBbxkQtTPTlvK0MZqQErX7Vugc8ghv6_4=
* Share: QDCcIDJu9T7wwFGtihj9INzmCZnFrMBGXwHvErjny04=RL3YiggPfDeDsobRG_d7DfGind3g1a7x1GMx00H_cks=
* Share: djKNQW9esYZyH2gIjWulP_muIJzbh_EL0UU9Dor4YcU=zvdasiwQxldYSawPc9FkXqJiLpV0VBWz2_Kl8v8Kjb4=
* Reconstructed string: 92b7d4bf87fd82916c2d234a3c1769ca

Now, before you go off and use this technique, you may want to consider a few things. You may want to explore ways to encrypt this secret so that only a named recipient can open it, or not even display the password to begin with and update the target system directly with your application. Because this involves a security-sensitive library, it’s definitely a good idea to have one of your security folks review the library and how it works. These exercises are left up to you. But even starting with our sample, you’ve gained the ability to enforce that using a certain password requires multiple people to begin with, which is a good thing.

Tying it all Together With Threat Stack

But now there’s another issue: You really only want the “emergency account” to be used in emergencies. How can you discourage the use of this account for non-emergency use? That’s where Threat Stack comes in. With Threat Stack, you can create an alert that is triggered when someone attempts to use this account — or when any action is triggered on its behalf. Simply make a rule of user = "emerg". Now when any event is triggered involving your backdoor account (named “emerg” in this case), you will see it in any of the services you’ve configured for alerting.

This was originally posted on the Threat Stack blog.

The idea that administrators don’t program reminds me of my first interview for a system administration internship back in May 2007. This is funny to me because one thing I remember clear as day was stating that I was not a programmer. Code was not something that I did. I just made computers work well and do useful things for people. Other people wrote code, I just lived in their world.

Yet, in the last week I found and patched a bug¹ in Debian apt (a minor one at that, but nonetheless something installed on many Linux machines across the world) and added functionality to the software we’re using to review submissions for LISA ‘16². This does not correlate with the 2007 me, who decided to be so resolute in what he could do and could (would?) not do. What gives?

I was one of those stereotypical kids who was good at computers. In short, I ended up making my town’s website at a young age, could fix computers and their software issues, played with Linux at a young age, life was good. What was more difficult to wrap my head around was learning how to actually program. I was given a copy of Visual Basic growing up; I wasn’t quite sure what to do with it. Our basic CS classes in high school had us reading in values and printing out algorithmically generated Christmas trees. I was not exceptional at math, a skill that people often referred to as extremely important to being able to write programs.

Here’s the thing, though: I was able to run my high school’s email server, and make sure backups worked for a small elementary school in the school district, and I was later asked to help my town buy and maintain computers at town hall and basically be the IT manager there. In considering my future, system administration didn’t seem like it got the same respect that developers did, but I felt decent at it. I felt decent at making systems reliable and avoiding data loss and working around the issues that developers seemed to throw at their users (both end users and administrators). Additionally, people seemed to find value in these skills, and even paid for this. I was a lucky high schooler, and undergrad student, no doubt. Speaking of undergrad, I sucked it up and did what I could to pass C++ and Java (the only coding requirements for my major) but wasn’t particularly stellar at either.

I remember being interested in automation – it was important to figure out out Windows Deployment Services because there was no way I was going to install Windows by hand on 20+ machines. Reducing patterns into shell scripts was a lot more palatable than going around doing things by hand. During my internship, I wrote a provisioning system in Perl because one of my first assignments was to get Red Hat Enterprise Linux 4 onto 14 machines. There was parsing to be done to move the infrastructure from NIS+ to LDAP. User accounts were previously done by email - we moved to a web-based system after I made one.

The entire time I was working on these things it never felt like programming-with-a-capital-P. I was writing in Perl, PHP, BASH and TCSH – not the compiled languages I had struggled with earlier. But I wasn’t a developer, I wasn’t mathing (is that a word?) and so it never felt real. No complicated algorithmic sorts here - just some SQL and outputting HTML and moving text around when I had to.

I think that I would have been a lot better off if I started with a different mindset: if you made a computer do something, congrats! you did a code. I’m not going to say this is the same as building a complex distributed system for production use, for sure (maybe someday!). I do think I would have been more comfortable with such a thing earlier (rather than now) if I had a few wins early on. Alas.

Today I mostly write in Ruby, and have reached the point where I can jump to another language and flail around enough to get something done. I “get” types now, thanks to toying around with writing something in Go. A few weeks ago I wrote an internal signup form in node.js. I still experience a certain amount of angst when I start writing something, but once I get going it’s okay.

I wrote this because I don’t think I’m the only person in the world who has learned how to code as a system administrator or operations engineer or whatever it’s called today. As software continues to eat the world, I don’t think the operations side of the house is going to get a free pass on continuing to maintain the status quo just because these issues are hard to write logic around. Indeed, some places are figuring out how to remove the human out of the loop as we speak. I see this change in the interview stories that people share in various ops-heavy communities (hangops, etc.), and in the kind of work that people at the middle or top of the field (that I look up to) are doing.

Which brings me back to the SRE book. I can understand the frustration with the way in which they write of (write off?) system administration, especially for those who have cut their teeth and evolved over time. But even 10 years ago, I didn’t have the impression that system administration involved actual engineering so much as just MacGuyver-ing around the decisions of folks who may or may not have run a system in production. Perhaps I was too naive. As the pendulum swings towards larger organizations being more comfortable running their own systems, more organizations are going to look at processes and techniques that have worked for others and try to apply the ones that make sense to their organization. Obviously, not everyone is a large organization that makes everything from scratch, but there are broader lessons to be learned here.

Administrators (myself included!) have complained about the disconnect between people who write software and us for decades. Now we finally have a chance to sit at the table in some organizations. My own hangups about code ability aside: this is actually exciting! We get to care about the things that are important to us! This shift carries with it some baggage³, but I suppose sitting at the table ain’t free, either.

I’m looking forward to a future of working with (and contributing to) systems that care about ops the way I always have, regardless of what it’s called. That’s pretty cool in my book.

Thanks to Julie Hansbrough, Jarrod Pooler, Karen Rivera, and Carolyn Rowland for their thoughts and corrections.

One solution to let these applications run and not have to keep older OS machines on your network is Docker. Docker does have a variety of base images available through their site, however typically they are newer OSes.

If you find yourself in the sad position of making a CentOS4 base image, here are some helpful pointers:

1. CentOS4 used up2date to pull down system updates and new RPMS. However, it also supports Yum, and the CentOS4 repos do have a repodata folder for yum use. Grab the appropriate repodata folder, and put it in your /etc/yum.repos.d folder. Make sure to add a line for enabled=0.
2. You will find the script here to be of particular value. I cut out the usage(), yum_config, and getops stuff.
3. Add in a test to make sure $target exists. What if you accidentally kill off /tmp and then run the script and you forget to test for if $target exists? This exercise is left to the reader (but see #7)
4. You will need to modify the first yum (...) install line to include --enablerepo=centos4 --disablerepo=*.
5. Need extra packages? Make sure to copy in your repo definition from /etc/yum.repos.d to $target/etc/yum.repos.d as the the groupinstall will provide a repo that may not be what you want it to be.
6. You will likely need to add in some important .i386/.i686 RPMs, especially if you are running the build script on a RHEL6 machine. Some good ones to include include libgcc and glibc.
7. Kill off the RPM repository.¹ rm -Rf $target/var/lib/rpm – the reason for this is that you are likely running this on a newer RHEL machine. Older versions of RPM will not be able to read the RPM database that you have created. If you really need the ability to do development and install RPMs in a docker repo to figure out what you need, after you have built a base image, run rpm --initdb and re-run your groupinstall command. Then, tar the RPM library and put it into the script so that you may experiment without having to perform this step manually. I do suggest that once you are done with dev, you just kill off the RPM library to begin with – no one should be installing RPMs inside your artisanal container, right?
8. Since you know your OS name, it may be beneficial to just set name=centos up front.
9. Bam. You have a CentOS4 image now!

There are arguments to be made for being able to use yum in a Dockerfile however, given that only one application i know of needs to be run like this, I find that building this once and keeping the scripts in revision control work well enough for me.

My Dockerfile is fairly simple - I have a FROM and MAINTAINER line, with an ADD and an ENTRYPOINT - the ADD simply drops in an init script that adds a local-to-container user with the appropriate uid/gid, then switches to that user and runs the application. X11 use is handled by proper management of the DISPLAY variable and/or by exposing /tmp/X11-Unix to the container.

Hopefully you find yourself not needing to support older apps. But, in the event you do, maybe this will help.

Realistically, this is not really that big of a deal, except that finding the appropriate instructions to do so is near impossible. People say “use certutil!” or “load up firefox go to https://ldapserver:636 and save the cert then copy the files” and this and that and ugh.

So here are some notes. I hope they help you.

• The NSS certificate DB for LDAP on Solaris lives in /var/ldap — with the rest of the ldap settings.
• The easiest way to “just make it work” is to load up firefox, import your CAs, go to https://yourldapserver:636 and permanently accept the certificate.
• Firefox will complain that this is not a port you rock HTTP on. So go to about:config, right click, make a new string value called network.security.ports.banned.override and add 636 to it.
• Go back to that site again.
• Copy ~/.mozilla/firefox/profilename/*db to /var/ldap

You’re done. Run ldapclient to set up LDAP and it should work fine. Do this, then use certutil -d /var/ldap -L and figure out how you may be able to script it. Or just run with it. Your call.

Note: This was posted a year ago. I should mention, you should use a fresh Firefox profile for this – no use in accidentially carrying over unnecessary secrets to a config that may be distributed out to many systems.