For a few years now we have been mandating a fairly strict authentication policy for those developers who commit directly to the git repositories housing the Linux kernel. Each is issued their own ssh private key, which then becomes the sole way for them to push code changes to the git repositories hosted at kernel.org. While using ssh keys is much more secure than just passwords, there are still a number of ways for ssh private keys to fall into malicious hands — for example if the developer’s workstation is compromised or if someone manages to access some poorly secured backups (I’m willing to bet that if you check various removable media lying around your work desk, you will discover that one of them contains a full copy of your home directory from that last time you needed to reinstall your workstation and needed to make a quick copy of your files).
Keeping that in mind, we wanted to further tighten our access requirements, but without causing undue difficulties for the kernel developers.
What is 2-factor authentication
Two-factor authentication is a pretty old concept, and I have previously written on this subject when discussing the steps we have taken to improve workstation security for all Linux Foundation systems administrators. In very basic terms, there are three main categories, or “factors” when it comes to authenticating you to a computer:
1. something you know: a password or a numeric PIN
2. something you have: a device or a card in your possession
3. something you are: your unique biometric signature, such as your fingerprint or your retinal scan.
We’ll leave “something you are” out of scope for this article and concentrate on the “something you know” and “something you have” factors. Unfortunately, even though ssh keys are very long and are stored on the hard drive of your workstation instead of kept in your memory the way a password is, they can’t be considered true “2-factor authentication,” even when the ssh key is protected by a passphrase — as I mentioned above, the ssh private key can be stolen or leaked. If we wanted to improve access security for kernel developers, we needed to add a true 2-factor authentication mechanism.
There is an extra twist to the problem, though — kernel developers work from anywhere in the world, which makes device provisioning extra difficult. We needed a solution that would allow people to enroll their own devices remotely and do most token management on their own.
Soft tokens vs. hard tokens
In 2-factor authentication parlance, a “hard token” is a dedicated physical device that is purpose-built to do nothing else but authentication. A “soft token,” on the other hand, designates a pure-software implementation that is running on a multi-purpose portable computing device (such as a smartphone). If you’ve ever set up “two-step verification” with your Google account or turned on the “code generator” for Facebook, you’ve used a 2-factor authentication soft token. If you’ve ever used an RSA SecurID “key fob” or a Yubikey, you’ve had firsthand experience with “hard tokens.”
Both hard and soft tokens have their advantages and disadvantages. The upside of soft tokens is their convenience. After all, most of us already carry in our pockets a powerful computing device that is more than capable of calculating and displaying 6-digit codes. The downside, though, is that smartphones are consumer-grade, globally networked devices that may or may not be receiving timely security patches. Should we really make them an integral part of our authentication scheme?
Hard tokens don’t have this problem, as they are either entirely self-contained or have a very limited way of interacting with other computers. On the other hand, unless all your users are within the same physical location, provisioning hard tokens — and especially replacing them when they get lost — becomes a major problem. With hard tokens, shipping costs and productivity lost due to wait times become major obstacles.
At the Linux Foundation, we wanted to make both options available and leave the decision of whether to use a soft token or a hard token in the hands of kernel developers themselves. After all, 2-factor authentication with a soft token is still dramatically more secure than no 2-factor authentication at all. That being said, we wanted to encourage the use of more secure hardware tokens, which is why we contacted Yubico, the makers of Yubikeys, in the hopes that they would be willing to offer a discount to Linux kernel developers. To our amazement and gratitude, Yubico went well above and beyond a simple discount and offered to donate a hundred yubikeys to all Linux kernel developers who currently hold accounts at kernel.org.
How do yubikeys work?
A yubikey is a cute little device that incorporates a chip capable of storing the pre-shared secret and calculating a one-time token. When you plug it into your computer’s USB port, the OS recognizes it as a keyboard, which means that this device works with pretty much all operating systems without a need for drivers or configuration. There is only one button on the device, which, when pressed, generates the one-time token and sends it to the computer as a sequence of keystrokes, as if the user had typed them on the keyboard. The user then submits the authentication form and sends both their password and their one-time token to the validation server.
When the validation server receives the one-time token, it performs the same computation with its own copy of the user’s pre-shared secret. If the values match, it is a good indication that the user has the device in their possession and should be allowed to log in.
In addition to Yubico’s own 2-factor implementation, yubikeys also support OATH’s HOTP standard, which is what we opted to use for our kernel.org needs. Doing so allows us to use both soft-tokens and hard tokens interchangeably (TOTP standard is an extension of the HOTP standard).
How does 2-factor auth work with git repositories?
Having the technology available is one thing, but how to incorporate it into the kernel development process — in a way that doesn’t make developers’ lives painful and unbearable? When we asked them, it became abundantly clear that nobody wanted to type in 6-digit codes every time they needed to do a git remote operation. Where do you draw the line between security and usability in this case?
We looked at the options available in gitolite, the git repository management solution used at kernel.org, and found a way that allowed us to trigger additional checks only when someone performed a write operation, such as “git push.” Since we already knew the username and the remote IP address of the developer attempting to perform a write operation, we put together a verification tool that allowed developers to temporarily whitelist their IP addresses using their 2-factor authentication token.
Here is what a developer would get if they attempted to git push from an IP address that hadn’t been 2-factor validated:
remote: IP address “x.x.x.x” has not been validated.
remote: FATAL: W VREF/2fa: testing mricon DENIED by VREF/2fa
remote: 2-factor verification failed
remote:
remote: Please get your 2-factor authentication token and run:
remote: ssh
This e-mail address is being protected from spambots. You need JavaScript enabled to view it
2fa val [token]
remote:
remote: If you need more help, please see the following link:
remote: https://korg.wiki.kernel.org/userdoc:gitolite_2fa
remote:
remote: error: hook declined to update refs/heads/master
The developer would then run the following command to validate their current IP address for the next 24 hours (putting their 6-digit one-time token where it says “[token]”):
$ ssh
This e-mail address is being protected from spambots. You need JavaScript enabled to view it
2fa val [token]
Valid TOTP token used
Adding IP address x.x.x.x until Sat Aug 9 28 20:29:31 2014 UTC
Once this is done, all future git operations from that IP address for that particular user would succeed until the validation expires after 24 hours. Developers may optionally validate their IP address for an extended period of time — up to a max of 30 days — which is handy if someone mostly works from the same location.
Protection of Mainline and Stable repositories
At this time, both the mainline and stable Linux kernel repositories are already protected requiring 2-factor authentication before a git push is accepted. As kernel developers congregate in Chicago to attend the annual Linux Kernel Summit, we are hoping that many more of them will choose to turn on 2-factor authentication on their own repositories hosted at kernel.org, to make it that much more difficult for an attacker to sneak in a malicious commit.
One thing for sure — giving out 100 yubikeys will definitely help encourage them. Thanks again, Yubico!