Why passwords are bad
Are you a systems administrator? Quick, which system in your infrastructure is most vulnerable to hacker attacks? No, it’s not the web server — though it’s a good guess. No, it’s not the firewall. The answer may surprise you — it’s your workstation.
Think about it — unless you’re working for an agency with extremely rigid security policies, you are probably able to connect to servers you administer right from your workstation. Perhaps not all the time — perhaps you have to establish a VPN connection first in order to be on the “inside.” Once that is done, however, your workstation becomes an extremely interesting target for malicious hackers, since at that time your workstation happens to be the least protected system that sits both on the outside and on the inside of your trusted network.
Did you know that your workstation is currently running software that was written for the sole purpose of letting others execute arbitrary code? You are looking at it right now. Yes, I do mean the browser. Did you ever think of browsers in such terms? That’s all they do, they download someone else’s code from the Internet and then execute it. Yes, all browsers implement a “sandbox” — a sanitized environment that is supposed to prevent downloaded code from doing any harm. The key phrase is “supposed to,” because quite often the browsers don’t get it right. Especially plugins, such as flash. Don’t get me started on flash.
And don’t get me started on humans. Did you know that we’re made out of meat? “Evil hackers” are well aware of that fact. They know that the easiest (and the stealthiest) way to hack into a system is not to relentlessly attack it looking for exploits, but to wait until the admins do something meat-headed. Like, you know that time when your mom posted that link with a funny cat, and the page tried to load a movie but ended up crashing your browser? And then you tried it again in another browser, just to make sure? See? Made out of meat.
So, let’s say the sandbox around your browser fails and attackers get a foothold on your system. What’s their next move? The malicious script will probably search your browser’s history and try to judge whether you’re an interesting target or not. Then the script will most likely install a keylogger and a backdoor:
• A keylogger is a small application that will record everything you type on your keyboard — it is particularly interested in anything that looks like a credit card number or an account password.
• A backdoor is a way for attackers to execute arbitrary commands on your workstation, and it doesn’t matter if your workstation is on a private network or not. Most backdoors just connect to some “command and control” server over an encrypted http connection and either download a prepared “payload” to execute, or hold the connection open for attackers to have full real-time shell access to your workstation from anywhere in the world.
With the keylogger and the backdoor running on your workstation, the next time you VPN in to your work and type in your root or sudo password, you’ll be handing the attackers the keys to your kingdom. Unless, that is, your servers require more than just a password in order to authenticate users — which is really what I’m here to talk about.
What is multi-factor authentication
The problem I described above is well-understood among security professionals, and one of the ways to prevent the workstation compromise from leading to server compromise is to require the use of more than just a regular password in order to get access to the servers — especially when it comes to obtaining elevated privileges (i.e. “su” or “sudo”). The approach is called “multi-factor-authentication” — in addition to using a password (“something you know”), you also have to prove that you have in your possession some kind of a physical token that was given to you by the authority that issued your credentials (“something you have”). You may additionally have to prove that you are the same biological entity that received those credentials (“something you are”). Remember the movie “Sneakers?” In order to get access to the secret facility, the hacker crew had to defeat all these three factors — get the card (“something you have”), the PIN (“something you know”), and record the mark saying “My voice is my passport. Verify me (“something you are”).
Implementing all three factors is extremely costly and inconvenient, though — and the “something you are” bit can’t really be used to authenticate unsupervised access (as demonstrated in the “Sneakers” — since “something you are” never changes, the attackers only have to record and replay one successful transmission in order to fool the system). Therefore, the only additional factor worth implementing to authenticate unsupervised remote access is “something you have.”
The most widely known “something you have” devices are smartcards and the ubiquitous RSA Security “fobs” — the former store an encryption key directly on the card that must be unlocked with a PIN in order to authenticate the user, and the latter convert timestamps into a set of digits that change every 30-60 seconds. We only mention smartcards in passing, as they won’t actually help in our scenario, since they will be helpfully plugged into the workstation and the attackers will know the PIN from the keylogger. Time-based tokens, on the other hand, are an effective measure to prevent a workstation compromise from leading to a server compromise (with a few caveats, which we’ll touch on later).
Time-based hardware tokens work by converting current time into a string of characters that can’t be predicted without knowing the “secret hash.” That secret hash is only installed on the hardware token itself, plus on the authentication server, which is how we make sure that the user actually has the hardware device in their possession. When someone tries to authenticate, they submit the value shown on their hardware token as part of their credentials. The verification server then performs the same math as performed on the token, and if the results match, then the user is allowed to log in.
What happens next is extremely important — the server records that the token has been used once and therefore cannot be used again. So even if the keylogger has a record of all the tokens submitted by the user, none of them can be reused again, even if attackers do it right away. That is why it is called “One-Time Password” and there is even an open standard behind it is called Time-based One-Time Password Algorithm, or “TOTP.”
Challenges of using hardware tokens
There are several companies that provide both open-spec and proprietary hardware tokens. There is RSA Security, which has been offering two-factor authentication services and proprietary hardware tokens since the dawn of IT time. There is also Yubico, which is a more recent arrival to the scene and uses an open OATH standard. Their product, yubikey, plugs into a USB port and the operating system recognizes it as a USB keyboard. Once a button is pressed on the yubikey, it sends a string of characters containing the one-time password.
However, implementing two-factor authentication using true hardware tokens can pose its own set of challenges — particularly if your administration team is not located in one physical office. Take us at The Linux Foundation — we have admins who are located on both coasts of the US and Canada, one in Japan, and now one in Australia. The task of provisioning physical tokens suddenly becomes very complicated and involves international shipping, which is either slow or extremely expensive. And what happens if the admin leaves their token in the pocket of their jeans and it goes through the wash? Are they out of commission for a few days until the new token is provisioned and shipped?
This is exactly the problem that caused grief for the Fedora Project. Despite being officially hosted and sponsored by Red Hat, Fedora Infrastructure is entirely separate from the rest of the Red Hat network and is administered both by Red Hat employees and by volunteers who are physically located anywhere in the world. A few years back, Fedora Infrastructure started requiring the use of yubikeys for two-factor authentication, but ran into provisioning problems mentioned above. Fedora either had to dedicate a budget for purchasing and shipping the yubikeys, or require that volunteers spend their own money — neither of which was quite optimal.
Enter smartphones
Thankfully, this is the age of mobile computing. Nearly all of us carry a powerful computer in our pocket that is more than capable of calculating and displaying TOTP tokens. Google recognized this a while back and released a free mobile app called “Google Authenticator,” available on most mobile platforms. Anyone can set up two-factor authentication for their Google Account using the Authenticator, but the best part is that it’s not just limited to Google’s services. Since TOTP is an open standard, any infrastructure can use Google Authenticator to provision their own software tokens and implement TOTP-based two-factor authentication for their services.
It is generally recognized that tokens on the smartphones aren’t as secure as true hardware tokens. After all, smartphones are part of a worldwide network and can be hacked just like any other computing device. However, since your mobile phone is not physically connected to your workstation, isn’t running the same operating system, and isn’t even using the same processor architecture, that helps make it highly improbable that a casual attacker will be able to compromise both your workstation and your smartphone in order to defeat two-factor authentication.
Fedora Project and Linux Foundation join efforts
Recognizing that we have similar problems — namely, the need for two-factor authentication and a very distributed IT team, the Linux Foundation and Fedora Project Infrastructure joined forces to come up with a two-factor authentication solution that 
would use the TOTP standard and utilize software tokens. We first reviewed LinOTP, which implements a number of two-factor authentication standards including TOTP, but unfortunately their “community” implementation artificially removes crucial features, making them only available as part of their commercial “enterprise” product. Namely, the features we required was support for LDAP-based user store and support for yubikeys, as Fedora wanted to continue providing yubikey support for those members of the team who had already obtained one. Neither of these features is available in the LinOTP community branch.
Then we looked at the Google Authenticator open-source PAM module, but it is not suitable for centralized implementation (it can only be securely used on one server at a time) and has a number of other shortcomings, such as an odd architectural decision to put the secret hash in the same file that is used to record the used tokens. Generally, you don’t want to allow your security-sensitive applications to write state data to the file containing the secret.
Unfortunately, there was really not much else available that was both open-source and satisfied our requirements. Faced with that conclusion, we decided to roll our own solution that would both pass our security reviews and implement the functionality that we require. The Linux Foundation brought to the table the server component, and the Fedora Infrastructure — the PAM module. The final implementation therefore uses:
1. totpcgi, which is the provisioning and token verification server, and
2. pam_url, which is the pam module that communicates with totpcgi over a mutually-authenticated SSL link
In late November 2012, Red Hat sponsored a “Fedora Activity Day” that brought both Linux Foundation and Fedora Infrastructure sysadmins under the same roof to knock out the final showstoppers and implement two-factor authentication that supported both yubikeys and smartphone-based TOTP tokens. Despite some need for late-night hacking, the project was a resounding success and both infrastructures are now requiring two-factor authentication.
Both totpcgi and pam_url are released under GNU GPL and we hope that other distributed infrastructure projects will adopt and extend our offering in order to implement two-factor authentication on their servers.
There are no magic bullets
Security is a process, not a product. There is no one thing you can install or implement that would provide any kind of assurance against malicious hacking. Two-factor authentication makes the attackers’ lives more difficult, but it, too, can be defeated given enough time. For example, if you cannot trust the system binaries on your workstation, you can never be sure if you are actually connecting to your servers or if the attacker is simply simulating the login and recording your one-time-token, which they can then use to log in to your actual servers.
Observe the following best practices to make it more difficult for attackers to exploit your workstation:
• Never turn off SELinux on your workstation. Your browser is actually running inside its own constrained SELinux domain, which means that even if attackers manage to escape the browser’s sandbox, they will have a hard time escaping the SELinux jail in order to gain a foothold on your system and install a keylogger.
• Use NoScript for Firefox or ScriptSafe for chrome/chromium. Only allow javascript and plugins on the sites you trust.
• Keep your workstation patched. Always apply critical security errata as soon as it is available.
• Require two-factor authentication on your workstation for sudo, or only do it by switching to a text console (Ctrl-Alt-F2) and logging in as root.
• When you need to use ssh, always execute it as /usr/bin/ssh. Don’t trust your $PATH.
• Do the same when you use “sudo” on your server. Always type “/usr/bin/sudo -i”.
• Require two-factor authentication when obtaining elevated privileges on all your infrastructure.
• Routinely review account activity logs on your servers. Software such as logwatch or epylog will help you detect anomalous logins.
While these steps won’t guarantee that you are safe from malicious attacks, at least you will make the attackers’ lives extremely difficult while only introducing mild inconveniences into your regular sysadmin routines.