Tuesday, April 15, 2014

The Heartbleed vulnerability: how does it apply to you?

We live in a world where technical vulnerabilities can sometimes be a dime a dozen. Let's face it, what with Microsoft's Patch Tuesday, the latest stream of Adobe threats, and the problems with Java and Javascript, it can be overwhelming to keep up on the latest big risks in IT and whether they really apply to your environment. This is compounded by the fact that many well-publicized vulnerabilities may not always have a visible impact, making us a bit lackadaisical or blasé.

heartbleed-1.png
However, if you work in technology, your job is to not be lackadaisical; it's your responsibility to take each risk seriously and give it your utmost attention since security is everyone's problem. Critical Internet Explorer flaws might not mean much if your users are all on Firefox, but what about the home machines they use to connect to the company? We're all in the same swimming pool when it comes to security.

With that in mind, a vulnerability known as Heartbleed (or CVE-2014-0160) was recently discovered in the OpenSSL 1.01 and 1.02 beta product. This is used on web servers, email servers, virtual private network (VPN) systems and some client applications, proving how widespread this threat can be.

Heartbleed hysteria went violently viral in a span of literal hours, kicking off on Monday, April 7. Just a few days later it grew bigger than the Super Bowl. As a system administrator I can state that I have rarely - if ever - seen a threat like this gain so much press so quickly. CNET has posted several articles on the topic, including Heartbleed bug also affects Cisco and Juniper equipment, while ZDNet covered How to protect yourself in Heartbleed's aftershocks andHeartbleed's engineer: It was an 'accident'.

What is Heartbleed?

The technical description states that "The (1) TLS and (2) DTLS implementations in OpenSSL 1.0.1 before 1.0.1g do not properly handle Heartbeat Extension packets, which allows remote attackers to obtain sensitive information from process memory via crafted packets that trigger a buffer over-read, as demonstrated by reading private keys, related to d1_both.c and t1_lib.c, aka the Heartbleed bug."

Essentially, it's a flaw in a product called OpenSSL which, ironically enough, is supposed to secure web traffic through encryption. This flaw is based on a "keep-alive" setting which can provide malicious attackers the ability to obtain up to 64 KB of unencrypted sensitive data from the memory space of a vulnerable OpenSSL server or client. It can expose passwords, emails and financial information or get private keys used for encryption - any of which could produce devastating results. The full technical details are here.

The technology-oriented comic strip XKCD summed it up nicely:

heartbleed-2.png

Anything running OpenSSL 1.0.1 through 1.0.1f is vulnerable to the Heartbleed threat. An advisory site called heartbleed.com designates these operating systems as being "potentially vulnerable":

On the same note, the site says these operating systems are not vulnerable:

  • Debian Squeeze (oldstable), OpenSSL 0.9.8o-4squeeze14
  • SUSE Linux Enterprise Server
  • FreeBSD 8.4 - OpenSSL 0.9.8y 5 Feb 2013
  • FreeBSD 9.2 - OpenSSL 0.9.8y 5 Feb 2013
  • FreeBSD Ports - OpenSSL 1.0.1g (At 7 Apr 21:46:40 2014 UTC)

Don't conclude only Linux servers are at risk; Windows servers may also susceptible to this condition if they happen to be using IIS with the wrong kind of OpenSSL.

This threat is not theoretical; this is really happening. It has impacted Yahoo, for instance (and they have scrambled to correct it). Google, Amazon and Facebook have also addressed the subject; Google claims they "fixed this bug early" and Amazon stated the same, claiming that their sites are no longer affected by it. However, Netcraft.com states that " Half a million widely trusted websites vulnerable to the Heartbleed bug. " To put it another way, Kelly Jackson Higgins of Darkreading.com wrote that " 17 percent of SSL-secured websites [are affected]."

Many websites your users connect to might be vulnerable - and worse, your systems might be compromised as well. Cue the scary horror movie bass.

Why is this suddenly a big deal?

The crazy thing about this one is that the vulnerability has existed since the end of 2011 and has been escalating for two years. The term "security through obscurity" may have applied before now since it wasn't well-known but, simply put, now it's a big deal since the bad guys have hopped on the train and are scanning servers looking for an opportunity to exploit them.

Furthermore, it's a special kind of threat because it consists of a double-whammy: both your clients and servers may be at risk. Last but not least, there is no way to tell whether your data or credentials have been affected, until they are misused by someone else.

What can we do to protect our systems and our customers/users/selves?

It seems everyone in the IT industry is weighing in on the topic. Dodi Glenn, senior director of security intelligence and research labs at ThreatTrack Security, said:

"If a server administrator is running 1.0.1 or 1.0.2-beta of OpenSSL, they should upgrade as soon as possible. The vulnerability has been fixed in OpenSSL 1.0.1g, however, if they cannot upgrade to the patched version, they can disable heartbeat support, which is where the vulnerability exists. If a company has been running with one of the vulnerable versions of OpenSSL for a decent amount of time, they should assume that their certificates and keys have been compromised. They should begin the process of replacing these keys and certificates, particularly if the server in question contained sensitive data. Additionally, companies running the vulnerable versions of OpenSSL should advise their customers to change their passwords. Administrators can upgrade their version of OpenSSL by visitinghttps://www.openssl.org/source.

If the server is running an older version of OpenSSL such as 0.9.8, they do not need to upgrade, as this bug was not found in this version. Additional information regarding the vulnerability (CVE-2014-0160) can be found here: https://www.openssl.org/news/secadv_20140407.txt....

"This bug is serious and you need to worry about it - whether you are a user or a systems administrator," said Gary McGraw, CTO of software security firm Cigital and speaker at Rock Stars of Cybersecurity. "If you are a user, change your passwords on all the websites you use. Really. Do it now. If you are a sys admin, see if you are vulnerable with tests athttp://filippo.io/Heartbleed/ or https://www.ssllabs.com/ssltest/index.html. Upgrade your server software. Retest. Generate a new SSL/TLS key and get a certificate for the new key. Revoke your old certificate. Have your users change their passwords."

The bug has existed for two years and it allows attackers to steal the keys that protect communication, user passwords, stored files, bank and financial information, and even social security numbers, in a way that goes unnoticed without leaving any trace. Some web sites have already announced they have fixed the problem, but it's difficult to know what data has been compromised."

With this advice in mind, I'll break the recommended action items down to two categories: systems and users.

Here's what you need to do for your systems

Look for what you need to patch

Since anything running OpenSSL might be at risk, you need to be aware of your environment and check all servers, devices or applications for anything running OpenSSL 1.0.1 through 1.0.1. This should apply to both internal and external-facing systems; don't assume a server to which only your local users have access is safe. You can check public websites for the Heartbleed vulnerability using this test page: http://filippo.io/Heartbleed/

Checking physical devices is likely going to be the easy part. It will be more of a challenge to figure out which applications are affected. Vulnerable applications might include programs like wget, curl, lynx, and perhaps other third party command line apps. According to security.stackexchange.com, these programs were found vulnerable to Heartbleed:

  • MariaDB 5.5.36
  • wget 1.15 (leaks memory of earlier connections and own state)
  • curl 7.36.0
  • git 1.9.1 (tested clone / push, leaks not much)
  • nginx 1.4.7 (in proxy mode, leaks memory of previous requests)

Since every company is different, ultimately this comes down to looking at where certificates are used. Redirect IT staff from other projects if necessary until you've got a list in your hand.

Assess the impact

Before you go charging off with a fire hose, take a few moments to figure out what this means for your company. Was a public-facing server at risk? You'll need to involve your customers. Was your VPN system in harm's way? Your local and remote workers are now part of this plot. Perform a risk assessment to determine what information may have been accessible in a potential attack, and what you might need to do beyond technical measures to remediate this situation. For instance, were trade secrets kept on a Sharepoint site accessible via the internet? Involve the departmental manager and begin a dialogue to assess what might happen if those secrets were accessed.

Notify, notify, notify

If you found affected systems in your company which are accessed by customers and users, keep them in the loop on what you're doing. Put up a notice on your website, send out emails, and make phone calls where necessary. Let them know you will need to patch these systems (and perhaps reboot them, involving downtime unless you have redundancy), reissue certificates and change passwords - all in that order. Advise them that patching must come first to block the threat, and advise them to hold off on changing passwords until further notice (since they'll just have to do it again after the patch work is finished).

Establish timelines as to when this will be done and keep them in the loop throughout the process.

Apply applicable patches/fixes

Update the appropriate systems to OpenSSL 0.9.8, 1.0.0 or 1.0.1g. An advisory site calledheartbleed.com recommends that: "If an upgraded package is not yet available for your OS, software developers can recompile OpenSSL with the handshake removed from the code by compile time option '-DOPENSSL_NO_HEARTBEATS'"

Internal or external-facing systems should be patched with equal priority, but I recommend starting with the external-facing ones.

Replace certificates on any impacted systems

It's very simple (in theory): any systems you patched should now have certificates reissued with new keys, whether from an inside or outside certificate authority. This includes both server and client certificates. This is an excellent reason why inherent familiarity with your certificates and how they work will be an asset whether on quiet or chaotic days.

Change passwords on any impacted systems

You may be tempted to take this step first, but don't - it's of no value until the affected servers are patched. Once this has been done and the certificates replaced, then you should proceed with password resets.

Here's what you need to do for your customers/users

Let your customers know your status

All patched and cleaned up? Explain to your customers what else you're doing to ensure the security of their accounts and data. Even if you didn't find any systems vulnerable to Heartbleed, you should still communicate your status to them since they'll likely have heard of this threat and want to know whether your company - and therefore their information - was at risk.

Provide guidelines to advise users what to do about external systems

Even though some say there's nothing users can do until the servers are patched , there are still some precautions you can provide to your users (and which you should follow as well) in their approach to outside systems out of your control.

Advise them to review this list provided by Mashable and this list provided by CNET to see which popular public sites and email services may have been impacted (note that none of the banking sites on the list appear to have been affected). Also have them check this " Top 10K vulnerable site list" to see if anything they use appears there. Have them manually check websites for the Heartbleed vulnerability using this test page: http://filippo.io/Heartbleed/.

If users access systems which are (or were) vulnerable - or there is any doubt - advise them to not to access these systems until they are clean (the CNET list will be continuously updated, and of course any reputable companies will notify them of their status as well), and the SSL certificates involved have been changed. Users should then change their passwords on these systems.

It's critical to emphasize that even sites that are now clean aren't necessarily off the hook; if there was a period of time when these were vulnerable that means data or credentials could have been harvested - and therefore the risk is not yet mitigated until these are replaced.

Yes, advising people to change passwords on what could be dozens of websites is sure to elicit groans of dismay. But regardless of what the latest vulnerability of the day might be, a thorough password change process should be familiar to any user or administrator. Password management utilities such as LastPass, SplashID, KeePass and Password Safe can help keep track of this and also enact strong, secure passwords.

Finally, instruct users not to use command-line tools to access untrusted systems across the internet, and to be prepared for this issue to go on for some time - it won't be fixed in a day or even a week. Like an unwelcome visit from an in-law, Heartbleed is going to be a topic on our minds for quite some time to come. In the end it may be proclaimed the single biggest system vulnerability of all time.

Chatting with an expert

I was fortunate enough to be able to talk about Heartbleed with an expert named Justin Morgan, the information security officer at Litle & Co., a Vantiv company. Even though some of these points have been covered here already, Morgan fielded some of my questions on the topic with additional insights.

Scott Matteson: What is unique about Heartbleed, and how did this get so big, so fast?

Justin Morgan: "What makes Heartbleed unique is that it is a very small bug that has gigantic ramifications. Previous attacks on SSL/TLS have often been cryptographic in nature, meaning some aspect of the SSL/TLS security model was broken, whereas the Heartbleed bug is a simple bounds checking error. What makes it big is unlike previous attacks, which reduced the security of encrypted data in transit, Heartbleed exposes memory on the compromised host itself (both servers and clients). This means that an attacker could potentially read the contents of other web sessions in memory, or even the keys to the kingdom--the SSL certificate's private key."

SM: What operating systems are most affected by Heartbleed?

JM: "Primarily Unix-like systems, including Linux. Apple OS X, iOS, and Microsoft Windows are not affected. However, it's important to realize that these systems may have third party software installed that has packaged the OpenSSL library."

SM: I know that impacted systems should be patched to a later version of OpenSSL. Which version is safe?

JM: "Only OpenSSL 1.0.1 through 1.0.1f; 0.9.8 and 1.0.0 branches are not vulnerable (1.0.2 beta is vulnerable as well). It's an interesting phenomenon to keep in mind: a critical vulnerability was introduced with new feature code (the TLS heartbeat extension, or RFC 6520). Adopting the bleeding edge version of critical security libraries will always have this risk."

SM: Can you elaborate on the changing of SSL certificates; should any and all public-facing certificates be regenerated or are there certain stipulations involved?

JM: "Any certificate that was ever hosted on an internet-facing vulnerable version of OpenSSL should be revoked and replaced. The cost of exhaustively evaluating whether a certificate was in jeopardy is almost certainly going to be higher than the cost of simply replacing the certificate. This is also a good opportunity to make sure that your certificate key length and signature algorithms are 'up to code.'"

SM: Is it safe to assume that ALL passwords on external sites should be changed?

JM: "Not necessarily. The exploit can only access data that is resident in memory, which is going to be a subset of all the information stored and transmitted by the vulnerable system. However, any application or site a user may have entered their credentials on in the last two years should be updated (and that's probably most of them). As a side note, key-based authentication to vulnerable systems is not particularly compromised (i.e. you do not need to replace your SSH keys that you may have used to access vulnerable systems, but you should still rotate them periodically anyway)."

SM: How likely is it that this vulnerability has been exploited on a large scale prior to discovery of the bug on April 7?

JM: "It's hard to say. It certainly wouldn't be the first time a vulnerability had been discovered and exploited prior to a fix being developed. However, a successful exploit is difficult to detect even with active monitoring, and nearly impossible retroactively. Given that the bug is over two years old, my gut tells me that large scale use of an exploit would have been detected, but there's no way to know for sure."

SM: How would you recommend system admins best evaluate their environment to determine which devices are vulnerable to Heartbleed; security scans, manual check or something different?

JM: "If an organization has a mature configuration management database, the fastest way to take a first pass would be to pull a list of all the installed versions and compare it against the list of vulnerable versions. It's also important to keep in mind some distribution ecosystems maintain separate versioning through backports, so administrators should consult their distribution-specific documentation.

Online scanners can potentially identify vulnerable servers, and there are a few that have recently popped up. However, it's important to keep in mind that the vulnerability affects SSL/TLS clients as well, which would not get picked up in a scan.

Finally, library version checking and online scanning may not capture third party software that has packaged up vulnerable OpenSSL libraries. Even Windows administrators could be running third party software that is vulnerable, so it's important to inventory your software and review the security bulletins issued by vendors."

Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)