Heartbleed vulnerability may have been exploited months before patch [Updated]

Update: Errata Security's Robert Graham has acknowledged that he was mistaken in his assessment, and that private keys could be at risk. The original story below has been marked up accordingly.

There’s good news, bad news, and worse news regarding the “Heartbleed” bug that affected nearly two-thirds of the Internet’s servers dependent on SSL encryption. The good news is that many of those servers (well, about a third) have already been patched. And according to analysis by Robert Graham of Errata Security, the bug won’t expose the private encryption key for servers “in most software" (though others have said several web server distributions are vulnerable to giving up the key under certain circumstances.) 

The bad news is that about 600,000 servers are still vulnerable to attacks exploiting the bug. The worse news is that malicious “bot” software may have been attacking servers with the vulnerability for some time—in at least one case, traces of the attack have been found in audit logs dating back to last November. Attacks based on the exploit could date back even further.

Security expert Bruce Schneier calls Heartbleed a catastrophic vulnerability. "On the scale of 1 to 10, this is an 11," he said in a blog post today. The bug affects how OpenSSL, the most widely used cryptographic library for Apache and nginx Web servers, handles a service of Transport Layer Security called Heartbeat—an extension added to TLS in 2012.

Heartbeat allows a connected Web client or application to send messages to keep a connection active during a transfer of data. When a Heartbeat message is received, the server usually simply echoes back what it got to the sender. However, starting with the initial implementation of Heartbeat in OpenSSL 1.01 (and in all subsequent releases up to OpenSSL 1.01f, including the OpenSSL 1.0.2 beta) the extension could be fooled into sending back the contents of its memory buffer by sending a request that advertised itself as 64 kilobytes long but in fact had no content—resulting in “Heartbleed.” Any information still in that buffer from a previous session, such as decrypted usernames and passwords, could be obtained by an attacker in the response message.

Your key is (probably) safe not safe at all?

In a post to the Errata Security blog, Robert Graham explained that it is highly unlikely that private key data would be stored in the memory buffer that could be leaked using the Heartbleed exploit. “What you can eavesdrop on with Heartbleed hacks is dynamic stuff, stuff that was allocated only moments ago,” he wrote. But that assertion has been refuted, and Graham has since rescinded it, as noted above. 

Researcher David Litchfield said that the default Web server distribution on FreeBSD will give up the private key if the attack is the first request after the server is reset. Others claim to have successfully retrieved keys under other circumstances with the exploit. Some test attacks have been tuned based on the memory allocation patterns for specific server versions. The Codenomicon team behind the disclosure of the vulnerability said that they were able to retrieve the private key from their own server's X.509 certificate. 

A scan performed by Graham Tuesday night found that of the 28 million servers and other devices that responded to an SSL connection request, only about 600,000 were vulnerable to attack using Heartbleed. “We also found 33,531 machines that had Heartbeats enabled, but which did not respond to the Heartbleed attack,” Graham wrote. “Presumably, this means a third of machines had been patched by the time we ran the scan last night.”

Closing the barn door

The reduced likelihood of exposure of private keys—which would allow an attacker to masquerade as the attacked server or client—is cold comfort, however, considering that usernames and passwords on some systems may have been exposed for months or years by the vulnerability, which has been part of every OpenSSL release since March 2012. There are signs that exploits for the vulnerability were in use by someone for some time before the vulnerability was revealed.

Ales Teska, of the mobile security provider SeaCat.mobi, wrote in an email to Ars that his company’s service, while not vulnerable to the Heartbleed exploit, acted as a sort of “honeypot” for attacks based on the exploit because of its use of OpenSSL. “Our OpenSSL-based software was logging Heartbleed attack attempts to its logs by coincidence,” he wrote, starting on March 23. When upgrading the OpenSSL on two test servers, he checked the logs of the servers and found “these two servers are actually logging such an attempt to the log file (as a generic OpenSSL handshake issue).” SeaCat.mobi has detailed the attempted attacks in a blog post.

Update: SeaCat and Teska have now qualified their comments: " EFF correctly pointed out that there are other tools, that can produce the same pattern in the SeaCat server log (see http://blog.erratasec.com/2014/04/no-we-werent-scanning-for-hearbleed.html#.U0Xq4OaSz-l ). I don't have any hard data evidence to support or reject this statement. Since there is a risk that our finding is false positive, I have modified this entry to neutral tone, removing any conclusions. There are real honeypots in the Internet that should provide final evidence when Heartbleed has been broadly exploited for a first time."

Terrence Koeman of MediaMonks told Ars he found signs of attempts to use the exploit dating back to November 2013. He used the packet content of a successful exploit of the Heartbleed vulnerability to check inbound packets logged by his servers and found a number of incoming packets from a network suspected of harboring a number of “bot” servers that were apparently scans for the vulnerability—sending Heartbleed-style requests to two different development servers in requests that were about five minutes apart.

Apache and nginx servers aren’t the only potential victims of the OpenSSL vulnerability. A malicious server can be used to attack client applications that use OpenSSL 1.0.1. Fortunately, few Web browsers do—Chrome on Android 4.1.1 may be vulnerable, for example, since it uses OpenSSL 1.0.1e, but most Chrome and all Firefox browsers use the Mozilla Network Security Services (NSS) libraries. Apple uses its own cryptographic services for OS X and iOS; an earlier, unaffected version of OpenSSL is included in OS X, but Apple discourages its use.