TLS Security vs. The Real World

By Bjarni Rúnar 2016-03-08, 11:46

Executive Summary

1. Versions of pagekite.py older than v0.5.7a use SSLv3, which is now considered insecure.

2. The public PageKite relays run on Debian Linux. Debian 8 does not support the SSLv3 protocol at all.

3. Old pagekite.py deployments need to be upgraded before May 2018. Sooner is obviously better; these old deployments can only communicate insecurely today. Legacy versions of pagekite.py will only be supported by the PageKite.net public relays for as long as Debian 7 is receiving security updates (until May 2018).

4. Libpagekite users should test their deployments to ensure their builds support TLSv1 or better; this depends on which version of OpenSSL you have linked with your app.

5. The latest versions of libpagekite and pagekite.py support versioned front-end relay discovery to make future compatibility issues easier to manage. If you are deploying PageKite in environments where you may have difficulty updating it in the future, please deploy pagekite.py v0.5.8e or libpagekite 0.91.x (or later).

White-label customers should also consult the very end of this post for more detailed configuration instructions.

How does PageKite use TLS/SSL?

PageKite is, in essence, a reverse proxy that has been split in two.

Reverse proxies and load balancers are a common feature of high-performance scalable web architectures. A reverse proxy like haproxy, nginx or Varnish is deployed as a public-facing "front-end", and it proxies requests to one or more origin servers at the "back-end".

PageKite works in largely the same way, except the reverse proxy itself comes in two pieces: the "PageKite front-end" runs in the cloud (usually on pagekite.net shared infrastructure) and the "PageKite back-end" runs on the same machine as the origin web server. The PageKite back-end creates a persistent tunnel to the front-end and together they relay the traffic, making your local servers accessible.

This traverses NAT and most firewalls, because the connection between the PageKite components is an outgoing connection from the back-end, usually secured using Transport Layer Security (TLS/SSL). The PageKite tunnel looks almost exactly like any other outgoing HTTPS connection, so all but the strictest firewalls let it through.

Encrypting the tunnel serves two purposes:

1. It prevents transparent proxies and buggy network security tools from corrupting the stream of data.

2. It prevents mischievous people on your local LAN (e.g. public WiFi) or at your ISP from listening in or modifying your traffic. In particular this prevenst eavesdroppers from trying to steal your PageKite credentials, which would allow them to impersonate your sites.

Although we recommend that any high-security applications use end-to-end encryption so the PageKite relays never see clear-text traffic (SSH does this by default, and our support for TLS/SNI makes this possible for HTTPS web sites too), the PageKite tunnel encryption is still very important.

That security, the encryption of the PageKite tunnel, is what the rest of this post is about.

TLS support in PageKite

The first draft of encryption for PageKite tunnels was added to pagekite.py (actually to the SOCKS support library) in June 2011. At first, we used Python's wrap_socket method unmodified, which was added to Python in version 2.6.

Later that summer (inspired by an informal review with Jacob Applebaum of the Tor project), support was added for configuring the cipher selection - and more importantly - support for the insecure SSLv2 protocol was removed.

At the time, due to limitations of the Python bindings, the only way to disable SSLv2 was to explicitly select SSLv3 or TLSv1. It was not possible to request "best possible" protocol negotiation and disable SSLv2; such features didn't arrive in Python until version 2.7.9 which was released near the end of 2014 and arrived in mainstream Linux distributions even later.

In 2011, TLSv1 support was not still universal and SSLv3 was still considered secure. So that's what we chose: PageKite was hardcoded to use SSLv3.

In the meantime, vulnerabilities were found in SSLv3 and its use is no longer recommended.

In September 2015, pagekite.py's SOCKS library was updated to use the new Python 2.7.9 features and deprecate SSLv3. This is done in such a way that when the system Python supports it, protocol negotiation is used to choose the best available standard. On legacy systems where the APIs do not support protocol negotiation without also enabling SSLv2, TLSv1 is the hard-coded fallback. When running as a front-end relay, SSLv2 and SSLv3 are disabled by default, unless the --tls_legacy flag is used, which will re-enable SSLv3 so as to support obsolete versions of pagekite.py.

Managing the Transition

Currently, most of the PageKite.net public relays are running pagekite.py on Debian 7 in legacy mode, thus supporting SSLv3.

Although this is insecure, we have a large number of legacy clients that depend on our services: version 0.5.6d is still by far the most popular version of pagekite.py, run by about 60% of all active users. Shutting them out overnight is not an option.

However, this can not last forever. This little company does not have the resources to handle OS security patching ourselves, we rely on the excellent Debian security team to keep our systems safe. Since Debian 7 will not receive security updates after May 2018, we aim to have the majority of our servers upgraded to Debian 8 or better well before that time.

This poses a problem: the OpenSSL library shipped with Debian 8 does not implement the insecure SSLv3 at all!

This means if we upgrade our existing relays, users of obsolete versions of pagekite.py will be unable to access the service. Further, due to limitations of how pagekite.py chooses its relays, even adding new servers to the pool will cause problems.

The only solution to this, is for new versions of pagekite.py (and libpagekite) to use a different set of relays. This is simple enough, relays are discovered through DNS lookups: frontends.b5p.us is the current relay pool name. Switching to a new name will allow us to direct different versions of pagekite.py to different sets of relays.

Learning from our past mistakes, we are from now on creating a new DNS name for every release of pagekite.py and libpagekite, to allow us to better manage similar problems in the future. I call this "versioned front-end relay discovery", and it is enabled in pagekite.py 0.5.8e and libpagekite 0.91.x, both released last week.

Please upgrade!

No new relay will be added to the old pool, quite the opposite, it will shrink over time.

So even if you are currently running a secure version of pagekite.py (0.5.8a is common), it is worth upgrading so you are using the modern set of relays.

A Brief Rant

I am very disappointed that the Debian OpenSSL packagers decided to break backwards compatibility. Debian 6 shipped Python 2.6, and Debian 7 shipped Python 2.7.3.

It wasn't until Debian 8 that the standard Python was recent enough to write secure Python SSL clients. Deciding in the same release to break compatibility with insecure clients, in effect means Debian 8 servers refuse to talk to Debian 7 clients (Python).

That's not cool!

Security is important, but so is reliability.

Unreliable systems cost users as much (probably more) than security vulnerabilities - and by breaking compatibility, Debian made the whole Internet a bit less reliable. Systems that worked just fine before will suddenly fail, just because someone upgrades their server operating system. The (quite reasonable) limits on Debian's long term support effectively guarantees this must happen.

But the fact is, that although servers are relatively easily upgraded, not everyone can easily upgrade their clients. Not every device has a sysadmin and not every device has automatic updates.

I think in the rush to improve security, this was overlooked.

It remains to be seen whether PageKite will be able to stick with the plan outlined above, or whether we'll be forced to keep using an outdated end-of-life operating system to provide support to devices in the field, gadgets people paid for and expect to keep working.

All we can do is hope for the best and encourage our users to upgrade.

Instructions for White-label Customers

If you are a PageKite white-label customer, you will want to update your device settings as soon as possible and ensure that new devices are shipped with up-to-date software and manageable settings.

libpagekite

1. Build against a modern version of OpenSSL

2. Update to version 0.91.160307C or later of libpagekite and use the pagekite_init_whitelabel (or Java equivalent) method to configure the library. This will automatically use versioned front-end relay discovery (and the appropriate dynamic DNS and SSL certificate settings).

pagekite.py

1. If possible, update to pagekite.py 0.5.8e or later.

2. Add the following front-end definition to your configuration file (after "defaults", if present):

   frontends=1:fe4_058.YOURDOMAIN.COM:443

If you are not version 0.5.8, replace the version number in the URL with the actual version of pagekite.py that you are using (digits only, no periods or letters). Versions 0.5.6 and up have corrosponding DNS names for front-end discovery.

(Future versions of pagekite.py will add a --whitelabel argument for similar convenience as offered by libpagekite).