Cloudflare's Workers Battle Spectre: How Serverless Code is Shielded from CPU Side-Channel Attacks

Cloudflare runs a global network that stores and serves website content, and its Workers platform lets developers run code on that network. But running code in the cloud means thinking about security at every layer. One of the biggest challenges is defending against processor vulnerabilities like Spectre. In a detailed blog post on July 29, 2020, Cloudflare's Kenton Varda explained how the Workers platform mitigates these threats. His post covers a variety of techniques, including restrictions on timing, threading, and process isolation.

The first weapon against side-channel attacks is to block the code from measuring time locally. In Workers, the Date object is tamed so that the start and end values of any operation always come out exactly the same. This removes the basic tool an attacker would need to build a timing side channel. A side channel is a way to infer secret data by observing indirect clues, like how long a computation takes. Cloudflare implemented this in mid-2017, well before Spectre was publicly announced. The team was worried about timing side channels from the very beginning and designed the system with that concern in mind.

Workers also cannot use multi-threading or shared memory. Multi-threading means running several tasks at once, and shared memory means multiple threads can access the same data. If either were allowed, a malicious Worker could race threads to create a hidden timer. So every event is processed on a single thread, and even if multiple Workers handle the same request, they run sequentially on the same thread. This prevents the creation of implicit timers.

These measures stop code from measuring its own execution time. But an attacker can still measure it remotely. The HTTP client sending a request can simply time how long it takes to get a response. That measurement is noisy because Internet traffic is unpredictable. But an attacker could perform the attack many times and average the results to filter out the noise. Cloudflare has already considered resetting application state between requests, but that does not help because the attacker's client can store the state and send it back on the next request. Cloudflare says that, with help from prominent Spectre experts, it has not been able to produce a real attack that works in production. Yet the lack of a working attack does not stop the company from building defense in depth. It is currently testing more advanced measures.

One of those measures is dynamic process isolation. Spectre attacks are very slow by nature. They need hours or weeks to succeed. But even after one second of execution, an attack leaves a signature. Spectre attackers deliberately create unnatural performance patterns, especially when they are forced to loop billions of times to get past earlier defenses. These patterns appear in CPU performance counters, which are special registers that record statistics like cache misses and branch predictions.

The problem with performance counters is that legitimate code can also look weird. A program with poor performance could get flagged as suspicious. Cloudflare does not want to shut down such programs. Instead, it can reschedule a Worker that shows suspicious performance into its own process. A process is a separate memory space for a program. Once a Worker is in its own process, the operating system's built-in Spectre defenses can kick in, much as they do for desktop web browsers. This protects the Worker without harming its functionality, although there is a slight overhead.

Isolating every Worker would be far too expensive. But isolating a few is acceptable. And the more CPU time a Worker uses, the lower the relative cost of isolation becomes. So Cloudflare uses process isolation for any Worker that is CPU hungry. It is also fine to reschedule a Worker at any moment, since the platform is flexible.

Cloudflare has been working on this method for a year with experts at Graz Technical University in Austria. That team co-discovered the Spectre vulnerability and has been responsible for a huge number of follow-on discoveries. The collaboration highlights the importance of research in staying ahead of security threats.

For website owners and IT teams, this deep dive shows how complex security can be behind the scenes. While Cloudflare protects its Workers from side-channel attacks at the CPU level, you can add another layer of protection for your own traffic by using a privacy-first DNS like AEU DNS, which blocks malicious domains and keeps your DNS queries encrypted. That way, even if attackers try to reach you through known bad sites, the request stops before it reaches your browser.