A Real World Comparison between Cloudflare Workers and Fastly Compute@Edge

Over the past couple months the race to build the best edge compute platform has really heated up. We're seeing companies like Cloudflare, Fastly, AWS and Fly building compelling platforms to run code as close to your users as possible. Gone are the days of single compute instances handling many requests, we're entering a new era of compute where each request gets its own isolated container and the ability to scale to thousands, even millions, of requests per second.

Although there are many comparisons still to be done between all platforms, I want to take this opportunity to focus on Cloudflare and Fastly as the two companies have been going back and forth in what I would consider a largely meaningless feud. The saga began with Cloudflare testing their JavaScript runtime against Fastly's JavaScript runtime (still in beta) in a basic hello world test. The test was simple: how fast can each runtime return a hello world response. In this case "hello world" simply meant replying with a JSON response of the current request headers. If you're just dying to know how fast each platform could return such a response, let me spoil it for you: really fucking fast!

Both platforms were returning this response in under 100ms up to the 90th percentile. I don't know about you, but we dont have many hello world endpoints in production so this wasn't exactly going to sell us in either direction. What we wanted was a more robust example of moving a traditional server workload to the edge, and thats what I plan to show you today. Importantly, I think a comparison between any of these platforms needs to go well beyond just time-to-first-byte. We care a lot more about developer experience, framework support, CI/CD, and the other things that make development teams happy and more efficient in their everyday work. My goal is to give a comprehensive overview of the following:

1. TypeScript Language Support
2. JavaScript Platform APIs
3. Deploying with GitHub Actions
4. Performance Comparison
5. Platform Limitations

Before we get into our comparisons, it's important to understand the production workload that I've re-written for each platform. Internally we call this product Pipe-Stream: it's goal is to take a list of MP3 files and stream them together, in order, as a single MP3 file. We use this technology for quite a few different things - but one of the obvious benefits is to swap out ad reads dynamically as we get new ad partners throughout the year. Our Podcast API pre-chunks our mp3 files into segments so they are 100% ready to be combined into a single stream. Pipe-Stream does not know anything about the mp3 spec, it's job is to simply concatenate the segments into one streaming response. And the "streaming" aspect of this service is really important. We have some MP3 files over 1GB in size and thus we do not want to pull all that data into memory. Streaming should be completely pass-through so we can optimize time-to-first-byte as well as runtime memory.

TypeScript Language Support

I'm happy to report that both platforms have excellent support for developing your application in TypeScript. Each platform provides first-class types for their platform API's, making it dead simple to ensure your code is always using their API's correctly. Each platform compiles your TypeScript using webpack, and the webpack config files are nearly identical between the platforms. As of writing, Cloudflare gets the slight edge in getting started as they provide a one-line-command to create a new TypeScript workers project. Fastly provides a similar one-line-command for a JavaScript project, but it's up to you to figure out how to add webpack and get it to build. Hint: copy the Cloudflare webpack file and dependencies.

JavaScript Platform APIs

Okay so the TypeScript support is nearly identical between the platforms, but what can we actually do in TypeScript (compiled to WASM) on each platform? There are some critical components coming from our production code running on Node.js that need to be available in each runtime:

1. Readable Streams
2. Writable / Transform Streams
3. HTTP Requests with Streaming Bodies

Readable Streams

A readable stream is perhaps the most important aspect of this entire project. Our goal is to stream each MP3 segment from the source (in our case Amazon S3) to the client. We must avoid reading the entire file into memory and instead stream the response directly to the client. This can only be achieved if the platform supports the concept of a readable stream. In Node.js this looks like:

import { Readable as ReadableStream } from 'stream'

In both Fastly Compute@Edge and Cloudflare Workers ReadableStream is a global class, so no need to import it to use it. Their implementations of ReadableStream follow the same spec as the Web API. This conformance to the Web API is a common theme that you will see throughout this post. Each platform makes a strong effort to conform to the Web API as much as possible, but each has their own differences and trade-offs which we will cover in another section.

Writable / Transform Streams

In order to successfully combine multiple readable streams into a single destination stream, we must pipe the streams through what's known as a TransformStream. A TransformStream is once again part of the Web API and it provides both a writable and readable stream. You can write data to the writable stream and that data is made available on the readable stream. In Node.js this looks like:

import { Transform, Readable, Writable } from 'stream'

export function combineStreams(streams: Readable[]): Readable {
  const stream = new Transform()
  _combineStreams(streams, stream).catch((err) => stream.destroy(err))
  return stream
}

async function _combineStreams(sources: Readable[], destination: Writable) {
  for (const stream of sources) {
    await new Promise((resolve, reject) => {
      stream.pipe(destination, { end: false })
      stream.on('end', resolve)
      stream.on('error', reject)
    })
  }
  destination.end()
}

This block of code is doing the following:

1. A function called combineStreams accepts an array of readable streams to stitch together
2. It creates a new TransformStream
3. Loops through each readable stream and pipes it to the transform stream
4. Prevents closing the transform stream during each pipe call
5. Closes the transform stream once all streams have been combined
6. Returns the transform stream synchronously so it can be used by the caller

Node's TransformStream differs from the Web API in one key way - it is both a Readable and Writable stream and thus can be returned as either without the caller knowing it is one or the other. The Web API provides a slightly different spec where a TransformStream is actually not a stream at all, but a class the exposes both readable and writable streams as properties of the class.

Here is the exact Cloudflare implementation of the previous Node.js code:

export function combineStreams(streams: ReadableStream[]): ReadableStream {
  const stream = new TransformStream()
  _combineStreams(streams, stream.writable)
  return stream.readable
}

async function _combineStreams(sources: ReadableStream[], destination: WritableStream) {
  for (const stream of sources) {
    await stream.pipeTo(destination, {
      preventClose: true
    })
  }
  destination.close()
}

Amazingly: this is less lines of codes than the Node.js implementation and even comes with an async version of Stream.pipe, greatly cleaning up our code. So how does Fastly compare? Well the Fastly implementation is actually identical to the Cloudflare implementation. This is a huge win for developers looking to experiment on both platforms.

HTTP Requests with Streaming Bodies

At this point we have the ability to successfully combine readable streams into a single readable destination stream. Now we need a way to actually download content from Amazon S3 and return the data as a readable stream. Lucky for us, both Cloudflare and Fastly implement fetch from the Web API, but with one major difference.

Fetching data with Cloudflare is as easy as:

const urls = [...]
const requests = urls.map(url => fetch(url.href))
const responses = await Promise.all(requests)
const streams = responses.map(res => res.body)

Although fetching data with Fastly is similar, there is one major difference in that the hostname of all fetched resources must be defined as Fastly Backends. This wont come as a surprise to anyone familiar with Fastly's VCL platform, but I bet this will be a major hangup for new customers coming from a more traditional web background. Assuming our hostnames are defined as backends, fetching data with Fastly is nearly as easy as Cloudflare:

const urls = [...]
const requests = urls.map(url => fetch(url.href, {
  backend: url.hostname
}))
const responses = await Promise.all(requests)
const streams = responses.map(res => res.body)

Deploying with GitHub Actions

We're finally ready to deploy our code to each platform, and at Barstool this involves setting up a GitHub actions workflow. I'm happy to report that both platforms provide a GitHub Actions steps that makes it dead simple to deploy your code. Here's our workflow for Cloudflare:

name: Deploy Application

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Use Node.js
        uses: actions/setup-node@v2
        with:
          node-version: 16
          cache: yarn

      - run: yarn install --frozen-lockfile

      - name: Deploy to Cloudflare
        uses: cloudflare/wrangler-action@1.3.0
        with:
          apiToken: ${{ secrets.CF_API_TOKEN }}

And here's our workflow for Fastly:

name: Deploy Application

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Use Node.js
        uses: actions/setup-node@v2
        with:
          node-version: 16
          cache: yarn

      - run: yarn install --frozen-lockfile

      - name: Deploy to Compute@Edge
        uses: fastly/compute-actions@beta
        env:
          FASTLY_API_TOKEN: ${{ secrets.FASTLY_API_TOKEN }}

Each step correctly uses the build command in your package.json which should compile your code with Webpack and then bundle it for deployment. There's technically a bit more to it than simply defining this workflow, as each platform has their own configuration file for setting up domains, environments, etc. That's slightly out of scope for this article, but I can assure you both are pretty complete and easy to define your infrastructure as code.

Performance Comparison

Finally. Let's see how these platforms perform with a real world use-case. In order to provide a fair comparison I'm going to run the tests without any CDN in front of our production workload on EC2, however, I am going to enable caching on the origin requests as each platform supports caching fetch requests and this is a critical part of the performance profile for our use case. If we don't cache the origin requests our Amazon S3 bill would be astronomical.

The way pipe-stream works is it provides a single route GET /stream and requires a JWT to be passed as ?token=. Inside the JWT is the list of urls that need to be combined into a single stream. We use a JWT to ensure a bad actor cannot use our service to stream whatever files they want. The JWT is signed by a Barstool private key so we can validate it and ensure the JWT came from one of our services.

The first file we are going to test is a 3MB file defined by the following JWT payload:

{
  "iat": 1624309200000,
  "exp": 1624395600000,
  "data": {
    "u": [
      "https://cms-media-library.s3.us-east-1.amazonaws.com/barba/splitFile-segment-0000.mp3",
      "https://cms-media-library.s3.us-east-1.amazonaws.com/barba/splitFile-segment-0001.mp3",
      "https://cms-media-library.s3.us-east-1.amazonaws.com/barba/splitFile-segment-0002.mp3"
    ],
    "c": "audio/mpeg"
  }
}

At runtime, the platforms will fetch each file in the array and stream them back as a single combined MP3 file. Here are the test urls for each platform:

1. EC2
2. Cloudflare
3. Fastly

I performed all tests from my apartment in Brooklyn, NY on a 1GB Verizon FIOS connection. At the time of testing, I was consistently getting 580mbps according to fast.com. The benchmark was performed using a custom Deno script which fetches each url 100 times and then computes the average time-to-first-byte (TTFB) and average time-to-download (TTD). Here are my results:

Next I wanted to test streaming a much larger file. Below are 3 more urls for a 70MB file:

1. EC2
2. Cloudflare
3. Fastly

And here are the  results from fetching these urls 10 times on each platform:

The first thing to notice is TTFB is considerly better on the Edge platforms. This shouldn't be much of a surprise, as this is precisely what the original blog posts were showcasing from both Fastly and Cloudflare. Those blog posts did a much more robust analysis, testing TTFB from a variety of locations around the world. For the sake of time I did not perform my tests anywhere other than Brooklyn, NY. Keeping this in mind, it's still hard to ignore Fastly's results when it comes to TTFB. The project we're testing is doing considerably more work in the WASM runtime than the original blog posts, yet Fastly's runtime is optimizedto such a degree that you would barely notice.

Things become more interesting when it comes to downloading the entire stream. Keep in mind we're not downloading a single static file, the runtimes are stitching multiple static files together. Both Cloudflare and EC2 had similar performance characteristics, but Fastly managed to stream the entire smaller file 33% faster and the larger file 25% faster. Given this project is very IO heavy, and we enabled caching on the origin requests, this is also testing the pure CDN performance of each platform rather than just the WASM runtimes.

[2022-02-04] Update:

A previous version of this blog post tested an earlier version of Fastly's SDK which led to much slower performance than both Cloudflare and EC2. Since the initial post, the Fastly team has released the latest version their JS SDK with a native TransformStream and fixes to pipeTo. In all of our tests using the latest version of the SDK, Fastly has outperformed both EC2 and Cloudflare.

Platform Limitations

Aside from the discussed limitations of Fastly's fetch API, there's another limitation that's present on both platforms: Content-Length headers are not returned on streaming responses, even if we pre-compute the content length and set it ourselfs. This is actually turning out to be a major blocker in terms of officially migrating away from EC2 and onto one of the edge platforms.

We've spoken to both Cloudflare and Fastly about this limitation, and their teams are aware and looking for ways to fix it. Although we've received minimal details about what it will take to implement a fix, it's clear there are issues in the Web API fetch specification that prevent setting a content-length when using chunked encoding. In our case, we're not really using chunked encoding as we know the total amount of bytes between the MP3 files, but the runtime doesn't. I found some additional details regarding the issue here.

[2021-12-18] Update:

The tech lead for Cloudflare Workers responded to us on HackerNews and showed us how to correctly stream with a Content-Length header. Cloudflare implemented a non-standard FixedLengthStream class that allows passing the known content length into the constructor. Updating our implementation was as simple as:

const responseStream = new FixedLengthStream(contentLength)

combineStreams(streams).pipeTo(responseStream.writable)

return new Response(responseStream.readable)

If you view any of the Cloudflare URL's above you will now see the correct Content-Length returned in all cases.

Final Thoughts

I think the future is incredibly bright for this new age of edge computing, and I'm excited for both Fastly and Cloudflare to continue to improving their platforms and taking away market share from the 3 big major cloud vendors. Clear competition across the industry can only mean a better product for us, the developers. If you're interested in working on projects like this, check out barstoolsports.com/jobs or shoot me an email at barba@barstoolsports.com