Gzip Compression with AWS Lambda and API Gateway HTTP API

In late 2017 we made the decision to go all in on Serverless. Our API's would move off of EC2 and traditional load-balancers and move to AWS Lambda and API Gateway. This led to some dramatic shifts in how we think about our architecture, and also many nights of frustration dealing with the shortcomings of these cutting edge services.

AWS has since filled many of the issues and gaps around Serverless architecture. Some of our favorites include - much faster startup times for Lambdas in a VPC, 15 minute execution times, and a shiny new version of API Gateway with a focus on cost and performance for HTTP API's.

API Gateway HTTP API's are a new product that provide much lower latency than the traidtional API Gateway REST API's. They also propvide a massive cost improvement making Lambda HTTP API's perfectly viable for production workloads. However, like many new AWS products, it's missing some key features of the prior version. HTTP compression in the form of gzip and brotli have become the standard way to reduce the number of bytes clients need to download to display a website, render json data, etc. Most CDN's offer gzip and brotli compression out of the box, and ours does as well, but with one major caveat - only for cacheable requests.

We choose Fastly over CloudFront for its incredible performance and feature set, as well as excellent pricing across the board. But one major limitation with Fastly is its inability to compress non-cacheable content. Before moving to HTTP API's, we would compress our content with API Gateway. This allowed our Lambdas to simply return a response and let API Gateway handle the rest. However, now that we've moved to HTTP API's, we needed a way to compress our responses in the Lambda runtime directly.

Lets start by creating a new file called compression.js and exporting a single function called compress:

const zlib = require('zlib')

exports.compress = (input, headers) => {
  ...
}

Our function needs to take in two parameters: the input string to compress, and the headers object which will allow us to look at the accept-encoding header to figure out which compression algorithm to use:

// Parse the acceptable encoding, if any
const acceptEncodingHeader = headers['accept-encoding'] || ''

// Build a set of acceptable encodings, there could be multiple
const acceptableEncodings = new Set(acceptEncodingHeader.toLowerCase().split(',').map(str => str.trim()))

Next we need to check certain encodings in priority order, and use that encoding if its present in our set. We will check brotli, gzip then deflate, in that order:

// Handle Brotli compression (Only supported in Node v10 and later)
if (acceptableEncodings.has('br') && typeof zlib.brotliCompressSync === 'function') {
  ...
}

// Handle Gzip compression
if (acceptableEncodings.has('gzip')) {
   ...
}

// Handle deflate compression
if (acceptableEncodings.has('deflate')) {
  ...
}

Finally, we can call the correct compression method on the zlib framework and return the compressed data along with the algoritm we used:

// Brotli
return {
  data: zlib.brotliCompressSync(input),
  contentEncoding: 'br'
}

// Gzip
return {
  data: zlib.gzipSync(input),
  contentEncoding: 'gzip'
}

// Deflate
return {
  data: zlib.deflateSync(input),
  contentEncoding: 'deflate'
}

// No Match
return {
  data: input,
  contentEncoding: null
}

Now that we can successfully compress any input based on the request headers, we can call our compression function right before we return our content in the lambda handler:

exports.handler = async (event, context) => {
  const res = await handleRequest(event, context)
  
  const { data, contentEncoding } = compression.compress(res.body, event.headers)
  
  return {
    statusCode: res.statusCode,
    body: data.toString('base64'),
    headers: {
      ...res.headers,
      'content-encoding': contentEncoding
    },
    isBase64Encoded: true
  }
}

Wrapping Up

AWS has already vouched to make the new HTTP API's feature complete with the old REST API's, but they made that promise nearly 2 years ago now and they have yet to add support for compression out of the box. Hopefully this tuturial helps you fill the gap in the meantime.