Little Bit of x, y & z

Follow me on Mastodon Go to my GitHub repo

Dev log: Making a simple Brotli compressor module with Rust and WebAssembly (Wasm)

~7 min read

The other day I ran into a problem: I needed to compress some data using Brotli, but being in a browser-like environment with limited Node support (Cloudflare Workers) for example using Zlib from Node was not an option.

This a log of how I created that small compression module using Rust and WebAssembly.

The end result can be found on my GitHub here.

Contents

Looking into existing options

Before proceeding to create my own module, I tried out a couple of the existing modules. Namely the aptly named brotli-wasm and wasm-brotli.

However, in the Cloudflare Workers environment they unfortunately did not work out of the box. The environment gave errors when trying to instantiate the WebAssembly code.

Turns out Cloudflare Workers requires a WebAssembly.Module instead of a WebAssembly.Instance as detailed in the Workers Wasm Rust documentation.

It’s also possible that I just didn’t know how to instantiate it properly.

Post facto note

There is also a JavaScript C++ port as well as the official Brotli library, which includes a js implementation.

I did not try these out, because I wanted to try out using a WebAssembly nor am I sure if they would work with Cloudflare Workers to begin with.

Making a Brotli compression module with Rust

The Cloudflare Workers docs have some information on WebAssembly. Further, the JavaScript docs give an example of writing code with the WebAssembly Text Format and then using wat2wasm to create a wasm file to be loaded in a Worker.

However, since I am aiming to have Brotli compression, I opt to use Rust to compile to Wasm due to being able to utilise a brotli crate.

Setting up the project

MDN has really nice documentation on WebAssembly and how to use Rust to compile to WebAssembly, which I follow and adapt to my own needs.

I have Rust already installed, so I skip this and install wasm-pack.

cargo install wasm-pack

Next I set up a new project.

cargo new --lib wasm-brotli-with-rust
cd wasm-brotli-with-rust

The default file ./src/lib.rs can be deleted and replaced with our own code as detailed in the MDN docs.

The MDN tutorial gives an example of a two-directional module, where one can both call the wasm code from JavaScript as well as call a JavaScript function within the Wasm module itself. I find this pretty cool, but also a potential place to watch out in order to keep the Wasm module API clean and easy to understand.

In my case, I start by adding brotli and wasm-bindgen as a dependencies.

cargo add brotli wasm-bindgen

This adds the dependencies to Cargo.toml. I further edit the file to add the basic package information (e.g. author, license). I also add the lib field defining the crate-type as cdylib.

The crate type cdylib basically means that the purpose of this code is to produce a “dynamic system library” “to be loaded from another language”.

The final Cargo.toml is like this:

[package]
...author, license etc...

[lib]
crate-type = ["cdylib"]

[dependencies]
brotli = "3.4.0"
wasm-bindgen = "0.2"

Writing the Rust code

Now that we have the dependencies installed, I take the example code from the MDN tutorial and modify it to my purposes.

I use the CompressionWriter from the brotli crate, which takes stdout, buffer_size, quality and lgwin parameters:

// CompressorWriter
pub fn new(w: W, buffer_size: usize, q: u32, lgwin: u32) -> Self

The quality can be set between 0 and 11, where 11 produces the smallest file size, but takes the longest time.

I am not exactly sure what the lgwin parameter does, but the docs state that it’s a “log of how big the ring buffer should be for copying prior data”, so I guess it is some kind of window size for the incoming data buffer.

let mut writer = brotli::CompressorWriter::new(
    stdout,
    4096,
    11,
    22);

The incoming data will be an Uint8Array in JavaScript and I don’t know how large the output will be, so I use the Rust Vec to initialise the output.

The resulting code looks like this:

use std::io::Write;

use brotli;
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn compress(data: &[u8]) -> Vec<u8> {
    let mut output = Vec::new();
    {
        let quality: u32 = 9;
        let lg_window_size: u32 = 20;
        let mut writer = brotli::CompressorWriter::new(
            &mut output,
            4096, /* buffer size */
            quality,
            lg_window_size as u32,
        );
        writer.write(&data).unwrap();
    }
    output
}

Compiling to WebAssembly

Having now successfully written a simple Brotli compression library in Rust, it’s time to compile and try it out.

To compile to a web target, I run the following command.

wasm-pack build --target web

In the resulting pkg directory, we can find the output with a package.json that conveniently declares the module, types and files fields, to the packages is ready-to-consume as an ES module in TypeScript.

Adding a little html file to test in the browser

Following the MDN tutorial further, I create an index.html in the root of the repository with the following contents.

<!doctype html>
<html lang="en-US">
  <head>
    <meta charset="utf-8" />
    <title>Brotli compression with Rust to WASM example</title>
  </head>
  <body>
    <script type="module">
      import init, { compress } from "./pkg/wasm_brotli.js";
      init().then(() => {
        const textEncoder = new TextEncoder();
        const input = "Hello world";
        const uncompressedData = textEncoder.encode(input);
        const compressedData = compress(uncompressedData);
        console.log("compressedData:", compressedData);
      });
    </script>
  </body>
</html>

To serve this, I start a python server with python3 -m http.server.

Opening the browser at localhost:8000 and the developer tools’ console, I can find the output of the compressedData.

One can verify the output with another Brotli decoder (e.g. Zlib from Node).

Using the Wasm module in Cloudflare Workers

The previous html example doesn’t make much sense, but I try the module also in my Cloudflare Workers project.

To get it working in Cloudflare Workers, a few extra steps are required.

First, the compiling target needs to be a bundler.

This achieved simply enough by running the following command.

wasm-pack build --target bundler

Second, in the resulting pkg folder, the .js file serving as the entry point to the module, needs to be modified as follows (taken from the Cloudflare Workers’ WebAssembly docs) substituting names and locations as necessary.

import * as imports from "./mywasmlib_bg.js";

// switch between both syntax for node and for workerd
import wkmod from "./mywasmlib_bg.wasm";
import * as nodemod from "./mywasmlib_bg.wasm";
if (typeof process !== "undefined" && process.release.name === "node") {
  imports.__wbg_set_wasm(nodemod);
} else {
  const instance = new WebAssembly.Instance(wkmod, { "./mywasmlib_bg.js": imports });
  imports.__wbg_set_wasm(instance.exports);
}

export * from "./mywasmlib_bg.js";

Finally, within the Worker itself, the package can be imported like any other named export from an ES module.

Size optimisations

Like noted in the MDN tutorial, building Wasm like this produces relatively large files.

After getting the above code to work, I proceeded to optimise the bundle size following the suggestions in the WebAssembly Rust book and Cloudflare Workers Rust documentation.

The unoptimised .wasm file is as follows:

$ wc -c pkg/wasm_brotli_bg.wasm
1488210 pkg/wasm_brotli_bg.wasm

Adding the following wasm-opt options to Cargo.toml

[profile.release]
lto = true
strip = true
codegen-units = 1
opt-level = 'z'

and running the compiler again results in a size reduction of about 15 %:

$ wc -c pkg/wasm_brotli_bg.wasm
1266526 pkg/wasm_brotli_bg.wasm

This is further compressed as I am using gzip compression for the Cloudflare Worker.

Perhaps there’s more that can be done, but that will have to be an adventure for another time.

Finally

Getting my feet wet with WebAssembly for the first time was interesting and educational.

I hope this log of my experience is interesting and optimally useful for someone.

If you have thoughts, experiences or further information that you think I might be interested in, please don’t hesitate to reach out for example on Mastodon.

Thank you for reading!

References