Dev log: Making a simple Brotli compressor module with Rust and WebAssembly (Wasm)
The other day I ran into a problem: I needed to compress some data using Brotli,
but being in a browser-like environment with limited Node support (Cloudflare Workers)
for example using Zlib
from Node was not an option.
This a log of how I created that small compression module using Rust and WebAssembly.
The end result can be found on my GitHub here.
Contents
- Looking into existing options
- Making a Brotli compression module with Rust
- Size optimisations
- Finally
- References
Looking into existing options
Before proceeding to create my own module, I tried out a couple of the existing modules.
Namely the aptly named brotli-wasm
and wasm-brotli
.
However, in the Cloudflare Workers environment they unfortunately did not work out of the box. The environment gave errors when trying to instantiate the WebAssembly code.
Turns out Cloudflare Workers requires a WebAssembly.Module
instead of a WebAssembly.Instance
as detailed in the Workers Wasm Rust documentation.
It’s also possible that I just didn’t know how to instantiate it properly.
Post facto note
There is also a JavaScript C++ port as well as the official Brotli library, which includes a js
implementation.
I did not try these out, because I wanted to try out using a WebAssembly nor am I sure if they would work with Cloudflare Workers to begin with.
Making a Brotli compression module with Rust
The Cloudflare Workers docs have some information on WebAssembly. Further, the JavaScript docs give an example of writing code with the WebAssembly Text Format and then using wat2wasm
to create a wasm
file to be loaded in a Worker.
However, since I am aiming to have Brotli compression, I opt to use Rust to compile to Wasm due to being able to utilise a brotli crate.
Setting up the project
MDN has really nice documentation on WebAssembly and how to use Rust to compile to WebAssembly, which I follow and adapt to my own needs.
I have Rust already installed, so I skip this and install wasm-pack
.
cargo install wasm-pack
Next I set up a new project.
cargo new --lib wasm-brotli-with-rust
cd wasm-brotli-with-rust
The default file ./src/lib.rs
can be deleted and replaced with our own code as detailed in the MDN docs.
The MDN tutorial gives an example of a two-directional module, where one can both call the wasm
code from JavaScript as well as call a JavaScript function within the Wasm module itself. I find this pretty cool, but also a potential place to watch out in order to keep the Wasm module API clean and easy to understand.
In my case, I start by adding brotli
and wasm-bindgen
as a dependencies.
cargo add brotli wasm-bindgen
This adds the dependencies to Cargo.toml
. I further edit the file to add the basic package information (e.g. author, license).
I also add the lib
field defining the crate-type
as cdylib
.
The crate type cdylib
basically means that the purpose of this code is to produce a “dynamic system library” “to be loaded from another language”.
The final Cargo.toml
is like this:
[package]
...author, license etc...
[lib]
crate-type = ["cdylib"]
[dependencies]
brotli = "3.4.0"
wasm-bindgen = "0.2"
Writing the Rust code
Now that we have the dependencies installed, I take the example code from the MDN tutorial and modify it to my purposes.
I use the CompressionWriter
from the brotli
crate, which takes stdout
, buffer_size
, quality
and lgwin
parameters:
// CompressorWriter
pub fn new(w: W, buffer_size: usize, q: u32, lgwin: u32) -> Self
The quality can be set between 0 and 11, where 11 produces the smallest file size, but takes the longest time.
I am not exactly sure what the lgwin
parameter does, but the docs state that it’s a “log of how big the ring buffer should be for copying prior data”, so I guess it is some kind of window size for the incoming data buffer.
let mut writer = brotli::CompressorWriter::new(
stdout,
4096,
11,
22);
The incoming data will be an Uint8Array
in JavaScript and I don’t know how large the output will be, so I use the Rust Vec
to initialise the output.
The resulting code looks like this:
use std::io::Write;
use brotli;
use wasm_bindgen::prelude::*;
#[wasm_bindgen]
pub fn compress(data: &[u8]) -> Vec<u8> {
let mut output = Vec::new();
{
let quality: u32 = 9;
let lg_window_size: u32 = 20;
let mut writer = brotli::CompressorWriter::new(
&mut output,
4096, /* buffer size */
quality,
lg_window_size as u32,
);
writer.write(&data).unwrap();
}
output
}
Compiling to WebAssembly
Having now successfully written a simple Brotli compression library in Rust, it’s time to compile and try it out.
To compile to a web target, I run the following command.
wasm-pack build --target web
In the resulting pkg
directory, we can find the output with a package.json
that conveniently declares the module
, types
and files
fields, to the packages is ready-to-consume as an ES module in TypeScript.
Adding a little html file to test in the browser
Following the MDN tutorial further, I create an index.html
in the root of the repository with the following contents.
<!doctype html>
<html lang="en-US">
<head>
<meta charset="utf-8" />
<title>Brotli compression with Rust to WASM example</title>
</head>
<body>
<script type="module">
import init, { compress } from "./pkg/wasm_brotli.js";
init().then(() => {
const textEncoder = new TextEncoder();
const input = "Hello world";
const uncompressedData = textEncoder.encode(input);
const compressedData = compress(uncompressedData);
console.log("compressedData:", compressedData);
});
</script>
</body>
</html>
To serve this, I start a python server with python3 -m http.server
.
Opening the browser at localhost:8000
and the developer tools’ console, I can find the output of the compressedData
.
One can verify the output with another Brotli decoder (e.g. Zlib from Node).
Using the Wasm module in Cloudflare Workers
The previous html example doesn’t make much sense, but I try the module also in my Cloudflare Workers project.
To get it working in Cloudflare Workers, a few extra steps are required.
First, the compiling target needs to be a bundler.
This achieved simply enough by running the following command.
wasm-pack build --target bundler
Second, in the resulting pkg
folder, the .js
file serving as the entry point to the module, needs to be modified as follows (taken from the Cloudflare Workers’ WebAssembly docs) substituting names and locations as necessary.
import * as imports from "./mywasmlib_bg.js";
// switch between both syntax for node and for workerd
import wkmod from "./mywasmlib_bg.wasm";
import * as nodemod from "./mywasmlib_bg.wasm";
if (typeof process !== "undefined" && process.release.name === "node") {
imports.__wbg_set_wasm(nodemod);
} else {
const instance = new WebAssembly.Instance(wkmod, { "./mywasmlib_bg.js": imports });
imports.__wbg_set_wasm(instance.exports);
}
export * from "./mywasmlib_bg.js";
Finally, within the Worker itself, the package can be imported like any other named export from an ES module.
Size optimisations
Like noted in the MDN tutorial, building Wasm like this produces relatively large files.
After getting the above code to work, I proceeded to optimise the bundle size following the suggestions in the WebAssembly Rust book and Cloudflare Workers Rust documentation.
The unoptimised .wasm
file is as follows:
$ wc -c pkg/wasm_brotli_bg.wasm
1488210 pkg/wasm_brotli_bg.wasm
Adding the following wasm-opt
options to Cargo.toml
[profile.release]
lto = true
strip = true
codegen-units = 1
opt-level = 'z'
and running the compiler again results in a size reduction of about 15 %:
$ wc -c pkg/wasm_brotli_bg.wasm
1266526 pkg/wasm_brotli_bg.wasm
This is further compressed as I am using gzip compression for the Cloudflare Worker.
Perhaps there’s more that can be done, but that will have to be an adventure for another time.
Finally
Getting my feet wet with WebAssembly for the first time was interesting and educational.
I hope this log of my experience is interesting and optimally useful for someone.
If you have thoughts, experiences or further information that you think I might be interested in, please don’t hesitate to reach out for example on Mastodon.
Thank you for reading!