Making Compression Faster

The idea that carried the whole project was to keep the orchestration in one library, then expose it through thin bindings so every language gets the same implementation and behavior.

pigzpp does not implement the low-level DEFLATE machinery by itself. Under the hood, it can select between two excellent open-source compression engines:

You choose with a single engine argument (auto, zlib, or isal) from the CLI or API. The WebAssembly build uses zlib-ng because ISA-L is x86-specific. pigzpp’s contribution is the thread-safe parallel pipeline around those engines: dividing input into blocks, scheduling work, combining checksums, producing compatible gzip/zlib/ZIP/PNG framing, and exposing the result through C++, Python, Go, Rust, and WebAssembly APIs. The speed belongs to the complete stack — pigzpp’s parallel orchestration plus the substantial optimization work already done by the zlib-ng and ISA-L contributors.

Everything below uses that same stack, benchmarked on a modest 5-core / 10-thread workstation (Intel Xeon W-2235) — deliberately not a 48-core server, to show these gains land on everyday hardware too.

Unless a section says otherwise, the numbers are best-of-three runs on a 128 MB English+Chinese Wikipedia corpus, level 6, with 8 threads. Throughput is measured over uncompressed input; ratio means input size divided by output size, so higher is smaller. These are results from one machine and corpus, not universal constants.

Here’s the whole story in one table; the rest of the post is the "why" behind each row.

Let me walk through each, because each one is a place where systems work and AI work cross paths.

The original motivation was boring and universal: I compress a lot of data, and gzip is single-threaded. pigz fixed that years ago, and pigzpp goes further by swapping in modern DEFLATE backends.

On a 128 MB multilingual-text corpus at level 6:

At a comparable ratio to pigz (2.81 vs 2.83), pigzpp’s zlib-ng backend is 2.4× faster; the ISA-L backend is 7× faster than pigz and 45× faster than plain gzip, trading ~9% ratio for the speed. It’s still a byte-compatible, drop-in gzip/pigz replacement, so nothing downstream has to change.

Almost all of my data-orchestration and augmentation code is Python — tools like mtdata (dataset acquisition) and data-augmentation pipelines like sotastream. These stream and re-compress huge corpora, and Python’s stdlib gzip is famously slow at that scale.

pigzpp’s pybind11 bindings call the C++ core in-process (no subprocess pigz fork tricks), and fire up all cores for large in-memory buffers:

That is up to 80× faster than stdlib gzip on this bytes API benchmark, fully interoperable (pigzpp.compress() ↔ gzip.decompress()). The exact gain depends heavily on input size and compressibility, but this is the path that matters in data-preparation jobs that repeatedly transform large in-memory batches.

Not every image is a compression job for pigzpp — plenty of datasets are JPEG, which is its own lossy codec and has nothing to do with gzip. But a good slice of multimodal work leans on PNG: segmentation masks, screenshots and UI captures, charts and diagrams, synthetic renders, and anywhere you specifically want exact pixels. And a PNG is, at its heart, a DEFLATE stream wrapped in some framing — the exact thing pigzpp is good at.

So pigzpp ships a small PNG encoder/decoder on the same core. Benchmarked on the 24-image Kodak set:

pigzpp encodes PNGs 8.6× faster than Pillow and is about 1.2× faster than OpenCV on this RGB set, at a comparable file size. OpenCV is already one of the fastest mainstream C++ image libraries available from Python, so that was the comparison I cared about most. When you’re writing millions of augmented image tiles or caching rendered eval samples, that adds up fast.

This one is pure infrastructure pain. Our model training runs in Docker, and anyone who builds images knows the loop: change one line, rebuild, wait. A chunk of that wait is compression — Docker’s default builder, BuildKit, uses Go’s single-threaded compress/gzip on its default gzip layer path.

Could pigzpp help? It needed Go bindings, so I added them (cgo over a small C ABI). Then I measured two things on the same 512 MB layer: a real docker buildx build with gzip and uncompressed exports, to estimate BuildKit’s compression cost; and pigzpp compressing the identical layer bytes outside BuildKit.

The codec comparison suggests ~12× less time in layer compression at a comparable ratio, or ~50× with ISA-L. This is not a claim that stock docker build is already 12× faster: BuildKit has no runtime hook to swap its gzip codec, so realizing that gain inside a build needs a small BuildKit patch (and the total build speedup depends on how much time the build spends compressing). Because pigzpp emits standard gzip, the resulting layers remain compatible with registries and docker pull. On a real python:3.12 image layer (637 MB), pigzpp’s parallel zlib-ng backend beats even klauspost/pgzip, the best pure-Go parallel gzip, at a better ratio.

This one I’ll admit was mostly for fun. Lots of systems people are migrating from C++ to Rust, and I wanted to know: has the Rust ecosystem produced a compression library that’s actually faster than what we have? So I added Rust FFI bindings and benchmarked pigzpp against the popular crates (128 MB text, 8 threads):

This is not really a verdict on C++ versus Rust. In this benchmark, the DEFLATE backend and parallelization strategy mattered more than the host language: pigzpp’s zlib-ng path was about 1.3× faster than gzp at a comparable ratio, while the speed-first ISA-L path was farther ahead at a lower ratio. The Rust binding simply lets Rust code use that same engine.

A lot of the documents we scrape for training data — .docx, .pptx, .xlsx, and .epub — are ZIP containers, often full of XML, HTML, and media assets. The same container format also appears in software packages such as .jar, .apk, and Python .whl files. "Open a document" is really "unzip an archive and read its members." ZIP is quietly one of the most useful container formats there is.

So pigzpp grew a native, multi-entry ZIP implementation with a Python API modeled on a useful subset of the standard library’s zipfile, with parallel per-member compression:

That is 13× faster at a comparable ratio and up to 31× on the speed-first path. The archives interoperate with zipfile, unzip, and other standard ZIP readers, with Zip64 for large files.

Finally, the newest corner. When I build a browser demo for a model — an in-page tokenizer, a client-side data viewer, an offline-capable app — I keep needing to compress or decompress right there in the browser. The usual options are the native CompressionStream or pure-JS libraries like fflate and JSZip.

pigzpp compiles to WebAssembly (zlib-ng + 128-bit SIMD). For repeatability, I ran the numbers below under Node 22 — a WASM runtime that also exposes the browser-style CompressionStream API — rather than claiming one number for every browser:

The threaded WASM build (Web Workers + SharedArrayBuffer) scales further — ~4.7× at 8 threads in Node; browsers require cross-origin isolation to enable that path. On the document use case above — reading a real 6 MB .docx (19 MB of XML) in the same Node benchmark — pigzpp-wasm unzips at 253 MB/s vs 111 for fflate and 87 for JSZip. Browser performance will vary by engine, but the module and API are the same ones a web application loads.

I like large speedup numbers, but I trust them more when the boundaries are visible:

pigzpp stands on several mature projects. zlib, created by Jean-loup Gailly and Mark Adler, defined the API and format ecosystem that much of this software still speaks. Mark Adler’s pigz demonstrated the parallel block-compression design and was the reference for compatibility and behavior. The zlib-ng contributors built and maintain the SIMD-accelerated zlib implementation used by pigzpp’s higher-ratio path. The Intel ISA-L contributors built the hand-optimized routines behind the speed-first path. The benchmark numbers in this post would not exist without their years of low-level optimization work.

What I built around them is the integration: a modern thread-safe C++ pipeline, backend selection, language bindings, ZIP and PNG APIs, interoperability tests, and benchmarks that make the trade-offs visible.

This expansion was also a continuation of the experiment from my previous post: how far can I take a real systems project by pair programming with AI agents? I used the agent for codebase exploration, implementation, bindings, tests, benchmark harnesses, and repeated cleanup. My role was not simply to ask for "make it faster" and accept the output. I chose the use cases, made the scope and architecture decisions, challenged suspicious benchmark results, supplied realistic datasets, caught incorrect assumptions (including an ISA-L level-mapping bug), insisted on interoperability tests, and reviewed the code before merging it.

That division of work felt much closer to pair programming than autonomous software generation: I provided direction and judgment; the agent provided unusually fast implementation and iteration. Each feature had to survive tests and reproducible benchmarks before I trusted it.

The result, on top of the original agent-built rewrite:

I find it genuinely remarkable that a widely-used C utility, first rewritten by an agent in ~70 minutes, has grown into a small multi-language compression stack — and that the same "clear requirements, inspect the result, verify with tests and benchmarks" recipe kept working at each step. (If you enjoy the "rewrite it in a fast language and measure" genre, I did the same thing to BPE training in From O(N) to O(log N).)

pigzpp is open source: github.com/thammegowda/pigzpp. If it saves you a few hours of waiting on compression, that’s a good day.