I Let Two AI Agents Race to Modernize pigz

Ever since I started working on machine translation, our datasets have been kept compressed. Initially I just used gzip/gunzip/zcat directly or via tar. But the datasets kept growing, and gzip is single-threaded.

Then I found pigz — Parallel Implementation of GZip. It uses multithreading to parallelize compression. Over the past few years I’ve compressed GBs to TBs of datasets with pigz at work. What takes hours to days with gzip can be reduced to minutes on a beefy server with 40+ cores. pigz is great — easy to install, clean CLI, just pipe data in and out.

But then enter Python. I build data orchestration frameworks in Python (e.g., mtdata). Python’s stdlib gzip module is convenient but painfully slow at scale. There are many Python libs promising faster gzip, but figuring out which ones are actually compatible with pigz/gzip output is its own headache. My go-to has been subprocess pigz — just fork the process and stream I/O. It works, but there’s overhead from process forking and kernel-level copies of data through STDIN/STDOUT.

I learned how to bind Python APIs to C/C++ code while working on Pymarian. Naturally I thought — why not do the same for pigz? But pigz was written as a standalone CLI tool. It’s not thread-safe. It has one single global context (struct g, ~60 mutable fields) for all state, so having a pigz API call for reading and another for writing in the same process is asking for trouble. I looked at refactoring it into a library…​ and gave up. Too much work for my brain.

I keep an eye on coding agents. I refer to benchmarks like Terminal Bench and SWE-bench to assess which one is better, but whenever people claim agents will automate coding forever, I want to test it on a real task — something I actually need done.

Modernizing pigz became my go-to litmus test. The task is well-defined and easy to verify:

I asked agents to rewrite pigz in C++23, using modern language features — threads, RAII, exceptions — and restructure it as a thread-safe library.

2024 (GPT/Codex): Nowhere close. I tried, failed, wasted hours. Commits here.

2025: Agents could rewrite pigz, but performance was degraded. They couldn’t fix the regression — still too much work.

2026: A different story entirely.

This weekend I launched two parallel Copilot agent sessions in two VS Code workspaces — one with Claude Opus 4.6, and the other with GPT 5.4 — and let them loose on the same requirements document.

Both got the same goals: rewrite pigz in C++23 as a thread-safe library called pigzpp, with at least performance parity and full gzip compatibility.

I shared the requirements, let them plan, clarified their questions, and then let them implement.

I asked: "We’ve matched pigz. Now can you beat it?"

The agent identified two key optimizations:

Result: up to 1.8x faster compression, 2.4x faster decompression vs original pigz (at default thread count). At single-thread, the gap is even wider — 5.7x faster compression thanks to zlib-ng’s SIMD-optimized DEFLATE.

The final pigzpp:

I then had the agent build Python bindings via pybind11. Unlike my old approach of forking a subprocess, pigzpp calls the C++ library directly — zero fork/exec overhead.

I benchmarked against every relevant Python compression library. All speedups below are relative to stdlib gzip (1.0x baseline).

Libraries tested:

Benchmarks on a 48-core Xeon, Ubuntu 24.04.3 LTS, Python 3.14.3 (verified identical on 3.13.11):

Agents can modernize real tools — and make them faster. This isn’t a toy demo. pigz is a widely-used utility written by Mark Adler, co-creator of zlib. A coding agent rewrote it from scratch in 70 minutes, matched performance, then beat it by 2x with targeted optimizations. That’s remarkable.

The Python ecosystem has quietly gotten fast. Python 3.14 will bundle zlib-ng in stdlib for Windows/macOS (cpython#91349). ISA-L is even faster on Intel CPUs — its igzip CLI hits 5.3 GB/s decompression, and the Python isal library dominates single-threaded file compression. The case for pigzpp Python bindings is strongest for the bytes API where multithreading gives a 10x advantage — but for file-at-a-time use, isal is the better choice. The CLI pigzpp remains useful as a faster drop-in replacement for pigz.

Agent style matters. Claude’s autopilot approach — just doing the work without asking permission at every step — was noticeably more productive than GPT’s pause-and-announce style. For a task with clear requirements, I want an agent that executes, not one that narrates.

Well-defined tasks make agents shine. The key was having clear, verifiable success criteria: compatibility and performance. If I couldn’t objectively measure whether the output was correct and fast, I wouldn’t have trusted either agent’s result.

pigzpp is open source: github.com/thammegowda/pigzpp