fast-pako

Why unzipping was the slow part

npm tarballs are gzip files, so anything that installs, inspects or serves packages has to unpack them, and any app that handles .gz downloads or writes archives does the same work in one direction or the other. Nodepod, which installs npm packages inside a browser tab, does it for every dependency. pako is a widely used JavaScript implementation of zlib, and gzip decoding is where most of a big install's decompression time goes: a 4.4 MB tarball takes pako 100 ms and fast-pako 17 ms.

Inflate is a tight loop. It refills a bit buffer, looks up a Huffman code in a table, and either writes a byte or copies a run from earlier output. A JavaScript engine runs that loop well, but it has to guard its type guesses, and pako keeps its bit buffer in a JavaScript number, which follows JavaScript's rules for integers. WebAssembly is typed ahead of time, so there are no guesses to guard, and it gives direct access to 64-bit integers and SIMD.

What makes the inflate loop fast

The state machine around the loop is pako's, case for case, so everything you can observe from outside stays the same. The loop itself is new, in the style of libdeflate: one decode table per code, a 64-bit bit buffer that refills without branching, the next table entry loaded ahead of time, and matches copied 16 bytes at a time.

The two checksums that come with a stream get the same care. adler32 uses SIMD, adding up 16 bytes per step. crc32 has a table-free path for big buffers: the CRC-32 polynomial divides a sparse polynomial with only five terms, so a long message can be reduced with plain XORs of 64-bit words instead of table lookups. Small buffers use slice-by-8 tables. That is where the 4x on plain inflate comes from, and why tarballs, which pay for the checksum too, gain a little more.

Why tiny inputs gain even more

Compressing 75 bytes is 20 times faster, and that has little to do with the engine. pako allocates its window and hash tables anew for every call, and for a few hundred bytes that allocation is most of the time. fast-pako pools its streaming sessions and builds its larger tables the first time they are needed.

Why compression stops near 2x

Level 1 is the fast, shallow setting, and there the speedup is about 3x. At levels 6 to 9, zlib searches harder for matches, and the speedup settles around 2x. The reason is that fast-pako's output must match pako's bytes exactly. zlib decides what to emit by walking the same candidate matches in the same order, so a faster search that looks at fewer candidates, or looks at them in another order, would produce a different, if valid, file. Code that hashes compressed output, or compares it to a fixture, would notice. So the match finder is a faithful port and the remaining speed comes from everything around it. At that point it is running at about the speed of native zlib.

Decompression has no such ceiling, because there is only one correct output, so the decoder is free to be as clever as it likes.

What is different

Where pako never returns, or hands back corrupt data, fast-pako throws an error that starts with fast-pako:. Examples are a chunkSize of 0, which makes pako loop forever, or a window smaller than 512 bytes, which makes pako write garbage. Each case is pinned by a test.

It also needs WebAssembly. Without it the first call throws a clear error. And pako's pako/lib/* and pako/dist/* subpaths are not provided, only the package's main entry.

On size, the picture is mixed and worth knowing. The npm tarball is 43.5 KB against pako's 412.5 KB, because nothing of pako is bundled. A browser bundle is 45.6 KB minified against 47.3 KB, but compresses less well: 30.6 KB gzipped against 15.1 KB, because compiled wasm compresses worse than JavaScript. Making it smaller cost more speed than the project's rules allow, and the attempts are written down.