Skip to content

Allocate the hash for an empty block, as C does - #218

Open
kunpai wants to merge 1 commit into
google:masterfrom
kunpai:hash-for-empty-block
Open

kunpai wants to merge 1 commit into
google:masterfrom
kunpai:hash-for-empty-block

Conversation

@kunpai

@kunpai kunpai commented Oct 28, 2026

Copy link
Copy Markdown

Allocate the hash for an empty block, as C does

Problem

C allocates the LZ77 hash once per block, before it looks at the block. ZopfliLZ77Optimal (C), ZopfliLZ77OptimalFixed (C) and ZopfliBlockSplit (C) each call ZopfliAllocHash (fixed, block split), which allocates seven arrays, about 960 KB, and only then does GetBestLengths return for an empty block.

The port makes its hash inside lz77_greedy (Rust) and get_best_lengths (Rust), after they return for an empty block (get_best_lengths). So compressing an empty input makes no hash in the port. The output is the same; the peak heap is not. For example, with an empty input (peak heap in bytes):

C Rust
ZopfliCompress, gzip format 983,086 18,561
ZopfliZlibCompress 983,074 18,553
ZopfliDeflate, btype = 1 983,087 153
ZopfliDeflatePart, btype = 1, instart = inend 983,087 153

Question

Which behaviour do we want: C's or the Rust port's? Specifically, for an empty block, should the hash be allocated anyway (like the C source), or not at all (like the Rust port)?

This PR implements C's behaviour.

Fix

lz77_greedy and get_best_lengths make their hash before the empty-block return (lz77_greedy, get_best_lengths). The two lines move; nothing else changes:

+    let window_size = crate::util::WINDOW_SIZE;
+    let mut hash = crate::hash::ZopfliHash::new(window_size);
     if instart == inend {
         return;
     }
 
-    let window_size = crate::util::WINDOW_SIZE;
-    let mut hash = crate::hash::ZopfliHash::new(window_size);

The examples above then hold what C holds. Replayed before and after (x86-64), peak heap in bytes:

C Rust main Rust patched
ZopfliCompress, gzip format 983,086 18,561 983,075
ZopfliZlibCompress 983,074 18,553 983,067
ZopfliDeflate, btype = 1 983,087 153 983,067
ZopfliDeflatePart, btype = 1, instart = inend 983,087 153 983,067

Downstream check. The output does not change: ZopfliGzipCompress on the repository's README gives the same 831-byte stream from the port before and after this patch, at the same peak heap, and gzip.decompress restores the file.

Notes for review:

  • This only costs memory: an empty block now holds about 960 KB for the length of the call, as in C. A non-empty block makes the same hash as before, just a few lines earlier.
  • C also makes a hash in ZopfliBlockSplit for the empty block, freed before ZopfliLZ77Optimal makes its own. The port's split returns before lz77_greedy for an empty range, so with this PR it makes one hash where C makes two in turn. The peak is the same, since C holds only one at a time.
  • The campaigns flagged these as memory.peak divergences.
  • The port's tests pass unchanged.

C's ZopfliBlockSplit, ZopfliLZ77Optimal and ZopfliLZ77OptimalFixed
allocate the hash (about 960 KB) before they look at the block, so an
empty block holds it too. lz77_greedy and get_best_lengths returned for
an empty block before allocating theirs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant