Inside git packfiles: deltas, the index, and the .pack format

In the previous post we saw that git stores every blob, tree, commit, and tag as a loose object — one zlib-compressed file each. That’s fine for a small repo, but a real project can have millions of objects, and a filesystem full of millions of tiny files is slow to traverse, wasteful on disk, and miserable to ship over the network.

The answer is the packfile: a single file that holds many objects, deduplicated and delta-compressed, paired with an index that makes any object findable without scanning.

A pack is two files that travel together:

.git/objects/pack/pack-<hash>.pack   # the objects
.git/objects/pack/pack-<hash>.idx    # the lookup index

The .pack format

The .pack file is a simple framed binary format:

1. A 12-byte header: the signature PACK, a 4-byte version, and a 4-byte count of objects.
2. The objects themselves, back to back, each with a variable-length header encoding its type and uncompressed size, followed by zlib-compressed data.
3. A 20-byte trailing checksum (SHA-1 of everything above).

Inside a pack, objects come in two flavors.

Whole objects

Stored more or less as-is: a type (OBJ_BLOB, OBJ_TREE, OBJ_COMMIT, OBJ_TAG) and the deflated content. Same data as a loose object, just without the per-file overhead.

Deltas

This is where packs earn their keep. Instead of storing a full copy, an object can be expressed as a delta against a similar base object — a list of “copy these bytes from the base” and “insert these new bytes” instructions. A new version of a large file might cost only a few hundred bytes.

There are two delta encodings:

• OBJ_REF_DELTA — the base is named by its 20-byte object id. Used for “thin packs” where the base may live outside this pack (handy on the wire).
• OBJ_OFS_DELTA — the base is named by a relative byte offset earlier in the same pack. More compact, and the default on disk.

Deltas can chain: object C is a delta against B, which is a delta against A. To reconstruct C, git resolves the chain back to a whole base object and applies each delta in turn. The pack.depth setting caps how long these chains get, trading compression for reconstruction cost.

The .idx index

You can’t binary-search the .pack directly — objects are variable-length and stored in packing order, not sorted by id. The .idx fixes that. Version 2 of the index contains:

• a fan-out table of 256 entries: fanout[b] is the number of objects whose id’s first byte is <= b, which narrows any lookup to a small range instantly;
• the sorted list of all object ids;
• a CRC-32 per object (to detect corruption during repack/transfer);
• the offset of each object within the .pack.

A lookup is then: read the first byte of the id → jump into the fan-out → binary-search the sorted ids → read the offset → seek into the pack. O(log n), no scanning.

Why this matters for object storage

Packfiles already do the hard part of turning “millions of immutable objects” into “a few large, self-describing, content-addressed blobs with a side index.” That’s almost exactly the shape you want on top of S3:

• the .pack is a large immutable object — cheap to store, cheap to range-read;
• the .idx lets you fetch a single object with one ranged GET instead of downloading the whole pack;
• because everything is content-addressed and immutable, packs are trivially cacheable and never need invalidation.

This is the core idea behind mgit: lean on git’s own packfile format so that a repository becomes a handful of objects in a bucket, and serving a clone is mostly a matter of streaming bytes you’ve already packed.