Packfiles: How Real Git Stores Millions of Objects

Loose objects are great for understanding but terrible for big repos. The Linux kernel has ~10 million objects — at one file per object that's 10 million inodes, 200+ GB of disk, and a git fetch that opens 100k TCP-sized files.

Real git uses packfiles in .git/objects/pack/:

.git/objects/pack/pack-9a8b7c6d....pack    # the data
.git/objects/pack/pack-9a8b7c6d....idx     # the index (sha -> offset)
.git/objects/pack/pack-9a8b7c6d....rev     # optional reverse index

Packfile structure

Header (12 bytes):

Body: that many objects, each:

<variable-length size + type header>
<zlib-compressed data>

Footer: 20-byte SHA-1 of all preceding bytes (integrity check).

Object types in pack

The 3-bit type field in the size header encodes:

Types 1-4 are "full" objects — decompress and you have the body. Types 6-7 are deltas: "I'm derived from another object plus these patches."

Delta compression — the win

Most blob versions are tiny modifications of a previous version. A delta says:

copy bytes 0..1234 from base
insert these 42 bytes
copy bytes 1276..end from base

If Foo.java changes by 5 lines, the delta is ~100 bytes vs storing the whole 50KB file. For source-heavy repos this is a 10-50x size reduction.

• REF_DELTA: identifies its base by 20-byte SHA-1
• OFS_DELTA: identifies its base by relative offset within the same pack (smaller header, used by git repack to keep packs self-contained)

git pack-objects does a heuristic delta chain search:

• Sort objects by type, then size, then path-similarity
• Try each as a delta against the previous N candidates
• Cap chain length (default pack.depth=50)
• Pick the smallest result per object

The index file (.idx)

The .idx lets you find an object's offset without scanning the pack. v2 layout:

header: \xff\x74\x4f\x63 + version=2
fanout[256]: cumulative count of SHAs with first byte <= i
sha1_list[N]: 20*N bytes, sorted
crc32_list[N]: 4*N bytes
offset_list[N]: 4*N bytes (high bit = "use 64-bit table")
[optional] offset_64[]: 8 bytes each
packfile_sha[20]: same as packfile's footer SHA
idx_sha[20]: SHA-1 of all preceding bytes in .idx

fanout lets you O(1) narrow to a slice of sha1_list, then binary-search. A repo with 1M objects can look up any SHA in ~20 comparisons.

When packs are created

• git gc — periodic maintenance; combines loose + old packs
• git push / git fetch — both ends send a single packfile
• git clone — server streams one big pack
• git repack -a -d — manual full repack

After packing, loose objects whose SHAs are in any pack become eligible for deletion (git prune runs by default after gc).

You won't implement a packfile parser in this course — but understanding that "loose objects = your homework; packs = real git" is essential.