| 1 | # 0008 — The log is a binary format |
| 2 | |
| 3 | **Status:** adopted |
| 4 | |
| 5 | ## Decision |
| 6 | |
| 7 | Both the index and entry headers are hand-rolled binary, not JSON. Big-endian, |
| 8 | magic-prefixed, versioned. `wal-codec.ts` owns encode and decode; `cursor.ts` |
| 9 | provides the read/write primitives. |
| 10 | |
| 11 | ## Why |
| 12 | |
| 13 | Replay reads the index on **every push and every materialization**. Object IDs |
| 14 | dominate its size, and JSON forces them to hex: 40 bytes where 20 will do, plus |
| 15 | quoting and key names. Binary halves the oid cost outright and removes parse |
| 16 | overhead from the hot path. |
| 17 | |
| 18 | The `Cursor`/`Reader` pair exists because the alternative is offset arithmetic |
| 19 | inline at every field, which reliably produces an off-by-one that corrupts a |
| 20 | log rather than throwing. |
| 21 | |
| 22 | ## Index layout |
| 23 | |
| 24 | ``` |
| 25 | magic u32 "GWAL" |
| 26 | version u8 |
| 27 | flags u8 |
| 28 | seq u64 |
| 29 | compactedThroughSeq u64 |
| 30 | refCount u32 |
| 31 | refName u16 length + utf8 |
| 32 | oid 20 raw bytes |
| 33 | layerCount u32 |
| 34 | ulid 16 raw bytes |
| 35 | packSha 32 raw bytes |
| 36 | size u64 |
| 37 | ``` |
| 38 | |
| 39 | ## Entry layout |
| 40 | |
| 41 | ``` |
| 42 | magic u32 "GENT" |
| 43 | version u8 |
| 44 | headerLen u32 ← the packfile begins at exactly this offset |
| 45 | ulid 16 raw bytes |
| 46 | createdAt u64 |
| 47 | pushedBy u16 length + utf8 |
| 48 | transitionCount u32 |
| 49 | refName u16 length + utf8 |
| 50 | oldOid 20 raw bytes |
| 51 | newOid 20 raw bytes |
| 52 | <packfile bytes> |
| 53 | ``` |
| 54 | |
| 55 | `headerLen` up front buys a real capability: entry metadata is readable with |
| 56 | `Range: bytes=0-65535` instead of downloading a half-gigabyte packfile. |
| 57 | `WalStoreService.readEntryHeader` does exactly that. |
| 58 | |
| 59 | ## Two fields that look like overhead and are not |
| 60 | |
| 61 | - **`version` |
| 62 | `WalCorruptError`. A log format that cannot be versioned cannot be evolved. |
| 63 | - **`flags` |
| 64 | format switch. Retrofitting a hash-length change without a flag byte means |
| 65 | rewriting every object in every repository. |
| 66 | |
| 67 | ## Consequence |
| 68 | |
| 69 | The format is now an on-disk contract. Any change to it needs a version bump and |
| 70 | a decoder that handles both. `wal-codec.spec.ts` round-trips every shape, |
| 71 | including the empty index, and asserts that oids are stored as raw bytes rather |
| 72 | than hex. |