phantomknight287/Portfolio

public

fix the git packet layout breaking responsivenesscbae6a8a month ago666 lines · 39.5 KB

1	---
2	title: "Building a Git Hosting Platform"
3	summary: "Things I learnt about git hosting platforms while trying to build one"
4	date: 2026-09-09
5	---
6
7	I've been building [Ghost](https://github.com/phantomknight287/ghost) and this blog post is on the learnings and design decision behind it.
8
9	TLDR;
10	<img src="/writings/its-all-git.jpg"
11
12
13	A git platform, behind all its HTTP controllers and other GUI, is just git invocations. But Before we move on to working of git hosting providers, we need to learn how git actually functions. Like not its internals, but its protocols.
14
15	So, git can push or pull data using 2 ways:
16	- SSH
17	- HTTPS
18
19	I will only be covering HTTPS in this article as I am yet to work on SSH based access.
20
21	HTTPs itself has 2 protocols:
22	- Git Smart Protocol
23	- Git dumb Protocol
24
25
26	The Platform that I've built only supports git smart protocol because I wasn't really able to find some good documentation on dumb protocol. However you can read the reference [here.](https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols)
27
28
29	### Some words you will keep seeing
30
31	Before the routes, here are the words git throws at you. I knew some of these vaguely and had to actually look them up while building this.
32
33	Object - anything git stores. A commit, a tree (a directory), a blob (a file's contents), or a tag. Every object is stored under the sha1 of its own contents, so the same file content in two repos has the same id.
34
35	oid - object id. The sha of an object, the thing you see as `a3f9c1...`. 40 characters when written as hex, 20 bytes when stored raw. All zeroes is a special oid meaning "nothing here".
36
37	ref - a name pointing at an oid. `refs/heads/main` is a branch, `refs/tags/v1
38
39	HEAD - the ref that says which branch is the current one.
40
41	Packfile - a bunch of objects zipped into one file, with some of them stored as deltas against others. This is what actually goes over the network. It starts with the ascii bytes `PACK`.
42
43	Thin pack - a packfile with deltas whose base objects are missing on purpose, because the sender knows the receiver already has them. Great for bandwidth, annoying for me later.
44
45	pkt-line - git's way of splitting a stream into messages. 4 hex characters saying how long the message is, then the message. The length counts those 4 characters too.
46
47	Flush packet - the pkt-line `0000`. It means "that section is done".
48
49	Sideband - a way of multiplexing progress messages and errors into the same stream as the pack data, so git can print "Counting objects" while sending you bytes.
50
51	<style dangerouslySetInnerHTML={{
52	.gh-fig{border:1px solid rgba(255,255,255,.12);border-radius:.6rem;background:rgba(255,255,255,.02);padding:1.1rem 1rem .9rem;margin:2rem 0;font-family:ui-monospace,SFMono-Regular,Menlo,monospace;font-size:.78rem;line-height:1.35;overflow-x:auto}
53	.gh-fig *{box-sizing:border-box}
54	.gh-cap{margin-top:.85rem;font-size:.72rem;color:rgba(255,255,255,.45);font-family:inherit}
55	.gh-node{border:1px solid rgba(255,255,255,.18);border-radius:.4rem;padding:.45rem .6rem;background:rgba(255,255,255,.04);white-space:nowrap}
56	.gh-flow{display:flex;align-items:stretch;gap:1.4rem;min-width:720px}
57	.gh-flow>.gh-node{align-self:center}
58	.gh-wires{flex:1;display:flex;flex-direction:column;gap:1.6rem;padding:.4rem 0}
59	.gh-wire{display:block}
60	.gh-wire .gh-lbl{display:block;color:rgba(255,255,255,.62);white-space:nowrap;margin-bottom:.4rem}
61	.gh-wire .gh-sub{display:block;color:rgba(255,255,255,.32);white-space:nowrap;margin-top:.45rem}
62	.gh-line{position:relative;height:1px;background:rgba(255,255,255,.16)}
63	.gh-dot{position:absolute;top:-3px;width:7px;height:7px;border-radius:50%;background:#7dd3fc;box-shadow:0 0 8px #7dd3fc;animation:gh-ltr 2.6s linear infinite}
64	.gh-dot.back{background:#fbbf24;box-shadow:0 0 8px #fbbf24;animation-name:gh-rtl}
65	@keyframes gh-ltr{from{left:0;opacity:0}10%{opacity:1}90%{opacity:1}to{left:100%;opacity:0}}
66	@keyframes gh-rtl{from{left:100%;opacity:0}10%{opacity:1}90%{opacity:1}to{left:0;opacity:0}}
67	.gh-pkt{display:flex;overflow-x:auto;align-items:stretch;gap:.35rem;padding:.3rem 0}
68	.gh-seg{border:1px solid rgba(255,255,255,.18);border-radius:.3rem;padding:.6rem .8rem;text-align:center}
69	.gh-pkt .gh-seg{flex:0 0 auto;white-space:nowrap}
70	.gh-seg small{display:block;margin-top:.3rem;font-size:.66rem;color:rgba(255,255,255,.4)}
71	.gh-seg.len{background:rgba(125,211,252,.12);border-color:rgba(125,211,252,.45);color:#7dd3fc}
72	.gh-seg.flush{background:rgba(251,191,36,.1);border-color:rgba(251,191,36,.4);color:#fbbf24}
73	.gh-steps{display:flex;flex-direction:column;gap:.4rem;min-width:520px}
74	.gh-step{display:flex;gap:.7rem;align-items:center;padding:.45rem .6rem;border:1px solid rgba(255,255,255,.1);border-radius:.35rem;animation:gh-sweep 6s ease-in-out infinite}
75	.gh-step .n{color:rgba(255,255,255,.35);width:1.1rem}
76	.gh-step.commit{border-color:rgba(52,211,153,.5);background:rgba(52,211,153,.07);color:#6ee7b7}
77	@keyframes gh-sweep{0%,100%{background:rgba(255,255,255,.01)}8%,16%{background:rgba(255,255,255,.07)}}
78	.gh-cols{display:flex;gap:1.6rem;min-width:640px;position:relative;align-items:flex-start}
79	.gh-col{flex:1;border:1px solid rgba(255,255,255,.14);border-radius:.45rem;padding:.6rem}
80	.gh-col h4{margin:0 0 .5rem;font-size:.72rem;color:rgba(255,255,255,.55);font-weight:400;font-family:inherit}
81	.gh-layer{padding:.25rem .45rem;border:1px solid rgba(255,255,255,.12);border-radius:.3rem;margin-bottom:.3rem;background:rgba(255,255,255,.03)}
82	.gh-layer.new{border-color:rgba(125,211,252,.45);color:#7dd3fc}
83	.gh-borrow{align-self:center;text-align:center;color:rgba(255,255,255,.45);padding:0 .4rem;white-space:nowrap}
84	.gh-borrow .arrow{display:block;height:1px;width:5rem;margin:.35rem auto;background:repeating-linear-gradient(90deg,rgba(125,211,252,.8) 0 6px,transparent 6px 12px);animation:gh-march .8s linear infinite}
85	@keyframes gh-march{to{background-position:12px 0}}
86	.gh-bar{display:flex;gap:.35rem;min-width:520px;margin-bottom:.5rem}
87	.gh-cell{flex:1;text-align:center;padding:.45rem 0;border:1px solid rgba(255,255,255,.14);border-radius:.3rem;background:rgba(255,255,255,.03)}
88	.gh-cell.missing{border-style:dashed;color:rgba(255,255,255,.3)}
89	.gh-cell.filling{animation:gh-fill 3.2s ease-in-out infinite}
90	.gh-cell.d1{animation-delay:.5s}.gh-cell.d2{animation-delay:1s}.gh-cell.d3{animation-delay:1.5s}
91	@keyframes gh-fill{0%,20%{border-style:dashed;color:rgba(255,255,255,.3);background:rgba(255,255,255,.02)}45%,100%{border-style:solid;color:#7dd3fc;border-color:rgba(125,211,252,.45);background:rgba(125,211,252,.08)}}
92	@media (prefers-reduced-motion:reduce){.gh-fig *{animation:none !important}}
93	` }} />
94
95	### Working of Smart protocol
96
97	Smart protocol depends on 3 HTTP routes.
98	- `/info/refs?service=`
99	- `/git-upload-pack`
100	- `/git-receive-pack`
101
102
103
104	### `/info/refs?service=
105	Git sends a get request to this route to check if refs in local store of git is same as what is on the server. If they are same, git shows `Already upto date` message.
106
107	The service is the fun part in this. It could be either
108	- `git-upload-pack`, the client wants to take stuff from you (clone, fetch, pull)
109	- `git-receive-pack`, the client wants to give you stuff (push)
110
111	Only those two. If you get anything else in that query param, someone is poking at your server, so throw an error there.
112
113	The response is not JSON. Git talks in pkt-lines, which I explained above, so the body looks like this:
114
115	```
116	001e# service=git-upload-pack\n
117	0000
118	<whatever git prints>
119	```
120
121	`001e` is hex for 30, and the line is 30 bytes counting those 4 characters. Then a flush packet to close that bit off.
122
123
124	<div className="gh-pkt text-sm
125	<div className="gh-seg len
126	<div className="gh-seg
127	<div className="gh-seg flush
128	</div>
129	<div className="gh-cap
130
131	The `<whatever git prints>` part is the ref advertisement, which is the list of every ref you have and the oid it points at. I did not write that list. Git writes it:
132
133	```ts
134	output.write(packetLine(
135	output.write(FLUSH_PACKET)
136
137	const child = spawn
138	// ^binary here could either be upload-pack or receive-pack
139	child.stdout.pipe(
140	```
141
142	`--advertise-refs` makes git print the list and quit instead of waiting around for the client to say something back.
143
144	One thing to watch: the content type has to be `application/x-git-upload-pack-advertisement`. If it's wrong, git assumes you are a dumb server and starts requesting files off you that you don't have, and the error you get back tells you nothing useful.
145
146	### `/git-upload-pack
147
148	This is a clone or a fetch. The client sends a body full of `want <oid>` lines for stuff it wants and `have <oid>` lines for stuff it already has, git works out the difference and sends back a packfile.
149
150	I implemented none of that. Spawn `git upload-pack --stateless-rpc <repo>`, pipe the request body into stdin, pipe stdout back into the response:
151
152	```ts
153	this.packProcess.streamUploadPack(
154	```
155
156	`--stateless-rpc` is important here. Normally `upload-pack` expects a live socket where it can go back and forth with the client a few times. Over HTTP each request is one shot, so that flag makes git do a single round and exit.
157
158	### `/git-receive-pack
159
160	This is a push, and it's the only route where I actually read the body instead of forwarding it, because this is where I find out what the client wants to change.
161
162	The body:
163
164	```
165	<pkt-line> "<old-oid> <new-oid> <ref>\0<capabilities>"
166	<pkt-line> "<old-oid> <new-oid> <ref>"
167	0000
168	PACK....
169	```
170
171	Those first lines are the command section. One line per ref being updated, saying "I think `refs/heads/main` is at `old-oid` right now and I want it at
172
173	Two things here I want to point out because both of them made my life easier later.
174
175	First, that `old-oid` is a compare and swap, and git did the heavy-lifting for me. If my stored ref doesn't match it, the push is a non fast forward and I reject it. I did not have to invent any of that, it comes with the protocol.
176
177	Second, if `new-oid` is all zeroes, the client is deleting the branch. Same route, same format, you just drop the ref.
178
179	After reading the commands and storing what I need, the whole body goes into `git receive-pack --stateless-rpc` and git writes the objects to disk. So no, I did not write a packfile parser. Please don't write a packfile parser.
180
181	## So what is a git host then
182
183	3 HTTP routes, 4 hex digits of framing, and `spawn('git')`. That is the transport, all of it. The UI, pull requests, stars, that little language colour bar, all of it is a normal web app sitting next to a folder of bare repos.
184
185	Which is fine until you remember that "a folder of bare repos" only works if you have one server and it never dies. That's where the rest of the work went.
186
187	## Write the useless version first
188
189	Before any of the storage stuff, here is a git host. It's about 40 lines, it has no database, no auth and no S3, and you can clone from it and push to it right now.
190
191	```js
192	import express from "express
193	import { spawn }
194	import path from "node:path
195	import fs from "node:fs
196
197	const app = express
198	const ROOT = "
199
200	function repoDir(req)
201	const { user,
202	const dir =
203	if (!fs.existsSync
204	return dir;
205	}
206
207	const pkt = (
208	(Buffer.byteLength(line
209
210	app.get("/:user/:repo/info/refs"
211	const service =
212	if (service !==
213	return res.sendStatus
214	}
215
216	res.setHeader("Content-Type
217	res.setHeader("Cache-Control
218	res.write(pkt
219	res.write("0000
220
221	const git =
222	service.replace("git-
223	"--stateless-rpc",
224	"--advertise-refs",
225	repoDir(req),
226	]);
227	git.stdout.pipe
228	});
229
230	for (const service
231	app.post(`/:user/:repo/
232	res.setHeader("Content-Type
233	res.setHeader("Cache-Control
234
235	const git =
236	service.replace("git-
237	"--stateless-rpc",
238	repoDir(req),
239	]);
240	req.pipe(git
241	git.stdout.pipe
242	});
243	}
244
245	app.listen(3001);
246	```
247
248	Then:
249
250	```sh
251	git clone http://localhost:3001/me/test.git
252	cd test && echo
253	git push origin main
254	```
255
256	So if it's that easy, what is the rest of this post about? Everything this version quietly gets wrong:
257
258	- Anyone can push to anyone's repo. There is no auth at all.
259	- The repos only exist on this one machine's `/tmp`. Restart the box and they might be gone.
260	- Two people pushing at once is decided by whichever git process gets to the ref file first.
261	- There is no way to see any of it in a browser.
262	- Push something over 1 MiB and it breaks, for a reason that took me an embarrassingly long time to find.
263
264	Every section after this is me fixing one of those.
265
266	## Auth, and why git has to be challenged first
267
268	Git has no concept of cookies. It speaks HTTP Basic, and only the server responds with 401 initially.
269
270	So the flow is: git asks for something, my server answers `401` with a `WWW-Authenticate: Basic realm="Ghost"` header, and only then does git either look up the credential helper or prompt for a username and password. If I return a 401 without that header, git just fails and the user never gets asked for anything, which looks like my server being broken.
271
272	The password is not the account password. It's a personal access token, made in account settings, exactly like GitHub does it. The username is ignored entirely. I used better-auth's [API keys](https://better-auth.com/docs/plugins/api-key) for it but you can use anything
273
274	```ts
275	const header = req
276	if (!header?.startsWith
277
278	// username is ignored, the password is the key, and a key can contain ":"
279	const decoded = Buffer
280	const key = decoded
281
282	const { valid,
283	return valid && apiKey
284	```
285
286	2 more things
287
288	Read and write are decided before anything runs. A clone of a public repo needs no auth. A push always does. Git asks for the receive-pack advertisement before it uploads anything, so the check is:
289
290	```ts
291	const isPush =
292	req.path.endsWith
293	req.query.service
294	```
295	We turn a user away at `info/refs` instead of during the actual push after they've spent their precious CPU cycles and ram compressing stuff(its expensive these days yk).
296
297	And this all runs in middleware, before the body is spooled anywhere, so a rejected push never touches my disk at all(disks are also expensive these days).
298
299	## Now the actual build
300
301	Ghost is a Turborepo. Next.js on the front, NestJS for the API, Postgres with Drizzle, and an S3 bucket. Bun everywhere. All the decisions are written down in `docs/` in the repo. Below are some of the crucial decisions that I made:
302
303	### Git gets the root, REST goes under `
304
305	`/:username/:repo/info/refs` matches almost every URL you can think of. I had it registered next to `/repositories` and for a while it worked, until I noticed which one wins is decided by module import order in `
306
307	So now it's `app.setGlobalPrefix('/api')` with the git routes excluded from the prefix. Everything REST sits under `/api` and the root is git's. Same split GitHub does with
308
309	The other option was serving git under `/git/:username/:repo`, which works and which I did not want, because then every clone URL I ever hand out has a `/git` in it. People copy paste those into READMEs. I want mine to read
310
311
312	### Keeping HTTP out of `
313
314	`GitService` takes streams and gives back `{ headers, body }`. The controller is the only file that knows HTTP is involved.
315
316	There were a few reasons:
317	1. I want to reuse `GitService` when I implement ssh.
318	2. Separation of concern, just to keep the code sensible and reduce mental load.
319
320	### S3 is the truth, disk is a cache
321
322	This is the decision the rest of the thing hangs off, and honestly part of it was my hardware.
323
324	I develop this on a Mac mini with a 256GB SSD. I cannot keep a bunch of test repos on it, and I definitely cannot clone something like a 7GB repo to see what my server does with it. If disk is where the data lives, then the size of my laptop is the size of my product, which is a stupid place to be.
325
326	The other half is the normal reason. A bare repo on local disk does not survive a crash, does not survive the machine getting replaced, and does not work at all once there is more than one server. A box that has never seen a repo still has to be able to serve it.
327
328	So the real state of a repo is a write ahead log in S3:
329
330	```
331	repos/<repoId>/index the only mutable object
332	repos/<repoId>/entries/<ulid>.pack immutable, header + packfile bytes
333	```
334
335	And the bare repos on disk are just a cache. They live in `os.tmpdir()`, so `/tmp/ghost/<repoId>.git` on my machine, keyed by the repo's row id and not by
336
337	### A push becomes real at exactly one line
338
339
340	<div className="gh-fig"
341	<div className="gh-steps
342	<div className="gh-step
343	<div className="gh-step
344	<div className="gh-step
345	<div className="gh-step
346	<div className="gh-step commit
347	<div className="gh-step
348	</div>
349	<div className="gh-cap
350	</div>
351
352
353	The reason I can do that is that a push is carrying two things that have nothing in common.
354
355	The objects are named after their own hash. Push the same blob twice and the second one lands on top of an identical file, so it does not matter if I upload it twice, and it does not matter if I upload it and then the push dies. Worst case some bytes sit in the bucket that nobody points at. Nobody can see them, no clone will ever ask for them, they just cost me storage.
356
357	The refs are the opposite. `refs/heads/main` is one value that everyone reads, so if two people move it at the same time one of them has to lose.
358
359	So I upload the big slow thing with nothing protecting it at all, and then the tiny 200 byte index write is the only place I need to be careful. Steps 1 to 4 can be repeated, interrupted, or thrown away and the repo does not notice. If the process dies at step 5, the repo is exactly what it was before the push started.
360
361	Free bonus, a push touching 5 branches is a single index write, so it's all or nothing. Local git needs `--atomic` to give you that.
362
363	S3 caveats that I learnt quite late:
364
365	- Do not treat 409 status code from S3 as a failure. `ConditionalRequestConflict` just means that two conditional writes raced and the one that threw that error failed, so try again.
366	- `IfNoneMatch: '*'` on the first ever push. Otherwise two pushes into a brand new repo can both see "no index here" and both happily write one.
367
368
369	### What happens when 2 people push at once
370
371	Nothing clever. They race and one of them loses.
372
373	Both read `seq = 5`, both build an index that says 6, both send it with `If-Match` on the etag they read. S3 takes the first one and rejects the second. The loser reads the index again, sees 6 is taken, and goes again as 7.
374
375	Where I messed up was doing the validation once before that loop. That felt fine until I thought about who actually lost the race:
376
377	If the winner pushed `develop` and I'm pushing `main`, my push is still completely valid.
378
379	If the winner pushed `main` too, then the `main` I checked against is gone. My
380
381	So the check has to run again on every attempt, against the index I just re-read. If it only runs once, two people pushing to the same branch within the same second both get a 200 and one of their commits quietly disappears. That is not a bug you find by clicking around.
382
383	Then there's the annoying case. A PUT times out and I have no idea whether it landed. If I retry, `oldOid` won't match anymore and I'd tell someone their push was rejected when it actually went through, which is a great way to lose trust in a git host. So the entry's ULID doubles as an idempotency key. Before I report anything, I check whether the index already lists my ULID, and if it does, that was me, the push is in.
384
385	### Why the log is binary and not JSON
386
387	I hand rolled a binary format. Big endian, magic prefix, version byte. I'm aware of how that sounds.
388
389	The index is read on every push and every time a cache catches up, so it's the hottest thing in the system. Most of its size is object ids, and in JSON those have to be hex, so 40 bytes for something that fits in 20, plus quotes and key names around each one. Storing raw halves that and there's no parse step.
390
391	Entry header:
392
393	```
394	magic u32 "GENT"
395	version u8
396	headerLen u32 the packfile starts at exactly this offset
397	ulid 16 raw bytes
398	createdAt u64
399	pushedBy u16 length + utf8
400	transitionCount u32
401	refName u16 length + utf8
402	oldOid 20 raw bytes
403	newOid 20 raw bytes
404	<packfile bytes>
405	```
406
407	`headerLen` at the front is the part I like. It means I can read who pushed an entry and what it changed with `Range: bytes=0-65535` instead of downloading a 500MB packfile to find out.
408
409	There's also a `flags` byte sitting there doing nothing. It's for sha256. Git is slowly moving off sha1 and if that ever reaches me, oids stop being 20 bytes and every reader has to know which size it's looking at. With the byte there it's one bit and a branch in the decoder. Without it I'd be guessing hash length from field offsets, or rewriting every entry in every repo. It costs 1 byte per object so I just put it in and left it at zero.
410
411	Same reasoning for `version`. Decode checks it and throws `WalCorruptError` on anything it doesn't know.
412
413	### Index holds a snapshot, entries hold the diff
414
415	I store both, which looked redundant until I tried dropping either one.
416
417	The snapshot answers "where is `main` right now" in a single GET with no history to walk. That's every ref advertisement and every cache catch up, so it has to be cheap.
418
419	The transitions answer "where was `main` at push 37". That's reflog, audit, and restoring to a point in time, none of which a snapshot can give you.
420
421	Keep only the transitions and every read costs you the whole history. Keep only the snapshot and you have deleted the reason you built a log instead of a folder.
422
423	### Catching the cache up
424
425	Before every read and every push, the cached bare repo gets brought up to whatever the log says. It remembers where it is in a `ghost-wal-seq` file inside the repo directory.
426
427	```
428	read index -> compare to cached seq -> index-pack each missing layer in order
429	-> reconcile refs to the snapshot -> repoint HEAD -> write cached seq
430	```
431
432
433	<div className="gh-fig"
434	<div className="gh-bar
435	<div className="gh-cell
436	<div className="gh-cell
437	<div className="gh-cell missing filling
438	<div className="gh-cell missing filling d1
439	<div className="gh-cell missing filling d2
440	</div>
441	<div className="gh-cap
442	</div>
443
444	The cache can be behind the log and can never be ahead of it, which is exactly what committing before touching disk bought me. So this only ever moves forward and I never had to write an undo path, which is good, because undo paths are where I would have put the bug.
445
446	3 things I found out the slow way here.
447
448	Pushed packs are thin. The client leaves out base objects it knows the server already has, and replaying that into an empty object store just fails. `git index-pack --fix-thin --stdin` fills in the gaps from what's already in the repo, and layers have to go in strictly one after another, never in parallel, because layer n's bases are sitting in the layers before it.
449
450	Refs get reconciled, not replayed. I take the index snapshot and apply it in one `git update-ref --stdin` batch, then delete any ref on disk that the snapshot doesn't have. Replaying transitions one at a time gets you to the same place for more money, and it breaks if the cache drifted even slightly. Applying a snapshot works from any starting state, so a stale cache, or one I poked at by hand, fixes itself.
451
452	HEAD is a trap. If a bare repo's `HEAD` points at a branch that doesn't exist, cloning it gives you an empty repo and no error whatsoever. That one cost me an evening. So after reconciling I point HEAD at `main`, then
453
454	## Showing the repo in a browser
455
456	The transport is done at this point but nobody can see anything. This part surprised me by being the easy half, because git ships plumbing commands that are meant to be read by programs and not humans.
457
458	Resolve whatever the URL says into a sha:
459
460	```
461	git rev-parse --verify <ref>^{commit}
462	```
463
464	List one directory, not the whole tree:
465
466	```
467	git ls-tree -z -l <ref>:<path>
468	```
469
470	`-z` gives you NUL separated records so filenames with spaces or newlines don't ruin your day, and `-l` adds the blob size. You get mode, type, oid, size and name per entry, and a mode of `
471
472	Read a file:
473
474	```
475	git cat-file --batch-check <- stdin: "<ref>:<path>"
476	git cat-file blob <oid>
477	```
478
479	I do the `--batch-check` first because it tells me the type and size without sending me the contents, so I can refuse to syntax highlight a 40MB binary before reading it. Also note the revision goes in over stdin, not argv, because a path starting with `-` in argv is an option waiting to happen.
480
481	Commits are `git log`, diffs are `git diff --numstat` plus `
482
483	### The one place this got slow
484
485	A directory listing on GitHub shows the last commit that touched each row. The obvious way to do that is one `git log -1 -- <path>` per entry, so a folder with 30 files costs 30 `git log` invocations, each of which walks history. On a repo with any real history that is unusably slow, and it's per page view.
486
487	So there's a table:
488
489	```
490	repository_path_commit(repositoryId, ref, path, commitSha, committedAt, subject)
491	```
492
493	Every path gets a row, and so does every ancestor directory of a changed file, so `src` carries the newest commit under `src/`. A directory listing is then one indexed query joined against
494
495	Second table tracks how far I've walked:
496
497	```
498	repository_ref_index(repositoryId, ref, indexedCommitSha)
499	```
500
501	If the stored sha is still an ancestor of the current tip, I only need to walk the new commits. If it isn't, someone force pushed or objects went missing, so I throw the rows away and rebuild. That check is one `git merge-base --is-ancestor` away.
502
503	Rows for deleted files stick around, which sounds like a bug and isn't, because the read path joins against `ls-tree`. If a file isn't in the tree anymore it never shows up, no matter what the table says.
504
505	### The rest of the data model
506
507	Nothing exotic, it's just Postgres:
508
509	- `user`, `session
510	- `repository` with `ownerId
511	- Unique index on `(ownerId, slug)` where the org id is null, and on `(organizationId, slug)
512	- `pullRequest` storing base and head as separate repository ids, because a fork PR spans 2 repos and a normal one doesn't.
513
514	The repo id is what everything else keys off, including the S3 log and the cache directory. Not `username/repo`. The first version used the path and then I renamed a repo in dev and watched it forget its entire history, which was a fun 10 seconds.
515
516	## 3 bugs that were worth the whole thing
517
518	### 1. Never leave a stream sitting there across an `
519
520	The push path used to do this:
521
522	```ts
523	const repoDirectory = await
524	const body = await
525	```
526
527	If nobody is reading a request stream, whatever arrives while you're busy is gone. Git writes its command section in its own socket write, so the chunk I dropped during that await was the ref updates. The body then started at the flush packet after them, and my parser said "no ref update commands", which is a completely accurate error message pointing nowhere near the actual problem.
528
529	It only shows up when the await is slow enough, so it appeared against real S3 and never once in tests.
530
531	Swapping those two lines fixes today's symptom and not the actual shape of the bug. The moment I add an async guard (and Basic auth for git is going to be one), the stream is sitting unread again during the guard, before my handler even runs. So the body gets drained in middleware now, which in Nest runs before guards. Nothing async can get between the socket and the first read.
532
533	A stream is not a value sitting there waiting for you. It's already running.
534
535	### 2. The empty POST is a probe, not a broken push
536
537	This one only shows up above 1 MiB, which is a lovely quality in a bug.
538
539	A push bigger than `http.postBuffer` (1 MiB by default) can't be buffered, so git sends it chunked. A chunked request can't be rewound, so if git gets a 401 it can't replay the body with credentials attached. Its answer is `probe_rpc()` in
540
541	From an actual push:
542
543	```
544	POST #1 bytes=4 content-length=4 head="0000"
545	POST #2 bytes=2428943 transfer-encoding=chunked head="00b600000000"
546	```
547
548	I was answering that probe with a 400, because a body with no ref update commands in it is obviously broken. So:
549
550	```
551	error: RPC failed; HTTP 400 curl 22 The requested URL returned error: 400
552	send-pack: unexpected disconnect while reading sideband packet
553	fatal: the remote end hung up unexpectedly
554	```
555
556	My tests pushed a 95 KB repo and a 178 KB repo. Both under `postBuffer`, so both used `Content-Length` and neither ever probed. Everything passed. Every real repo failed every time.
557
558	The fix is 4 lines. If the whole body is one flush packet, return 200 with an empty body and don't touch anything.
559
560	### 3. `
561
562	Pushed the Next.js repo. 2.25 GiB packfile.
563
564	```
565	RangeError: data is too long
566	at Hash.update (node:internal/crypto/hash:144:22)
567	at PushTransactionService.commitPush (push-transaction.service.ts:46)
568	code: 'ERR_OUT_OF_RANGE'
569	```
570
571	Node's native hash throws when you give it more than `INT_MAX`, so 2,147,483,647 bytes. My pack was 2,415,919,104.
572
573	`createHash('sha256').update(pack)` was never going to work on a real repo.
574
575	Fixing the hash alone would have moved the crash down one line, since every one of these was the same problem:
576
577
578	\| Was \| Now \|
579	\| --- \| --- \|
580	\| `readStream(req)` into a Buffer \| spooled to a temp file \|
581	\| `createHash().update(pack)` \| `pipeline(body.open(offset), hash)
582	\| `Buffer.concat([header, pack])` \| an async generator \|
583	\| `putObject({ Body: buffer })` \| stream + explicit `ContentLength
584	\| `readEntryPack()` -> Buffer \| `openEntryPack()
585
586	The body is a `GitRequestBody` now, which is `{ size, open(start?) }`. Not a plain
587
588	The test that guards this streams 2.15 GiB through `commitPush` in about a second.
589
590	## Pull requests, or 2 logs that can't see each other
591
592	Every repo has its own log keyed by its row id. Forking copies the parent's layers into the fork's keyspace and writes an index naming them, and after that moment the two logs never exchange another byte. Separate keyspaces, separate sequence numbers, neither one can read the other.
593
594	So a PR from a fork is a diff between 2 commits living in 2 different object stores. Alice's log has no `B4`, Bob's log has no `A1`, and
595
596
597	<div className="gh-fig"
598	<div className="gh-cols
599	<div className="gh-col
600	<h4>repos/repo_bob (base)</h4>
601	<div className="gh-layer
602	<div className="gh-layer
603	<div className="gh-layer
604	<div className="gh-layer new
605	</div>
606	<div className="gh-borrow
607	objects
608	<span className="arrow
609	borrowed
610	</div>
611	<div className="gh-col
612	<h4>repos/repo_alice (fork)</h4>
613	<div className="gh-layer
614	<div className="gh-layer
615	<div className="gh-layer
616	<div className="gh-layer new
617	</div>
618	</div>
619	<div className="gh-cap
620	</div>
621
622	What saved me is alternates, which is a git feature I had never touched before this.
623
624	Normally a repo reads objects from its own `objects` directory and nowhere else. Alternates let you tell git "also look in this other directory if you can't find something here". It's the same mechanism `git clone --shared` uses so a local clone doesn't duplicate the object store. You can set it permanently in
625
626	```
627	GIT_DIR=/tmp/ghost/repo_bob.git
628	GIT_ALTERNATE_OBJECT_DIRECTORIES=/tmp/ghost/repo_alice.git/objects
629
630	git merge-base B4 A1 -> B3
631	git diff --numstat B3 A1
632	git log B3..A1
633	git merge-tree --write-tree B4 A1
634	```
635
636	Now git is running inside Bob's repo, but when it goes looking for `A1` and can't find it, it checks Alice's object directory and finds it there. So `merge-base` works, the diff works,
637
638	The env var is the important half. Writing it into `objects/info/alternates` would make Bob's repo permanently depend on a directory that belongs to a fork he does not control, and if I ever delete that fork, Bob's repo is now missing objects. The env var lives for exactly one `spawn`, so a PR read borrows the objects for a few milliseconds and then the link is gone.
639
640	Nothing gets copied, nothing gets written to either repo, and there is nothing to clean up after. A pull request in Ghost is a database row naming 2 `(repository, ref)` pairs and holds no git state at all. No `refs/pull/*`, no snapshot branch, no copied objects. It can't drift from the repos it describes because there is nothing there to drift.
641
642	The obvious alternative is fetching from the other cache directory first, and it's worse in 3 ways. Fetching a raw sha needs `uploadpack.allowAnySHA1InWant` set on the source. Fetching a ref instead dumps it in `FETCH_HEAD`, which is one file per repo, so 2 people opening PRs at the same time stomp on each other. And either way you're copying objects that already exist one directory away.
643
644	A scratch ref like `refs/ghost/pr/*` is worse still, because materialization deletes every ref the snapshot doesn't carry. So it survives right up until the next push to that repo. Works great locally, dies under traffic.
645
646	The one thing here you cannot get wrong is packing the merge commit. When both tips are in the same log, excluding both is correct. Do the same across a fork, excluding `A1` because hey, I can see it right there, and you write an entry containing only the merge commit. Bob's log is then broken forever, because any server replaying it gets a merge commit whose second parent isn't there, and `--fix-thin`
647
648	So the exclusion list is what the target log already has. Not what my process happens to be able to see at that moment.
649
650	Also `git merge-tree --write-tree` exits 1 both for a conflict and for a commit it can't read. You tell them apart by stderr, a conflict leaves it empty and an unreadable commit writes `not something we can merge`. Treat every exit 1 as a conflict and a fork whose objects were never lent reports conflicts instead of failing, and that bug looks like a product decision instead of a bug.
651
652	## What isn't built
653
654	This is an engineering reproduction, so don't put real data in it. The holes:
655
656	SSH. Everything above is HTTP. SSH is a different transport for the same 3 verbs and I haven't done it.
657
658	Garbage collection. Entries uploaded by a push that then lost the CAS are orphans that nobody references, so storage grows with failed pushes. The sweep is easy to get wrong though. "Delete entries no index references" will happily delete a push that is mid upload right now. It has to be unreferenced and older than a few hours. The ULID has a timestamp in it so the age is free.
659
660	Compaction. Reading the whole log on every push is O(history). `compactedThroughSeq` is in the format for a checkpoint scheme and is currently always 0.
661
662	Push failures over the sideband. `NonFastForwardError` renders as JSON right now, and git clients do not enjoy that. Needs a pkt-line status encoder.
663
664	Webhooks, issues, CI. Nope.
665
666	Code is at [github.com/phantomknight287/ghost](https://github.com/phantomknight287/ghost). It works. Still don't put anything you care about in it.