flaxo/ghost

public

Source code of Ghost - On Ghost

Add repository settings, and make deletion purge everything145628016 days ago292 lines · 11.2 KB

1	# 0016 — A pull request spans two logs, and lends objects rather than copying them
2
3	Status: adopted
4
5	## Decision
6
7	A pull request is a database row naming two `(repository, ref)` pairs. It stores
8	no git state of its own: no `refs/pull/*`, no snapshot branch, no copied
9	objects.
10
11	Everything the request needs to answer — the merge base, the commit list, the
12	diff, whether it merges cleanly — is computed from the two repositories' caches
13	at read time, with the other side's object store lent through
14	`GIT_ALTERNATE_OBJECT_DIRECTORIES`.
15
16	The merge itself is an ordinary push: a merge commit and a packfile, handed to
17	`PushTransactionService.commitPush`, the same commit point a `git push` uses.
18
19	## The problem this exists to solve
20
21	[0004](0004-object-storage-is-the-truth.md)
22	write-ahead log, keyed by the repository row id:
23
24	```
25	repos/<repositoryId>/index
26	repos/<repositoryId>/entries/<ulid>.pack
27	```
28
29	`forkRepository` calls `WalStoreService.copyLog`, which copies the parent's
30	layers into the fork's keyspace and writes an index naming them. After that
31	instant the two logs never exchange another byte. They are independent, they
32	advance on different sequence numbers, and neither can read the other.
33
34	So a fork request is a diff between two commits that live in **two different
35	object stores**. Nothing in the read path could see both until this decision.
36
37	## Worked example
38
39	### Case 1 — a branch in one repository
40
41	Alice owns `alice/repo-a`. She branches `feature` off `
42
43	```
44	repos/repo_a1b2/index seq 4
45	refs { refs/heads/main: A3, refs/heads/feature: A5 }
46	layers [L1, L2, L3, L4]
47
48	/tmp/ghost/repo_a1b2.git ghost-wal-seq: 4
49	```
50
51	One log. `commitPush` wrote both refs, so `A3` and `
52	layers `L1..L4`. Nothing is lent, because there is nothing to lend:
53
54	```
55	merge-base A3 A5 → A2
56	diff --numstat A2 A5
57	log A2..A5
58	```
59
60	Merging builds a merge commit `M` and packs it with **both
61	because both are already in the log:
62
63	```
64	pack-objects --revs ← "M\n^A3\n^A5\n" one object
65	```
66
67	```ts
68	commitPush({
69	repoId: 'repo_a1b2',
70	transitions: [{ ref:
71	body: fileBody(pack
72	packOffset: 0,
73	})
74	```
75
76	Result: seq 5, `refs { main: M, feature: A5 }`. The head branch is left alone.
77
78	Deleting it on merge is one more transition in the same entry, because
79	`applyTransitions` reads `ZERO_OID` as a delete:
80
81	```ts
82	{ ref: 'refs/heads/feature',
83	```
84
85	That would make "merge and delete the branch" a single compare-and-swap rather
86	than two. It is deliberately not wired to the merge endpoint: it only ever works
87	for a request inside one repository, since a fork's branch lives in a log this
88	push cannot touch, and whether to delete is the author's call rather than a
89	default.
90
91	### Case 2 — a fork
92
93	Alice forks `bob/repo` to `alice/repo-b`, branches `
94	request back into `bob/repo`.
95
96	Before the fork, Bob is at seq 3:
97
98	```
99	repos/repo_bob/index seq 3, refs { main: B3 }, layers [L1, L2, L3]
100	```
101
102	The fork copies the layers byte for byte and writes an index naming them:
103
104	```
105	repos/repo_alice/index seq 3, refs { main: B3 }, layers [L1, L2, L3]
106	```
107
108	Same ULIDs, same bytes, separate keyspace. This is the last time the two logs
109	agree on anything.
110
111	**Alice pushes `feature
112
113	```
114	repos/repo_alice/index seq 4, refs { main: B3, feature: A1 }, layers [L1..L4]
115	repos/repo_bob/index seq 4, refs { main: B4 }, layers [L1, L2, L3, L4']
116	```
117
118	Both read seq 4. The numbers are unrelated: `L4` and `L4'` are different ULIDs
119	holding different objects. **Alice's log has no `B4
120	Neither cache can answer `merge-base B4 A1` on its own — `cat-file -t` on the
121	other side's tip fails outright.
122
123	Reading the request. Materialize both caches, then run every command in the
124	base's directory with the head's object store lent to it:
125
126	```
127	GIT_DIR=/tmp/ghost/repo_bob.git
128	GIT_ALTERNATE_OBJECT_DIRECTORIES=/tmp/ghost/repo_alice.git/objects
129
130	git merge-base B4 A1 → B3
131	git diff --numstat B3 A1
132	git log B3..A1
133	git merge-tree --write-tree B4 A1
134	```
135
136	Zero bytes are copied, nothing is written to either repository, and there is
137	nothing to clean up afterwards.
138
139	Merging. Same lent store, and the pack is where the care goes:
140
141	```
142	git merge-tree --write-tree B4 A1 → T
143	git commit-tree T -p B4 -p A1 -m "..." → M
144	printf "M\n^B4\n" \| git pack-objects --revs <prefix>
145	```
146
147	```ts
148	commitPush({
149	repoId: 'repo_bob',
150	transitions: [{ ref:
151	body: fileBody(pack
152	packOffset: 0,
153	pushedBy: bobUserId,
154	})
155	```
156
157	```
158	repos/repo_bob/index seq 5, refs { main: M }, layers [L1, L2, L3, L4', L5']
159	repos/repo_alice/index seq 4 ← untouched
160	```
161
162	Bob's log is self-contained again. Alice's fork is unchanged and her `feature`
163	branch still exists. Only bytes crossed, and they crossed as a normal push.
164
165	## Why alternates instead of a fetch
166
167	The obvious alternative is `git fetch <otherCacheDir> <sha>` before each read.
168	Three things are wrong with it:
169
170	- Fetching a raw sha needs `uploadpack.allowAnySHA1InWant` on the source, so it
171	either requires config on every cache or an `--upload-pack` override.
172	- Fetching a ref instead lands in
173	repository. Two concurrent requests against the same base clobber each
174	other's.
175	- It copies objects that already exist a directory away.
176
177	A scratch ref is worse. `reconcileRefs` deletes every ref the index snapshot
178	does not carry, and its `for-each-ref` has no namespace filter, so a
179	`refs/ghost/pr/*` ref survives exactly until the next push lands on that
180	repository — the kind of lifetime that works in development and fails under
181	traffic.
182
183	Alternates have none of these problems: process-scoped, no file, no copy, no
184	cleanup, and safe under concurrency because nothing is written.
185
186	## The pack must exclude the base, never the head
187
188	This is the one place a wrong shortcut is unrecoverable.
189
190	In case 1, `^A3 ^A5` was correct because both tips were already in that log. The
191	same instinct applied to case 2 — excluding `A1` because "we can see it" — writes
192	an entry containing only `M`. Bob's log is then permanently broken: a node that
193	replays `L1..L4', L5'` gets a merge commit whose second parent is unreachable,
194	and `index-pack --fix-thin` cannot help, because there is nothing on that node
195	to complete the pack against.
196
197	So the exclusion set is what the target log already holds, never what the
198	current process happens to be able to see:
199
200	```
201	include: M
202	exclude: [B4] the base tip, and only the base tip
203	```
204
205	Everything reachable from `M` and not from `B4` — the merge commit, `
206	every tree and blob Alice introduced — is read through the alternate and written
207	into the pack. `merge.spec.ts` builds exactly this pack, replays it into a bare
208	repository that has only the base's objects, and asserts the full history walks.
209
210	## Conflicts and errors both exit 1
211
212	`git merge-tree --write-tree` exits 1 for a conflict and
213	cannot read. They are told apart by stderr: a conflict writes the tree and the
214	conflicted paths to stdout and leaves stderr empty; an unreadable commit writes
215	`not something we can merge` to stderr.
216
217	Treating every exit 1 as a conflict is the dangerous reading — a fork whose
218	objects were never lent would report "conflicts" instead of failing, and the
219	bug would look like a product decision. `mergeTree` returns null only on exit 1
220	with empty stderr, and rethrows otherwise.
221
222	## Concurrency
223
224	`assertFastForward` is a compare-and-swap on the expected `oldOid`, with no
225	ancestry check of its own. A merge commit therefore passes it exactly when the
226	base tip is still what the merge was built against.
227
228	If Bob pushes `B5` while a merge is in flight, the CAS fails with
229	`NonFastForwardError` and the merge is retried from `merge-tree` against `B5
230	The entry already uploaded is orphaned, which [0010](0010-orphans-are-garbage.md
231	defines as garbage rather than corruption, so there is nothing to unwind.
232
233	Mergeability is never stored. It is a function of two moving branch tips, and a
234	cached answer is wrong the moment either side is pushed to.
235
236	## What the API exposes
237
238	```
239	POST /api/repositories/:username/:repo/pulls
240	GET /api/repositories/:username/:repo/pulls
241	GET /api/repositories/:username/:repo/pulls/:number
242	GET /api/repositories/:username/:repo/pulls/:number/commits
243	GET /api/repositories/:username/:repo/pulls/:number/files
244	GET /api/repositories/:username/:repo/pulls/:number/patch
245	POST /api/repositories/:username/:repo/pulls/:number/merge
246	PATCH /api/repositories/:username/:repo/pulls/:number/close
247	```
248
249	`head` is `branch` for a request inside one repository and `owner:branch
250	fork. `owner:branch` resolves only within the fork network — the base itself, a
251	fork of the base, or the repository the base was forked from — because two
252	repositories with no `parentRepositoryId` between them are not a pull request,
253	they are unrelated history.
254
255	Base visibility governs every read: a request from a private fork into a public
256	repository is readable by anyone who can read the base, which is the same rule
257	the diff itself implies. Merging requires write on the base. Closing requires
258	the author or write on the base.
259
260	`number` is allocated per base repository with
261	`select coalesce(max(number), 0) + 1`, which two concurrent opens can read
262	identically. The unique index on `(baseRepositoryId, number)` rejects the loser,
263	and the retry reads the winner's number. A partial unique index on
264	`(baseRepositoryId, baseRef, headRepositoryId, headRef) where state = 'open'`
265	stops duplicate open requests for a branch pair, while leaving closed and merged
266	rows as history so the same branch can be proposed again.
267
268	## Consequences
269
270	- A pull request holds no git state, so nothing about it can drift from the
271	repositories it describes.
272	- Reading one costs two `readIndex` calls (both usually no-ops, since
273	`materialize` returns early when the cache is at the index seq) and a handful
274	of git processes. No object is ever copied for a read.
275	- Both caches must be on the same node. That is already true of every read path,
276	and it is the assumption to revisit first if the API is ever sharded by
277	repository.
278	- The merge pack excludes only the base tip, so objects on the base's other
279	branches can be packed a second time. That is entry size, not correctness;
280	excluding every base ref is the fix if it ever matters.
281	- Renaming a user or a repository is now safe. Logs and caches are keyed by row
282	id, which is what [0011](0011-deferred.md
283	cross-repository reference meaningful at rest.
284
285	## Addendum: a merged request reads only the base
286
287	Once merged, a request's range is frozen as `mergeCommit^1..headSha`, read from
288	the base cache alone: the merge pushed every commit it needs into the base
289	log. This keeps a merged request's commits and diff after its head branch is
290	deleted or its head repository is gone
291	([0020](0020-deletion-is-a-tombstone-then-a-purge.md
292	branches, which now contain each other, from reporting an empty diff.