pippinmole/ghost

public

Source code of Ghost - On Ghost

git: add a manual script that finds and repairs repositories whose log cannot be replayed107915e10 days ago46 lines · 4.7 KB

1	# 0034 — A push is verified before the log commits it, and the log stores the indexed pack
2
3	Status: adopted. Amends [0005
4
5	## Decision
6
7	`GitService.receivePack` used to commit a push to the log and only then hand it to `git receive-pack`. Every check receive-pack makes therefore ran after the commit point, and a push it refused was already permanent. One crafted request from anyone with write access could make a repository unreadable for good:
8
9	```
10	POST /owner/repo/git-receive-pack refs/heads/evil -> abab…ab, empty pack
11	→ receive-pack: "ng refs/heads/evil unpacker error"
12	→ the log: refs { …, refs/heads/evil: abab…ab }
13	→ every materialize: update-ref fails, every clone: 500
14	```
15
16	Deleting the ref through git cannot repair it: a delete has to name the ref's current value, and a ref the cache never holds is never advertised.
17
18	Now a push passes three checks before `commitPush`:
19
20	1. Names, at the commit point itself (
21	2. Objects, in
22	3. Reachability, in the same place. Every object the new refs reach beyond the existing refs must be in the quarantine, looked up there alone.
23
24	The third check exists because the cache is not the log. It also holds objects the log never received: the trees `merge-tree` writes on every pull request view, merge commits from merges that failed, test merges a size limit refused. Their names are computable, and a test merge's is deterministic. A ref at one of them passes every check against the cache and fails on the first node that rebuilds from the log.
25
26	## The log stores the indexed pack
27
28	A thin pack's delta bases come from whatever the receiving side holds. Validated against the cache, they could be objects only the cache holds. The log therefore stores the pack `index-pack --fix-thin` wrote into the quarantine, with every base it borrowed appended, rather than the bytes received. A node replaying the log never needs a base from anywhere else.
29
30	The cache still receives the original request through `receive-pack`, which resolves the same bases from the same cache.
31
32	## Consequences
33
34	- A push git would refuse now gets a 422 naming the ref and the reason, instead of landing and breaking the repository.
35	- A push is indexed twice: once into the quarantine, once by `receive-pack` into the cache. Moving the quarantine's pack into the cache instead would save the second pass.
36	- Layers grow by whatever delta bases a thin pack borrowed. Layers written before this decision may still be thin, so replay keeps `--fix-thin` and sequence order.
37
38	## Repairing a log broken before this decision
39
40	`src/scripts/repair-repositories.ts` (`node dist/scripts/repair-repositories.js [--apply]`, run by hand) checks every repository and, with `
41
42	1. Replay the log into an empty scratch repository, never into the cache: the cache can hold objects the log lacks and so hide exactly this damage. An entry that does not index is retried with the local cache lent for delta bases; if that works, a new entry holding the self-contained pack replaces it in place under the old entry's header. Otherwise its layer's size becomes 0, which replay skips. Both keep every layer at its sequence number, so caches stay aligned.
43	2. Check every ref. A ref whose history the scratch repository cannot walk gets the missing objects from the local cache, written back as a new entry. Then a ref is dropped when it is still unwalkable, names a branch at anything but a commit, has a name git refuses, or cannot coexist with another ref; of two that cannot coexist, the one whose entry came later goes.
44	3. Commit. Layer swaps are one compare-and-swap of the index. Dropped refs and rescued objects are one ordinary
45
46	Run it on the API host, where the caches are: elsewhere it finds everything but can rescue nothing.