flaxo/ghost

public

Source code of Ghost - On Ghost

it now supports push and pull50ad2a1a month ago76 lines · 3.2 KB

1	# 0015 — Nothing holds a packfile in memory
2
3	Status: adopted — fixes a hard failure above 2 GiB
4
5	## Decision
6
7	A push body is spooled to a temp file by middleware and passed down as a
8	`GitRequestBody` — `{ size, open(start?) }` — never a `Buffer
9	uploading, and replay all stream.
10
11	## What broke
12
13	Pushing the Next.js repository (2.25 GiB packfile):
14
15	```
16	RangeError: data is too long
17	at Hash.update (node:internal/crypto/hash:144:22)
18	at PushTransactionService.commitPush (push-transaction.service.ts:46)
19	code: 'ERR_OUT_OF_RANGE'
20	```
21
22	Line 144 is the native call. Node's C++ `Hash::HashUpdate` throws
23	`data is too long` when the input exceeds `INT_MAX` — 2,147,483,647 bytes. The
24	pack was 2,415,919,104.
25
26	`createHash('sha256').update(pack)` was never going to survive a real
27	repository, and [0011](0011-deferred.md
28	process heap… the first thing that breaks at scale."* This is that.
29
30	## Every place a large pack was fatal
31
32	Fixing the hash alone would have moved the failure one line down. All of these
33	were 2 GiB landmines and all are now streams:
34
35	\| Was \| Now \|
36	\|---\|---\|
37	\| `readStream(req)` into a Buffer \| spooled to a temp file \|
38	\| `createHash().update(pack)` \| `pipeline(body.open(offset), hash)
39	\| `Buffer.concat([header, pack])` \| `prefixed()
40	\| `putObject({ Body: buffer })` \| `Body
41	\| `readEntryPack()` → Buffer \| `openEntryPack()
42	\| `runGit({ input: Buffer })` \| `input: Buffer \\| Readable
43
44	## Why `
45
46	The push path needs the body twice: once to read the command section, once to
47	feed `receive-pack`. And the WAL needs the packfile alone, from `packOffset`
48	onward. A single-pass `Readable` cannot serve that, but neither may the handler
49	hold the bytes.
50
51	`{ size, open(start?) }` gives multiple passes and arbitrary offsets while
52	staying HTTP-agnostic, so [0002](0002-streams-not-http-objects.md
53	tests pass `bufferBody(...)`, production passes `fileBody(...)`, and neither the
54	service nor the log knows the difference.
55
56	## Reading the command section without reading the pack
57
58	`readReceivePackHeader` reads at most 1 MiB from the front of the body, parses
59	the pkt-lines, and returns `packOffset` and `packSize`. The packfile is never
60	touched — the log records where it is, not what it contains.
61
62	`ContentLength` on the upload is now mandatory: the body is a stream, so the SDK
63	cannot measure it, and measuring it in memory is precisely what is being
64	avoided.
65
66	## Consequences
67
68	- Every push writes its body to `os.tmpdir()` and deletes it when the response
69	closes. Disk must fit the largest push.
70	- The upload is still a single
71	but a stream cannot be rewound, so a mid-flight failure means re-uploading the
72	whole entry. Multipart via `@aws-sdk/lib-storage` is the follow-up
73	([0011](0011-deferred.md
74	this is a performance problem, not a correctness one.
75	- `push-transaction.service.spec.ts` streams 2.15 GiB through `commitPush` —
76	past `INT_MAX` — in about a second, which is the regression guard.