flaxo/ghost

public

Source code of Ghost - On Ghost

propose a separate delivery service for emails and webhooks741914e14 days ago49 lines · 4.5 KB

1	# 0031 — Outbound delivery is its own service, and Postgres is its queue
2
3	Status: proposed
4
5	## Decision
6
7	Anything that leaves the system over the network — thread emails today, webhooks next — is sent by `apps/delivery`, a separate process. The API decides what to send; delivery decides how to get it there.
8
9	Delivery happens in two stages:
10
11	1. Fan-out, in the API.
12	2. **Delivery, in `apps/delivery
13
14	A job is a finished message:
15
16	```ts
17	type DeliveryJob
18	\| { kind
19	\| { kind
20	```
21
22	`idempotency_key` is unique, so a fan-out that runs twice enqueues once. A webhook body is frozen at fan-out, so a redelivery sends the same bytes.
23
24	The queue is the `delivery_job` table. A worker claims due jobs with `FOR UPDATE SKIP LOCKED`, sets a lease (
25
26	Failures retry with exponential backoff and full jitter, capped, until the job is marked `dead`. A webhook is signed at send time, with HMAC-SHA256 over `timestamp.body`, so a late retry carries a fresh timestamp. The destination is checked for private, loopback, link-local and metadata addresses on the IP actually dialed, and redirects are not followed.
27
28	The service is written in Go with `pgx`. Its tables and migrations stay in `packages/db`; the service never migrates. The job shape is declared in TypeScript and mirrored as a Go struct, and one shared JSON fixture is decoded by tests on both sides.
29
30	## Why
31
32	Delivery depends on things outside our control: mail relays, and customer endpoints that are slow, down or hostile. Isolating it means a flood of retries or a hanging endpoint cannot take memory, connections or event-loop time from the API, and it can be deployed, scaled and debugged alone.
33
34	Fan-out stays in the API because it needs the domain: permissions, subscriptions, templates. Doing it inside the request's transaction would make a comment on a repository with thousands of watchers write thousands of rows while the user waits, so it runs after the commit, from the outbox.
35
36	Rejected:
37
38	- Redis pub/sub. Messages published while no subscriber is listening are lost, and publishing after the database commit reintroduces the dual write the outbox exists to prevent. Redis Streams keeps messages but not the atomicity, and adds infrastructure. Postgres already holds the data and the transaction.
39	- Holding the claim's transaction across the send, as
40	- Node for the service. A Node process idles at several times the memory of a Go one, and the host bills for it. Go's
41
42	## Consequences
43
44	- Delivery is at least once. Receivers deduplicate on the `X-Ghost-Delivery` header; emails can repeat if a worker dies after sending and before recording it.
45	- Ordering is not guaranteed, even per endpoint. Per-endpoint order would let one failing delivery hold back every later one.
46	- The repository has a second language, and the job shape is declared twice. The shared fixture is what keeps the two in step.
47	- The service must be able to read webhook secrets, so they are stored encrypted, not hashed.
48	- The queue's throughput is Postgres's. Past a few thousand jobs a second sustained, a dedicated queue fed from the outbox would replace the table; the contract would not change.
49	- Attempts accumulate until a pruning job exists.