| 1 | # 0032 — GitHub imports run outside the API, which owns the retries |
| 2 | |
| 3 | **Status:** adopted |
| 4 | |
| 5 | ## Decision |
| 6 | |
| 7 | Importing a GitHub repository is done by `apps/importer`, a separate Bun service. The API decides when an import runs and whether it is retried; the importer only does the work. |
| 8 | |
| 9 | 1. **Start, in the API.** |
| 10 | 2. **Dispatch, in the API.** |
| 11 | 3. **Work, in the importer.** It fetches branches, tags and |
| 12 | |
| 13 | Every importer callback carries the attempt's `claim_token` and renews a two-minute lease; the importer also sends a heartbeat every 30 seconds. The API retries with exponential backoff and full jitter, up to six attempts, when the importer refuses a job, reports a retryable failure, or lets its lease lapse. A callback for any attempt but the current one gets 409 and the importer stops. A new attempt revokes the previous attempt's key. |
| 14 | |
| 15 | Imported rows are authored by `ghost-importer`, a user seeded by migration whose username contains a hyphen, which Better Auth's username validator refuses, so no person can claim the name. The GitHub author is credited at the top of each body. Issue and pull request numbers are kept, so `#123` in imported text still resolves. Writes are upserts keyed by tag name and issue number, so a retried attempt overwrites rather than duplicates; opening an issue or pull request is refused while an import is unfinished, since the import owns those numbers. |
| 16 | |
| 17 | ## Why |
| 18 | |
| 19 | A large repository takes minutes to clone and push. Doing that inside the API would hold its memory, disk and event loop, and a deploy would kill the work silently. A separate service can be scaled, limited and restarted on its own. |
| 20 | |
| 21 | The push goes through the same HTTP path as any user's push, so the write-ahead log, code search indexing and contribution counts behave exactly as they would for a push from a laptop. That needs a credential, hence the short-lived API key. |
| 22 | |
| 23 | Domain writes stay in the API, which already owns issue numbering, comment counts and the pull request invariants; the importer never touches the database. |
| 24 | |
| 25 | Rejected: |
| 26 | |
| 27 | - **The importer claims jobs from Postgres itself**, as |
| 28 | - **Calling `GitService.receivePack |
| 29 | - **A dedicated bot user named `ghost |
| 30 | |
| 31 | ## Consequences |
| 32 | |
| 33 | - Imports are optional: without `IMPORTER_URL`, `IMPORTER_SECRET`, |
| 34 | - The API key gives the importer the requester's full access until the attempt ends or the key expires. Deleting the repository mid-import leaves the key to expire on its own. |
| 35 | - `refs/pull/*` refs are pushed and so are advertised to clones, as on GitHub. |
| 36 | - A user pushing to the repository during an import can make the import's non-forced push fail; that attempt is retried and fails for good if the conflict stays. |
| 37 | - Not imported: release assets, LFS objects, line comments on pull requests, reactions, assignees and milestones. |