| 1 | # Observability |
| 2 | |
| 3 | Both apps speak OpenTelemetry. `compose.yaml` runs `grafana/otel-lgtm`, one container holding the OTLP collector, Prometheus, Tempo, Pyroscope, Loki and Grafana. |
| 4 | |
| 5 | - Grafana: http://localhost:3100, user `GRAFANA_USER`, password `GRAFANA_PASSWORD`. Anonymous access and sign-up are off. |
| 6 | - OTLP: `http://lgtm:4318` inside the compose network, `http://localhost:4318` from the host |
| 7 | - Pyroscope: `http://lgtm:4040`, `http://localhost:4040` from the host |
| 8 | |
| 9 | `GRAFANA_PASSWORD` has no default: compose refuses to start without it. Grafana only reads it when it creates the admin user, so changing it later needs the reset below, or a new volume. |
| 10 | |
| 11 | ```sh |
| 12 | docker compose exec lgtm |
| 13 | ``` |
| 14 | |
| 15 | Five wrong passwords in five minutes locks the account for five minutes, and the message stays "Invalid username or password" throughout. Wait it out rather than trying a sixth. |
| 16 | |
| 17 | Nothing is exported until `OTEL_EXPORTER_OTLP_ENDPOINT` is set, and nothing is profiled until `PYROSCOPE_SERVER_ADDRESS` is: without a collector listening, the exporters retry and bury the logs. |
| 18 | |
| 19 | ## What is collected |
| 20 | |
| 21 | `apps/api/src/instrumentation.ts` starts the Node SDK before Nest loads, so the auto-instrumentations patch `http`, ` |
| 22 | |
| 23 | `apps/web/src/instrumentation.ts` is Next.js's own instrumentation hook, registering the same exporter for server-side renders, route handlers and outbound `fetch`. |
| 24 | |
| 25 | Tempo's metrics generator turns every span into `traces_spanmetrics_*`, so latency percentiles and error rates exist for both services without either of them emitting metrics of their own. |
| 26 | |
| 27 | ## Application metrics |
| 28 | |
| 29 | `apps/api/src/lib/metrics.ts` holds the counters. They are recorded in `PushTransactionService.commitPush`, the single point every accepted push passes through, so a push counts once and a retried CAS attempt does not count twice. |
| 30 | |
| 31 | | Metric | Prometheus name | |
| 32 | | --- | --- | |
| 33 | | `ghost.pushes` | `ghost_pushes_total |
| 34 | | `ghost.push.bytes` | `ghost_push_bytes_total |
| 35 | | `ghost.ref.updates` | `ghost_ref_updates_total |
| 36 | |
| 37 | `AppStatsService` observes instance totals once per collection: `ghost_repositories` (by visibility), ` |
| 38 | |
| 39 | A meter taken before the SDK starts is a no-op for the life of the process, which is why `instrumentation.ts` is the first import in `main.ts`. In tests there is no SDK, so the instruments record nothing and nothing has to be stubbed. |
| 40 | |
| 41 | ## Profiling |
| 42 | |
| 43 | `@pyroscope/nodejs` pushes wall, CPU and heap profiles from the API. It is a push protocol of its own, not OTLP, hence its own switch. Profiles flush once a minute, so a process that lives less than that uploads nothing. |
| 44 | |
| 45 | Only the API is profiled. `@pyroscope/nodejs` ships a native module that the Next.js standalone build would have to carry, and the web server's own cost already shows up in its traces. |
| 46 | |
| 47 | ## Dashboard |
| 48 | |
| 49 | `grafana/ghost.json` is provisioned as the home dashboard, `grafana/dashboards.yaml` is the provider that loads it. Request rate, error rate, p50/p90/p99, slowest endpoints, outbound calls, traces over 500ms, the API's CPU and wall flame graphs, then an application row: totals, commits over time, repositories by visibility, pushed bytes and pushes. |
| 50 | |
| 51 | A frame that is wide in the wall graph but narrow in the CPU one is waiting on postgres, S3 or git rather than computing. |
| 52 | |
| 53 | The browser is not instrumented — these are server-side numbers. Real user monitoring needs a Faro receiver, which this container does not run. |