Observability — OpenTelemetry traces + Loki logs

Card-Core exports distributed traces and structured application logs over OTLP/gRPC to the existing OpenTelemetry Collector. The app never talks to Loki or Tempo directly; the collector fans out. Nothing here runs a collector — one variable points the app at yours.

card-core ──OTLP/gRPC :4317──► OTel Collector ──► Loki   (logs, queried with LogQL)
                                              └─► Tempo  (traces)
      └─ the same JSON lines also go to stdout (Promtail/Alloy can scrape them)

Everything is off until a collector endpoint is configured. With no OTEL_* endpoint in the environment, src/main.ts never loads the OpenTelemetry SDK at all: no gRPC connection is attempted and Nest keeps its pretty console logger. That keeps a plain npm run start:dev checkout and the test suite silent.

What you get

• Traces for HTTP (in and out), Express, Postgres (pg), gRPC, Redis and the rest of @opentelemetry/auto-instrumentations-node. /health requests produce no server span — infra probes poll it constantly and would drown the backend. fs instrumentation is off for the same reason.
• Logs as one JSON object per line, every line stamped with the active traceId / spanId, so a log line links back to its trace in Grafana. The same record goes to stdout and over OTLP — docker logs / kubectl logs stay useful, and the stdout copy is the fallback when the collector is down.
• Resource attributes on both signals: service.name (the Loki service_name label), service.version, deployment.environment (NODE_ENV).

Files

Path	Role
src/tracing/env.ts	endpoint/enablement resolution shared by both signals
src/tracing/tracer.ts	NodeSDK bootstrap (traces, auto-instrumentation)
src/tracing/nest-logger.ts	OtelNestLogger — a Nest LoggerService that also exports OTLP logs
src/tracing/observability.module.ts	flushes both pipelines during app.close()
src/otel.ts	the bootstrap main.ts requires before anything else

Load order is load-bearing

src/main.ts starts with require("dotenv").config(), then require("./otel") — both above every import. Auto-instrumentation patches http, pg and friends as those modules load, and cannot retroactively wrap something Nest has already imported. dotenv comes first because the Nest ConfigModule only reads .env once AppModule is constructed, far too late to configure an exporter. Do not turn those requires into imports.

Configuration

Full commentary lives in .env.example; the short version:

Variable	Effect
OTEL_EXPORTER_OTLP_ENDPOINT	collector for both signals, e.g. http://otel-collector.observability.svc.cluster.local:4317. A bare host:port gets http:// prefixed
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT / ..._LOGS_ENDPOINT	per-signal override
OPENTELEMETRY_GRPC_ORIGIN	legacy alias for both, lowest precedence
OTEL_SDK_DISABLED=true	kills both signals. NODE_ENV=local does the same
OTEL_TRACING_ENABLED / OTEL_LOGS_ENABLED	force one signal on (falls back to http://localhost:4317) or off
OTEL_SERVICE_NAME	service.name; defaults to card-core
OTEL_SERVICE_VERSION	service.version; set it to the image tag or git sha in CI
LOG_LEVEL	unchanged meaning — the level floor, applied before anything is written or exported

That is the whole setup: OTEL_EXPORTER_OTLP_ENDPOINT per environment, never in code.

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://otel-collector.observability.svc.cluster.local:4317
  - name: NODE_ENV
    value: production

Verifying

1. Startup prints [tracing] OpenTelemetry SDK started { serviceName: 'card-core' }. OpenTelemetry is disabled (no SDK started) means no endpoint resolved, or NODE_ENV=local, or OTEL_SDK_DISABLED=true.
2. Every log line is single-line JSON. A request-scoped line must carry a non-empty traceId; an empty one means the load order above got broken. Lines from crons and boot legitimately have none — no request, no span.
3. LogQL in Grafana:

{service_name="card-core"}                                   # everything
{service_name="card-core", level="error"}                    # errors only
{service_name="card-core"} | json | context="PaymentsService" # one Nest provider
{service_name="card-core"} | json | traceId="4bf92f35..."     # one request

4. Tempo: search by service name (card-core), open a span, follow the log↔trace link.
5. Failure mode: diag is set to ERROR, so exporter connection failures print to stderr. Silence plus no data in Grafana usually means the wrong port (4317 is gRPC, 4318 is HTTP) or https:// against a plaintext collector. A dead collector never fails a request or blocks shutdown — export errors are swallowed and the stdout copy of every line survives.

Loki notes

service.name becomes the service_name label; level, context, traceId, spanId arrive as structured metadata, which needs limits_config.allow_structured_metadata: true on Loki (the default in 3.x). Only promote low-cardinality keys to real labels — never traceId.