Skip to main content

Logging: platform log page

Nexus aggregates logs from every component — nexus-core, nexus-worker, nexus-telegram, and the per-run nexus-agent-runner pods — into one queryable place: the Admin UI Logs page. Use it to confirm agents connect, NATS and MongoDB are reachable, and LLM calls aren't erroring.

Why this design (NATS and MongoDB)​

The two systems play different roles:

ConcernChoiceWhy
TransportNATS (plain pub/sub)Every component already has (or can get) a NATS connection — including the confined, ephemeral agent pods, which have no MongoDB credentials and shouldn't. Decouples emitters from storage.
Storage / queryMongoDB capped collectionA capped collection is fixed-size, insertion-ordered, and self-trimming — ideal for logs. It's queryable (component / level / text / time) to back the UI page.
Node-level baselinestdout (JSON)kubectl logs and any cluster log stack keep working unchanged.

Logs are published on the plain (core) NATS subject logs.<component>, deliberately outside the nexus.> JetStream stream, so high-volume log traffic never bloats the durable event log.

Pipeline​

tracing::info!/warn!/error!  (any component)
        │  CaptureLayer  (nexus-observability)
        ▼
  in-process channel  ──pump──▶  NATS  logs.<component>   (also → stdout JSON)
                                          │
                            log sink (nexus-worker) subscribes logs.>
                                          ▼
                          MongoDB capped collection  `logs`
                                          ▲
                       GET /v1/logs (nexus-core)  ◀── Admin UI “Logs” page (polls)
  • Capture — nexus_observability::init("<component>") installs a tracing subscriber with a JSON stdout layer plus a capture layer that turns each event (severity ≥ INFO by default) into a LogRecord on an in-process channel.
  • Ship — long-lived services spawn pump_logs(...) to publish records to logs.<component>. The short-lived agent runner instead drains its buffer to NATS right before exit, so agent connectivity / LLM-call logs aren't lost.
  • Sink — the worker runs the single logs.> subscriber and inserts each record into the capped logs collection.
  • Read — GET /v1/logs?component=&level=&search=&since= returns newest-first (capped $natural order). The UI polls every few seconds while Live is on.

Configuration​

Env varDefaultPurpose
NEXUS_LOGinfostdout EnvFilter (e.g. nexus_worker=debug,info).
NEXUS_LOG_CAPTUREinfoMinimum severity captured for the page (debug/info/warn/error).
NEXUS_LOGS_CAP_BYTES268435456 (256 MiB)Capped collection size.
NEXUS_LOGS_CAP_DOCS1000000Capped collection max documents.

All components need NEXUS_NATS_URL set to ship logs; the worker additionally writes to MongoDB. If NATS is unreachable a component still logs to stdout — it just won't appear on the page.

Troubleshooting​

  • Page is empty — check the worker's log sink subscribed (log sink: subscribe failed on stderr means NATS is down) and that the capped logs collection exists.
  • An agent's logs are missing — agent pods must have NEXUS_NATS_URL and egress to NATS (allowed by the agent NetworkPolicy). Run-level outcomes also appear on the Runs page regardless.
  • No logs anywhere — nexus_observability::init(...) must run at startup; it installs the only subscriber.