Skip to main content

nexus-gitops

Everything needed to deploy Nexus on Kubernetes. This repo is named nexus-gitops (not nexus-helm) because it holds more than Helm: the Argo CD app-of-apps, per-environment values, the infra manifests and the secret-bootstrap script.

Structure (as of 2026-09-05)​

nexus-gitops/
  argocd/
    root.yaml                  # app-of-apps: applies everything under argocd/apps
    apps/
      project.yaml             # AppProject "nexus" (destinations + allowed chart repos)
      infra.yaml               # ns + MongoDB/NATS/Qdrant from infra/
      nexus.yaml               # charts/nexus with environments/prod/values.yaml
      redis.yaml               # nexus-redis (CloudPirates OCI chart) — Apalis queue
      langfuse.yaml            # Langfuse chart 1.5.35; no secrets in git (existingSecret refs)
  charts/nexus/
    Chart.yaml · values.yaml
    templates/
      core-, worker-, telegram-, ui-, docs-, oracle-deployment.yaml
      solana-oracle-, solana-exec-deployment.yaml
      executor-, executor-live-deployment.yaml   # perp executors — disabled (retired 2026-08-28)
      mobhost-statefulset.yaml · mob-configmaps.yaml
      metrics.yaml             # metrics Services + ServiceMonitor + Grafana dashboards ConfigMap
      alerts.yaml              # PrometheusRule nexus-platform-alerts (5 groups)
      networkpolicy.yaml · agent-networkpolicy.yaml
      agent-credentials.yaml · planner-credentials.yaml
      ingress.yaml · rbac.yaml · serviceaccount.yaml · secrets.yaml (doc only)
    files/
      dashboards/nexus-platform.json · nexus-treasury.json
      mobs/accum-desk/         # the live desk
      mobs/trading-desk/, mobs/dream-mob/   # retired; dead config, still shipped
  environments/{dev,staging,prod}/values.yaml
  infra/
    README.md · bootstrap-secrets.sh · namespaces.yaml
    mongodb/statefulset.yaml · mongodb/backup-cronjob.yaml
    nats/statefulset.yaml · qdrant/statefulset.yaml

There is no Flux, no k8s/ directory and no argocd/nexus-<env>.yaml; the Argo CD apps are the delivery mechanism, and CI (nexus-platform, nexus-ui, nexus-docs, nexus-solana) commits image tags straight into environments/prod/values.yaml on every push to main — Argo then deploys.

Role​

  • The nexus Helm chart: core, worker, telegram, ui, docs, oracle, the accumulation mob host, the two Solana venue services, and NetworkPolicies.
  • Argo CD app-of-apps (argocd/root.yaml) for the chart, the infra manifests, Redis and Langfuse.
  • Per-environment values (dev / staging / prod). Prod is the live deployment.
  • Infra manifests: namespaces, MongoDB 7, NATS + JetStream, Qdrant — all single-replica on microk8s-hostpath volumes.
  • A daily mongodump CronJob (03:15, 14-day retention) for nexus-mongodb and the HARVEST desk database — to an on-node 40 Gi PVC only; there is no off-site copy and no restore drill yet (audit H9 / 2026-09-05 O1): a local backup on the same node and disk, not disaster recovery.
  • bootstrap-secrets.sh — creates every Kubernetes Secret from environment variables (sourced from the git-ignored .secrets).

Not in this repo (manual Helm releases, see cluster-helm-values/README.md in the workspace): kube-prometheus-stack (kps), Keycloak, Harbor, the shared mongodb / postgresql releases. Langfuse was one of these until 2026-09-05.

Secrets​

Created out-of-band by infra/bootstrap-secrets.sh; never in git.

NamespaceSecretKeys
nexus-mongodbnexus-mongodb-authMONGO_INITDB_ROOT_USERNAME, MONGO_INITDB_ROOT_PASSWORD
nexus-qdrantnexus-qdrant-authQDRANT_API_KEY
nexusnexus-redis-authredis-password (read by the Redis chart)
nexusnexus-appNEXUS_MONGODB_URL, NEXUS_NATS_URL, NEXUS_REDIS_URL, QDRANT_URL, QDRANT_API_KEY, JWT_SIGNING_KEY; optional OPENAI_API_KEY, TELEGRAM_BOT_TOKEN / _ALLOWED_USERS / _BROADCAST_CHATS, NEXUS_LANGFUSE_*, CRYPTOPANIC_*, SANTIMENT_API_KEY, WHALE_ALERT_API_KEY
nexusnexus-uiNEXUS_CORE_URL, NEXUS_API_KEY, NEXUS_UI_API_KEY, CEX_MANAGER_API_KEY, SOLANA_EXEC_API_KEY, AUTH_SECRET, KEYCLOAK_ISSUER / _CLIENT_ID / _CLIENT_SECRET, APP_BASE_URL, CANONICAL_HOST (derived from APP_BASE_URL, 2026-09-06 — below)
nexusnexus-exec-keyexec.json — the Solana EXEC signer. Holds real funds. Written only from EXEC_KEY_FILE, never generated.
nexusnexus-exec-authEXEC_API_KEYS — the signer's caller-key map (audit 2026-09-05 S1; enforced since 29c983a). Callers get their own SOLANA_EXEC_API_KEY in nexus-app / nexus-ui.
nexus-agentsnexus-agentGIT_TOKEN, GITHUB_TOKEN, GIT_USERNAME (envFrom into runner pods). No LLM API keys: prod agents run on the mounted subscription CLI logins.
nexus, nexus-agentsregcredpull secret for registry.bitview.club/nexus

bootstrap-secrets.sh semantics (rewritten 2026-09-05 after it was found to delete live keys): optional keys are written only when non-empty, so an unset variable never blanks a live value; before each write it diffs the intended key set against the live Secret and aborts if kubectl apply's three-way merge would drop a key (FORCE=1 overrides).

Scoped RBAC​

rbac.yaml gives nexus-core / nexus-worker a Role in nexus-agents limited to Jobs, Pods and pod logs:

verbs: [create, get, list, watch, delete]
resources: [jobs, pods, pods/log]

It is enforced: the cluster authorizer is Node,RBAC since 2026-09-05 (audit C1 closed). See Security & permissions.

Secrets added on 2026-09-05 (all created by infra/bootstrap-secrets.sh, none in git): nexus-nats-auth (NATS_PASSWORD_NEXUS, NATS_PASSWORD_RUNNER) in nexus-nats; nexus-app gained NEXUS_AGENT_NATS_URL (runner credentials), DESK_MONGO_URI / HARVEST_MONGO_URI (nexus_rw on the host mongod), CEX_MANAGER_API_KEY and NEXUS_API_KEY; nexus-mongodb-auth gained HARVEST_MONGO_URI (backup_ro) for the nightly dump. The NATS cutover runbook is infra/nats/AUTH-CUTOVER.md.

Observability wiring​

templates/metrics.yaml (values monitoring.serviceMonitor / monitoring.dashboard): metrics-only Services for worker / telegram / oracle, a metrics port on the core and mob-host Services, one ServiceMonitor selecting nexus.dev/metrics=true, and two Grafana dashboards delivered in one sidecar ConfigMap nexus-grafana-dashboard (grafana_dashboard=1, folder Nexus): Nexus Platform (files/dashboards/nexus-platform.json) and, since 05c3ffd (2026-09-05), Nexus Treasury (files/dashboards/nexus-treasury.json, uid nexus-treasury, 30 panels — NAV vs benchmarks, components, exposure, attribution, flows & costs, marks, execution safety; the component panels read nav_component since a5912c7 — component is the registry's global label and broke the worker scrape, see Operations → Metric labels).

templates/alerts.yaml (value monitoring.alerts: true): the PrometheusRule nexus-platform-alerts, five groups — nexus.backups, nexus.signer, nexus.auth, nexus.agents (since 9f6aa48) and nexus.financial (15 rules, since 05c3ffd). The financial thresholds are values monitoring.financial.{drawdownWarnFrac,drawdownCritFrac,dailyLossWarnFrac} (0.10 / 0.20 / 0.05). The group and its metric contract are described under Operations → Alerting and Measuring success → As built; the worker exporter it reads is deployed since nexus-platform ac38aa1 (2026-09-05 17:12 UTC).

Worker rollout and the budget gate (worker.*)​

Three things changed on the worker Deployment on 2026-09-05:

  • strategy: Recreate (441291a). The worker is the single Apalis consumer; with RollingUpdate the outgoing pod and the new one overlap, the old binary can fetch a job kind it cannot decode, and the job stays in-flight under a shared consumer name whose heartbeat the new pod keeps alive (the first treasury-nav run was lost that way at 17:12 UTC). One consumer version at a time; the cost is a short gap in sweeps per roll.

  • POD_NAME from the downward API (078055c), read by nexus-platform 9063b8d to name the consumer nexus-jobs-<POD_NAME> — a dead pod's in-flight jobs are reclaimed as orphans. Runbook: Operations → Apalis queue.

  • worker.budget (d848adf) — the portfolio budget gate (audit F6 / H5):

    worker:
      budget:
        mode: enforce            # chart default; prod: log
        maxSolExposureFrac: ""   # code default 0.60 → ACCUM_LIMIT_MAX_SOL_EXPOSURE_FRAC
        maxBtcExposureFrac: ""   # code default 0.40 → ACCUM_LIMIT_MAX_BTC_EXPOSURE_FRAC
        minLiquidReserveUsdc: "" # code default 5000 → ACCUM_LIMIT_MIN_LIQUID_RESERVE_USDC

    mode becomes ACCUM_LIMIT_MODE; the three limits are emitted only when set. environments/prod/values.yaml is worker: { budget: { mode: log } } — observe first: the gate evaluates, reserves and counts would_block: but blocks nothing. The remaining ACCUM_LIMIT_* limits are plain worker env (not templated). The flip and what to watch are under Operations → Portfolio budget gate; the limits under HARVEST §5.

Reserve floors (worker.reserves)​

Added 2026-09-06 by 2c8d76a. The H4 reserve floors are code-owned — the decision officer may ask for more liquidity, never less — and the chart now exposes them so the owner can move a floor without a code change:

worker:
  reserves:
    minExecGasSol: ""   # empty = code default 0.2 SOL  → ACCUM_MIN_EXEC_GAS_SOL
    minFloatUsdc: ""    # empty = code default 5000 USDC → ACCUM_MIN_FLOAT_USDC

Each key is emitted only when non-empty (charts/nexus/templates/worker-deployment.yaml). Prod (environments/prod/values.yaml) sets reserves: {minFloatUsdc: "750"} — two owner decisions, in order. 2026-09-06 (2c8d76a) took it 5 000 → 2 000: the $5 000 floor was sized for the now-retired cbBTC arm, and with no armed BTC buy, operations plus a confirm-window hedge need $2 000 while a rung or arm funds itself with a lending_withdraw under the owner lending policy. 2026-09-09 (f9d8965) took it 2 000 → 750, when the standing stables reserve was removed altogether — "i don t need reserve because for me kamino is the reserve and it s easy to withdraw or to cancel orders to buy the ladder" — so the float covers operations only and anything above a named use counts as sleeping money in the cycle prompts (HARVEST §4). minExecGasSol is left empty everywhere, so the gas floor stays the 0.2 SOL code default (the desk's own prompt asks for 0.5 and is clamped up, never down).

The floor is not only a decision-officer clamp: since nexus-platform 3ce6c78 the executor also caps a lending_supply at exec USDC − float floor and skips it entirely when nothing sits above the floor — the 02:30 UTC incident where the officer sized 2 937 USDC from a float the floor had already clamped back up to 5 000 (HARVEST §5).

Lending hygiene knobs and the desk prompt (worker.hygiene)​

Two commits carry the owner lending policy of 2026-09-05 (yield-first, rare moves — HARVEST §5):

  • worker.hygiene (d5318ad, ≈ 21:11 UTC; Argo rolled the worker, live ≈ 21:15 UTC). The chart values are the policy numbers — prod does not override them:

    worker:
      hygiene:
        utilRecall: "0.95"             # → ACCUM_UTIL_RECALL
        utilRecallHard: "0.985"        # → ACCUM_UTIL_RECALL_HARD
        utilRecallConfirmReads: "2"    # → ACCUM_UTIL_RECALL_CONFIRM_READS
        utilRepark: "0.88"             # → ACCUM_UTIL_REPARK
        liqFloorX: "10"                # → ACCUM_LENDING_LIQ_FLOOR_X
        rotateMinSpread: "0.02"        # → ACCUM_ROTATE_MIN_SPREAD
        rotateConfirmSweeps: "48"      # → ACCUM_ROTATE_CONFIRM_SWEEPS
        rotateCooldownSecs: "604800"   # → ACCUM_ROTATE_COOLDOWN_SECS
        minHoldSecs: "172800"          # → ACCUM_LENDING_MIN_HOLD_SECS

    Each key is emitted only when set (empty = code default, the same number). The running worker (7d6ed59) reads ACCUM_UTIL_RECALL, ACCUM_UTIL_REPARK and ACCUM_ROTATE_MIN_SPREAD on a single read — so the recall line moved from 0.92 to 0.95 and the rotation spread from 1.5 pp to 2 pp immediately; the confirmation, cooldown and hold knobs took effect with nexus-platform fdd7912, rolled 21:54 UTC. Meaning of each knob: Operations → Lending hygiene.

  • The desk prompt (3f90adb, the OWNER LENDING POLICY block in the decision officer's message, files/mobs/accum-desk/mob.definition.json): keep the JLP reserve; no same-venue lending round trips for APY or utilisation (reserve moves are the reserve manager's job, rate-limited by the values above); the only lending_withdraw the officer may emit funds a confirmed rung / approved order or answers the liquidity floor (available liquidity below 10× our position, or utilisation ≥ 95 % on two consecutive reads, ≥ 98.5 % once); 85–95 % utilisation is the accepted band — state the read and the liquidity multiple, and hold. A mob-definition change needs an accum-host restart: the host loads the pack from the ConfigMap at start, and a files-only commit does not roll the StatefulSet (an image bump does). This one was restarted by hand ≈ 21:15 UTC.

Trap, resolved 2026-09-06: environments/prod/values.yaml used to carry two top-level worker: keys — an inline map near the top (replicas, a stale tag, fallbackEarliest) and a block at the end that CI bumped (budget, tag). Helm's YAML loader is non-strict and keeps the last one, so the top line was dead and fallbackEarliest: "2026-10-01" (the §5 gate the operator moved up on 2026-08-31) silently fell back to the template default 2027-06-01. As of gitops 91232c7 there is one worker: line carrying replicas, tag, fallbackEarliest, budget and reserves together. The operator restored ACCUM_FALLBACK_EARLIEST on the running pod on 2026-09-05 ≈ 22:0x UTC; the lesson stands — a duplicated top-level key in a values file is silent, so verify the env on the running Deployment after any values edit that touches a merged block.

Ingress and the canonical host (ingress.host / ingress.aliases)​

charts/nexus/templates/ingress.yaml serves ingress.host plus every ingress.aliases entry; prod is host: nexusapp.dev, aliases: [www.nexusapp.dev], certManagerIssuer: letsencrypt-prod (docs.nexusapp.dev is on the TLS list only). Since gitops 91232c7 every alias routes normally to nexus-ui:80 and the app owns the canonical redirect — the separate nexus-alias-redirect Ingress is gone.

Two failed attempts are worth keeping, because both failure modes are silent-ish:

  • 329bab4 added a nginx.ingress.kubernetes.io/permanent-redirect Ingress whose backend named port 3000 while the Service listens on 80: the backend was unresolvable and nginx answered a plain 404 (fixed in d41d779).
  • ingress-nginx then rejected the annotation value outright because it contained $request_uri — variables are not allowed there.

So the redirect moved into nexus-ui. bootstrap-secrets.sh adds CANONICAL_HOST to the nexus-ui Secret, derived from APP_BASE_URL (scheme and path stripped), right beside APP_BASE_URL itself; it is a required key, so an empty derivation fails the bootstrap rather than shipping a blank. The Secret reaches the container through envFrom on the UI Deployment. Note the caveat the app page records: Next.js inlines process.env into the Edge middleware bundle at build time, so the UI's rule derives the target from the request's own Host header and treats CANONICAL_HOST as an override when the runtime exposes it (nexus-ui → Canonical host).

Known stale comment: ingress.yaml still attributes the 308 to "nexus-ui next.config" — it was written against ad31a86 and the logic now lives in src/middleware.ts.

Signer caller auth (solanaExec.auth)​

solanaExec.auth: { mode, secret } — mode is log (decisions counted only) or enforce (401/403); secret names the Secret holding EXEC_API_KEYS (nexus-exec-auth, created by bootstrap-secrets.sh). Prod (environments/prod/values.yaml) is mode: enforce since 29c983a (2026-09-05 16:27 UTC) after 90 min of log mode with 0 caller denials (audit S1). Callers receive their SOLANA_EXEC_API_KEY from the nexus-app / nexus-ui Secrets. Rotation: edit .secrets → bootstrap-secrets.sh → roll the exec and the callers.

Dependencies​

  • MongoDB · NATS JetStream · Qdrant · Redis · ingress-nginx (public) · cert-manager (letsencrypt-prod) · kube-prometheus-stack CRDs (ServiceMonitor) · Calico for NetworkPolicy enforcement · Keycloak (identity.bitview.club, also served as identity.nexusapp.dev; the realm issuer switch is pending) for the UI login · Harbor (registry.bitview.club) for images.