Orchestration — in-cell Dagster

OpenDome’s orchestrator is Dagster, run per cell: every tenant runs its own Dagster — a daemon plus a code-location gRPC server, no webserver — inside its own namespace. It is deployed by charts/tenant-mgmt-api when you set:

helm install mgmt charts/tenant-mgmt-api \
  -n tenant-demo \
  --set tenantId=demo \
  --set runLauncher.mode=dagster \
  --set dagster.enabled=true

Dagster is a toggle inside the management chart, not a separate chart or a shared multi-tenant deployment. The OpenDome-operated product runs the exact same per-cell orchestrator — a self-host is simply one cell.

Prerequisites

Ingest lands synced data into Iceberg via the cell’s Trino/Nessie, so install charts/lakehouse before driving a sync. The CNPG operator must be installed (the dagster database is created via a CNPG Database CR by default — dagster.db.manageCnpg=true; set it false and pre-create the DB if your install identity lacks that permission).

The five sync-family jobs

The cell code location serves one tenant (TENANT_ID) and loads five jobs:

Job	What it does
connector_sync_job	connector-runner K8s Job (Meltano tap → S3 landing)
ingest_landing_job	ingest K8s Job (landing Parquet → Iceberg raw)
dbt_run_job	dbt-runner K8s Job (raw → curated)
unstructured_sync_job	unstructured-runner K8s Job (documents → Lance)
identity_resolve_job	identity-runner K8s Job (cross-source entity resolution)

How runs fire

The cell never talks to Dagster directly:

• The cell only inserts runs rows as QUEUED. Launch-sensors atomically claim them (/runs/claim, callback-token authed — one launch per run), create the worker K8s Job, and self-report to /complete.
• Schedule-sensors evaluate connector crons and register due runs idempotently per cron window, plus a nightly dbt cron.
• Chaining sensors register the follow-up run when a step succeeds: ingest-after-sync (structured and unstructured), dbt-after-ingest, identity-after-dbt.
• All sensors default to RUNNING — there is no UI to enable them in the daemon-only topology.

No always-on Dagit

There is no Dagster webserver in the cell. Runs are observed through the management API and Kubernetes (Jobs, logs); the repository’s per-cell Dagster observability runbook covers debugging runs without a UI.

Configuration

The chart sets the environment on the daemon/code-location pods. The important ones: TENANT_ID, MGMT_NAMESPACE, CELL_MGMT_URL (the cell’s own tenant-mgmt-api), CELL_CALLBACK_TOKEN (must match the cell’s RUN_CALLBACK_TOKEN), and the worker Job images (CONNECTOR_RUNNER_IMAGE, DBT_RUNNER_IMAGE, INGEST_IMAGE, UNSTRUCTURED_RUNNER_IMAGE, IDENTITY_RUNNER_IMAGE) — all digest-pinned chart defaults.

Here it is the callbacks port

The orchestration reads the cell’s callbacks listener (callbacks.port, :8081 by default; the service port when callbacks.enabled=false). The same variable name in tenant-semantic-api and tenant-catalog-api is the management port (:8080), which serves the policy snapshot. They are separate listeners, so one shared value would send one of the two to a port that does not answer it — let the chart set each deployment.

CONTROL_PLANE_URL and CONTROL_PLANE_CALLBACK_TOKEN are the previous names of these two variables. The cell still reads them when the current ones are unset, so existing manifests keep working.

Run callbacks authenticate with the chart-aligned RUN_CALLBACK_TOKEN, not with ActorTokens — a deliberately separate internal channel from the bearer contract.

Deeper reference

• dagster/README.md — the code location, jobs and sensors.
• docs/per-cell-orchestration-dagster.md — the design (decisions D1–D5).
• docs/runbooks/per-cell-dagster-cutover.md — the cutover runbook.
• docs/pipeline.md — the landing → raw → curated conventions.