Ingest documents (unstructured)

Unstructured documents (PDF, DOCX, email, audio) are ingested into a per-tenant Lance dataset so the engine’s RETRIEVE(...) form can serve them. Documents do not use Meltano/Singer (that path is tabular-only) — the unstructured-runner is a separate runner that reuses the same connector model, S3 conventions and in-cell Dagster (unstructured_sync_job).

Two stages, one image

extract   source → Tika/unstructured (+ whisper for audio) → PII mask → chunk
          → s3://tenant-<id>/landing-docs/<connId>/*.jsonl   (text + metadata, NO vectors)
load      landing JSONL → embed → atomic per-doc upsert → Lance
          → vector + scalar indexes → completion callback

Splitting the expensive extraction from the cheap embed/load means an embedding-model or ACL change re-runs only load against the landing checkpoint — you don’t re-transcribe an audio archive to change embedders.

Sources

Bytes come from pluggable source providers: local/S3/Azure/GCS object stores, Google Drive, Gmail, Microsoft 365 mail and calendars. Non-sensitive settings go to a ConfigMap and credentials to a Secret (fail-closed); the Dagster op mounts both into the Job. The external-side setup for the SaaS sources is under Configure your sources.

Text-ish formats, PDF (text-based), DOCX, PPTX and email work with the default image; audio/video transcription needs the image built with whisper enabled. Unsupported formats are skipped cleanly and per-document errors never abort the batch.

The embedder is an external endpoint

The embedder that vectorizes passages and queries is consumed as an endpoint the cell points at — it is never an in-cluster workload, so a plain helm install (or a kind cluster) never drags in a GPU stack.

Runtime	Use
hash	Dev/smoke default — deterministic, no model
openai-compatible endpoint	A gateway or any endpoint you provide
ollama	A native host process, reached through a scoped egress NetworkPolicy

Why never in-cluster

Running the embedder inside the tenant’s Kubernetes puts fleet-scale GPU compute inside every cell and makes small clusters unbootable. Where a node has a GPU, ollama runs natively on the host (systemd), outside the cluster; managed tenants get an EU-hosted gateway endpoint unless they bring their own. The runtime is a Helm value (embedder.*), not a redeploy.

Privacy at ingest

• PII masking happens before landing: without declared redaction versions, body text and configured metadata fields are masked and only that masked representation is loaded.
• With named redaction versions, the runner persists one separately-embedded Lance table per declared profile. Versioned ingestion requires a facet acl_field and a non-empty default ACL — extract and load refuse to write if either is missing.
• A profile or detector change re-materializes affected documents; if any write fails, the runner removes the document from every materialization (base first) rather than leaving an older, less-redacted generation readable.

Consume it

Once loaded, passages are served only through the governed RETRIEVE(...) form of POST /osl/query — ACL’d, redacted server-side, with citations preserved.

Deeper reference

• cells/unstructured-runner/README.md — stages, source providers, formats, environment.
• docs/osl/lance-extension.md — the UnstructuredFacet / JointEntity model this pipeline feeds.