Operations

Frank is built to be operated as a set of clear runtime surfaces: API, UI, source worker, transform worker, Temporal workflows, Dagster assets, Iceberg tables, logs, traces, and run records.

Services

Service	Purpose
api	FastAPI application, route registration, pattern sync, schema library access, AI endpoints, and admin APIs.
ui	SvelteKit application for source, transform, pipeline, model, ontology, and settings workflows.
source-worker	Temporal worker for source discovery and extraction.
transform-worker	Temporal worker for transform lifecycle and reconciliation work.
worker	General Temporal worker for AI and platform workflows.
Dagster	Asset materialization, schedules, sensors, and pipeline execution visibility.
Temporal	Durable workflow execution for async source, transform, and orchestration jobs.
Iceberg REST + MinIO/S3	Lakehouse catalog and object storage.
Trino	Query engine for transform execution and previews.
Loki	Persistent log querying for run details.
OpenTelemetry collector	Trace export for API and worker paths.

Local stack

cd ../common-infra
docker-compose up -d

cd ../frank-low-code-pipeline
make up
make status

Common access points:

Surface	URL
API docs	http://localhost:8002/docs
UI	http://localhost:5175
Health	http://localhost:8002/health
Dagster	http://localhost:3000 or configured Dagster URL

Startup work

Production database migration is a separate, fail-closed release step. The API does not start until the one-shot migration service reaches the single Alembic head. Local development retains the image entrypoint migration behavior, but a non-zero Alembic result stops startup rather than becoming a warning.

After that gate, API startup performs platform initialization:

• Runs through FastAPI lifespan setup.
• Initializes Iceberg client.
• Ensures raw namespace.
• Initializes AI transformer services.
• Initializes FIWARE SDM registry and schema libraries.
• Syncs source patterns from JSON files into the database.
• Syncs SQL transforms and transform patterns from filesystem config.
• Registers transform event listeners for Dagster code location reloads.
• Initializes OpenTelemetry and Langfuse instrumentation.
• Registers the complete API router set.

Run lifecycle

Frank stores lightweight run summaries in Postgres and sends detailed work to the relevant runtime.

Work	Runtime	User-facing records
Source discovery	Temporal source worker	Discovery workflow status.
Source sync	Temporal source worker + Iceberg	Sync run history, logs, source status.
Transform materialization	Dagster + API callback	TransformRun records, Dagster run ID, logs, lineage.
Pipeline sandbox	API + worker orchestration	Sandbox workflow status and step results.
Ontology sync	Temporal / Dagster sensor path	OntologySyncRun records and backing dataset history.
AI assistance	Martha workflow execution	AI trace/execution IDs and structured response payloads.

Logs

Useful CLI commands:

frankctl sources logs <source-id> <run-id> -f
frankctl transforms logs <transform-id> <run-id> -f
frankctl runs get <workflow-id>
frankctl runs wait <workflow-id>

Useful Compose commands:

make logs
make logs-api
make logs-ui
docker-compose logs -f source-worker
docker-compose logs -f transform-worker

The API and workers use structured JSON logging so Loki queries can filter by fields such as workflow ID, Dagster run ID, transform ID, source ID, and trace ID.

Traces

OpenTelemetry is initialized in the API and workers. Dagster-triggered transform paths, source worker paths, and AI paths include trace context where available.

Relevant env:

OTEL_EXPORTER_OTLP_ENDPOINT=alloy:4317

Source operations

Operational checklist:

1. Source is ready or active.
2. Discovery schema is current.
3. Streams are enabled and configured.
4. Incremental streams have cursor fields.
5. Merge streams have primary keys.
6. Target config matches the desired Bronze namespace/table convention.
7. Sync history shows successful runs.

CLI:

frankctl sources get <source-id>
frankctl sources streams list <source-id>
frankctl sources history <source-id>
frankctl sources sync <source-id>

Source credential operations

Connector fields declared with credential: true or type: password are write-only on every read, export, log, trace, and error surface. Runtime storage is an explicit deployment policy:

Mode	Deployment contract
legacy_inline	Current mutable/base Compose compatibility during Generation A. Write-only values are persisted inside redacted Source.source_config; ordinary updates preserve hidden values. No Vault reference or credential-lifecycle operation is available.
vault	Immutable release contract. Values live in the shared platform Vault KV v2 Source namespace; PostgreSQL and Temporal carry only an opaque tenant-owned reference and exact version.

Base docker-compose.yml selects SOURCE_CREDENTIAL_MODE=legacy_inline. Supply values through the Source create/update UI or its write-only credential_values request field. Keep transient requests outside Git and shell arguments. The API merges values only at persistence and never returns them. sources credentials set|rotate are Vault-only and fail in this mode.

The immutable release overlay selects SOURCE_CREDENTIAL_MODE=vault. Create a Source with ordinary config, then stream the exact declared credential object into the CLI:

frankctl sources create -f source.yaml
frankctl sources credentials set <source-id> \
  --name <tenant-local-name> --values-file - < credential-values.yaml
frankctl sources credentials rotate <source-id> \
  --values-file - < rotated-values.yaml

Keep any temporary values file untracked and mode 0600, or supply stdin from the approved secret manager. Vault-mode Source readback and pipeline export contain only safe metadata and the opaque credential_ref. Rotation updates the exact Vault version without recreating Source, stream, schedule, cursor, ledger, or run state.

The one-shot cutover gate writes and exact-version verifies every registered credential before removing only structurally declared fields from PostgreSQL. All affected rows commit together after the locked inventory succeeds. Before commit, a failure rolls back PostgreSQL and quarantines unreferenced Vault versions; after commit, recovery fixes forward or restores a matched, checksummed PostgreSQL/Vault artifact pair. After commit, returning that database to legacy_inline or an older binary is unsupported. Detaching or deleting a Source never deletes Vault data; revocation requires an unreferenced credential and retains Vault history for operator-managed recovery.

Transform operations

Operational checklist:

1. Transform is hydrated.
2. can_run_now is true in the API/UI.
3. Current artifact runtime matches the expected execution engine.
4. Dagster code location has loaded the asset.
5. Last run outcome is not already running.
6. Logs are available for the run.
7. Output table and lineage edges match expectations.

CLI:

frankctl transforms get <transform-id>
frankctl transforms trigger <transform-id>
frankctl transforms runs <transform-id>
frankctl transforms logs <transform-id> <run-id>

Pipeline operations

Pipeline deployment path:

1. Draft or update pipeline.
2. Validate DAG.
3. Run sandbox.
4. Review step results.
5. Activate.
6. Monitor runs.

CLI:

frankctl pipelines get <pipeline-id> --include-version
frankctl pipelines validate <pipeline-id> --timeout 600

Source worker liveness

The unified source worker runs synchronous discovery and extraction activities. Its activity-slot limit must not exceed its executor-thread count. The shipped defaults are eight executor threads, eight activity slots, 100 workflow slots, and five pollers for each task type. Set the TEMPORAL_SOURCE_* capacity and liveness variables in the ecosystem ../.env; Compose deliberately does not repeat them so operator recovery settings are not shadowed.

Check the same probe used by Docker:

docker compose exec source-worker \
  python -m backend.temporal.source_worker_health

A healthy result requires a fresh source-worker event-loop watchdog and a current matching workflow poller on TEMPORAL_SOURCE_TASK_QUEUE. The result also reports activity-poller freshness, but a missing activity poller is diagnostic rather than fatal while every activity slot is occupied. A nonexistent queue, stale watchdog, connection failure, or missing workflow poller returns a non-zero status. This is a process and workflow-queue liveness check, not evidence that activities are progressing or that the sync backlog is draining. Use Temporal backlog, activity latency, heartbeat, and sync-run outcomes for that readiness decision. If the event-loop watchdog itself cannot run, the worker exits instead of continuing without liveness evidence.

Do not blindly restart a worker into an existing workflow backlog. First record the queue backlog and pollers, pause or bound new scheduled starts under the approved recovery plan, and monitor CPU, memory, failures, and schedule-to-start latency while work drains.

Ontology operations

Before syncing a backing dataset:

1. Entity type exists and is the intended version.
2. Iceberg table exists and has expected columns.
3. Property mappings include the primary key column.
4. Relationship mappings include target type and target key.
5. Health check passes.
6. Sync history is reviewed after trigger.

In production and shared staging, every Frank tenant must have an exact entry in ONTOLOGY_TENANT_BINDINGS_JSON. Wildcards and whitespace-normalized aliases are rejected. Service/delegation callers also require the ontology:sync capability. The API checks this policy on declaration, update, preflight, health, entity read, and trigger; the V2 worker re-checks it before Iceberg or ontology I/O so a revoked binding also stops delayed work.

API:

GET  /api/v1/backing-datasets/capabilities
GET  /api/v1/backing-datasets/{id}/health
POST /api/v1/backing-datasets/{id}/sync?force=true
GET  /api/v1/backing-datasets/{id}/sync-history
GET  /api/v1/backing-datasets/{id}/sync-history/{run_id}

Compare attempted_sync_spec_fingerprint with applied_sync_spec_fingerprint on the exact run. effective_spec_changed means mappings or another effective execution input replayed an unchanged snapshot; unchanged must report status: skipped and rows_synced: 0. A failed run must retain the prior applied fingerprint. When recording snapshot evidence, use snapshot_id_exact; the numeric snapshot_id remains in API output for legacy clients but can lose precision when parsed as a JavaScript number.

Rolling out sync contract v2

The V1 Temporal contract is immutable during the drain window: OntologySyncWorkflow and activities sync_to_ontology, update_sync_status, and emit_ontology_lineage_edge stay on ontology-sync-task-queue. V2 uses OntologySyncWorkflowV2, the corresponding *_v2 activities, ontology-sync-v2-task-queue, and the separate ontology-worker-v2 service. Only the V1 worker owns reconciliation schedules.

Use this migration-first order:

1. Apply the additive s39_bd_sync_spec_fingerprint migration.
2. Start ontology-worker-v2 while retaining the V1 worker.
3. Use Temporal task-queue describe/visibility evidence to verify live pollers and one executable workflow on both queues. Container health alone is not sufficient.
4. Route new API sync starts to V2 only after the V2 poller is proven live.
5. Update consumers and the CLI, then allow V1 histories and retries to drain.

Rollback routes new API starts back to V1 first. Keep the V2 worker until its workflows are terminal, and do not downgrade the additive schema while any V2 API, worker, history, or retry can still use it. Local verification of these steps is not production deployment evidence.

Connector-executor worker releases (#665)

The executor policy admits only immutable Frank-built connector derivatives. RSS, S3 and SFTP Bulk are admitted by exact reviewed OCI index digest, selected linux/amd64 platform-manifest digest and Docker config admission ID. Every other connector remains closed until its own build, scan, compatibility and review gates land. Official Airbyte images and mutable tags are never fallback paths.

Run .github/workflows/rebuild-airbyte-connector-images.yml without publication first. A candidate must pass exact source/license materialization, the bounded secret-safe input gate derived from every exact local COPY across the selected Dockerfile-stage ancestry. Reachable ADD, external or dynamic stage copies, and non-cache or context-bearing RUN --mount inputs are denied. Reachable ONBUILD and BuildKit # syntax= parser frontends are forbidden, and every reachable external stage must root at an immutable @sha256 reference. The reachable external roots must exactly equal the manifest-owned build_base and runtime_base; a digest alone does not establish review. Other leading RUN options are denied. Each BuildKit daemon is separately started from the pinned digest with bounded resources, bridge networking, job-owned labelled Docker-volume state, and no insecure entitlement. The workflow removes only its labelled daemon container; it never explicitly removes the state volume, which remains owned by the disposable GitHub-hosted runner until teardown. Buildx uses the remote driver, and the workflow reads back both boundaries before building. The workflow removes and reads back its labelled daemon before JavaScript post hooks run. It disables setup-buildx's duplicate post cleanup and build-push's optional post-job build-history summary/record upload. Provenance and SBOM are still generated by the main publication build and verified from the registry; the optional Docker job summary is not admission evidence. Frank's named steps likewise own registry logout and delete the isolated Docker config before later action post hooks. The login action's optional post-job logout is disabled, and both normal and fallback cleanup read back that exact path as absent, and no later action-owned hook may touch that credential state. The candidate must also pass its CLI contract, a positive secret-scanner canary, and full-image vulnerability and secret scans with Critical=0 and High=0. There is no vulnerability exception path. Publication creates only a run-specific unadmitted GHCR candidate in a write-capable job that ends after retaining its digest and safe identity metadata. A dependent job limited to contents: read and packages: read reads back its raw OCI index digest and attestations and repeats the CLI and scan gates. Run real check, discover and read compatibility, then update the manifest and executor policy through a reviewed PR. Until published_image contains the accepted digest, the connector must remain fail closed. The RSS compatibility manifest is baked into the image and must never be supplied as a runtime mount.

Candidate identity and the credential-safe supply summary share one 30-day admission window because the export gate requires both. If either artifact has expired, dispatch a fresh reviewed publication and admit that exact new run; do not reconstruct evidence from GHCR, the manifest, an older export or a workflow rerun that still references the expired workflow definition. The credential-free image export keeps its separate three-day transfer window.

GHCR remains private. The server never receives a GHCR credential and never pulls a connector at runtime. The manual .github/workflows/export-airbyte-executor-images.yml job has package-read authority only, verifies the accepted publication evidence, and emits a closed, credential-free export. Before saving, it assigns each image one deterministic non-registry load reference derived from connector plus its config admission ID. After deleting registry credentials, the job hashes every content-addressed blob and requires the Docker compatibility manifest and preferred OCI import graph to resolve to the same config/layers, exactly one expected named root and one coherent OCI or Docker schema-2 media family. If containerd retains a distribution-source annotation, it must exactly match the admitted GHCR repository. If Moby emits its tagged-save repositories compatibility file, that file must name only the expected local repository/tag and the final layer digest already bound by both manifests. The job then reloads the validated export and proves both the exact three-image ID inventory delta and that every local reference resolves to its expected ID with no other tag. The schema-2 manifest binds each published OCI index digest to the exact selected platform manifest, Docker config admission ID, local load reference and checksummed archive. The canonical server runbook repeats the exact file, checksum, blob and dual-graph validation, loads the archives offline, and authenticates both the config ID and the archive's single OCI-root ID. Docker's classic image store addresses the loaded graph by config ID; its containerd store addresses the same graph by OCI-root ID. Bootstrap accepts only the exact engine-observed value among those two authenticated identifiers, then installs one connector-to-executable-ID policy and hash for both broker and AuthZ. Missing, extra, duplicate, arbitrary, tagless, differently referenced, split-brain, drifted-repositories, policy-hash-drifted or changed bytes fail closed. The local reference makes the archive addressable but is never executed. The runtime policy is derived only from the installed, hash-verified admission snapshot. Bootstrap stops and reads back the prior broker, engine and AuthZ and disables their boot activation before replacing active policy or units, enables only the resolved generation, and starts a fresh hash-verifying AuthZ before engine and broker. The resulting bootstrap journal is schema 2; schema 1 journals are rejected as incompatible. Install-only invocation is rejected because it could replace a live policy generation without completing the stop, preload, bind, and restart transaction.

Source and discovery workers do not run Docker or receive any Docker API socket. Each receives one read-only role directory containing only broker.sock; the broker owns the private connector runtime, digest policy, work directories, networking, resources, and outcome ledger. The two workers also receive separate read-only PostgreSQL and object-store secret files through generic *_FILE names. Credential values are absent from Compose and process environment metadata. Both workers run as the image's non-root UID/GID, have a read-only root filesystem and data mount, drop every Linux capability, enable no-new-privileges, and use a bounded noexec /tmp tmpfs as each worker service container's only writable path. Separately, every broker-started connector receives reviewed noexec,nosuid,nodev tmpfs mounts at /tmp and /airbyte/tmp; connector HOME and TMPDIR point to the smaller bounded /airbyte/tmp mount.

Broker traces use common-infra's private source-executor gateway at https://127.0.0.1:14317. The gateway is OTLP/gRPC traces-only over TLS 1.3, runs as a dedicated non-root identity, and is published on host loopback only. The broker receives the public CA file, never the gateway key or an auth credential. A successful TLS handshake is not observability acceptance: the existing Alloy upstream must be available and one credential-safe broker trace must be read back from Tempo through the canonical server runbook.

The accepted #665 connector-execution boundary is retained by the #667 release deployer. The old executor-only deploy command is retired and refuses current deployment. The canonical server runbook uses scripts/deploy_release.py from an exact clean merged SHA, verifies its OCI revision and content-addressed image ID, and requires the installed, distinct /run/frank-airbyte-executor/source and /run/frank-airbyte-executor/discovery broker endpoints to be live before and after cutover. Broker paths are not operator-selectable deployment arguments. A known failure stops both workers and leaves Temporal work queued. Restoring the host Docker socket is never a rollback. Worker recreation is --no-build --pull never, and the deployer re-inspects the exact local image ID after worker stop and immediately before Compose. The path-resolution checkout is not used to build or mount worker source, neither worker receives the operator env file, and missing image state cannot fall back to a registry.

The host bootstrap creates the broker virtual environment without ensurepip and uses the canonical runbook's reviewed system python3-pip package to install the hash-locked, binary-only dependency set into that exact interpreter. This is replay-safe after a partial virtual-environment creation: it reuses an executable interpreter, repeats the locked install under a separate bounded timeout, and reruns the broker import probe. Missing host pip, a missing interpreter, timeout, dependency failure, or import failure stops before any executor service starts. The completed release is root-owned and claims-group-readable: directories are setgid 2550 so a same-SHA recovery retains group inheritance, executables are 0550, and regular files are 0440. Bootstrap rejects unexpected external links, hardlinks and ownership drift, normalizes every non-link mode to that closed set, then executes a credential-free runtime/import probe as the exact AuthZ and broker users. A root-only release or a runtime that imports only as root is not deployable evidence. The user executor unit is static; only the system engine wrapper owns its lifecycle. Before enabling linger, bootstrap removes only the exact legacy default.target.wants link created by older releases. It accepts only the literal absolute target or exactly ../frank-airbyte-executor.service; aliases and cancellation components fail closed. Only after validating that link does it seal that existing parent as root-owned 0755, revalidate the same inode/target, unlink that leaf, and preserve siblings. Absent or rejected state leaves parent metadata unchanged. It then explicitly starts the engine identity's user@<uid>.service, requires exact active, stops the old system broker, engine, AuthZ and user executor, requires exact inactive and disabled authority units, and only then reloads the user manager. An unknown link, manager failure, stale active user executor, or reload failure stops before egress, image preload, or new executor service activation.

Persistent Docker uses namespace socket group zero so rootless listener creation does not attempt to assign the host's default docker group from inside RootlessKit. After readiness, the engine user unit changes only the exact socket to the reviewed dedicated Docker-socket group and mode 0660, then verifies it is a Unix socket. This post-start handoff runs on every daemon restart. Adding the broker to the engine primary group or applying a one-time root chgrp is not an accepted substitute. Any activation failure explicitly stops and reads back both the three system authority units and the exact rootless user unit; failure in one cleanup plane does not skip the other.

Docker 29.6.2 discovers the Frank AuthZ definition only from the scoped rootless daemon registry at $HOME/.local/lib/docker/plugins/frank-airbyte-authz.spec. Bootstrap requires the engine home and every managed registry parent to be root-owned 0755, and the exact spec to be a root-owned, single-link 0444 regular file whose bytes name only the reviewed AuthZ socket. Any same-name flat or nested .spec or .json definition under /etc/docker/plugins, or any ambiguous nested definition in the scoped registry, stops activation. Bootstrap removes the old $HOME/.docker/plugins/frank-airbyte-authz.spec leaf only when its parent metadata and exact bytes still match; it preserves the directories and all siblings. The AuthZ system unit itself is Type=notify, NotifyAccess=main and reports READY=1 only after its policy socket is bound and mode 0660. SIGTERM exits through a bounded request loop and removes only the exact socket inode bound by that process, so an ordinary systemd restart can rebind. A bind collision or replaced socket is preserved, denied readiness and must be investigated rather than blindly unlinked.

The broker is also Type=notify, NotifyAccess=main, with a 15-second startup bound. It emits READY=1 only after its Docker/image attestations pass and both the source and discovery sockets exist as separate, single-link 0660 Unix sockets owned by the broker and their distinct role groups. It never removes a pre-existing pathname. On SIGTERM, both listeners stop and remove only their own unchanged socket inodes. systemctl start therefore cannot succeed while the broker's Docker ping, image admission, socket ownership or role separation is still failing.

The persistent and transient rootless daemons use systemd native readiness with Type=notify and NotifyAccess=all. The latter is required because child dockerd, not the RootlessKit main PID, emits readiness. Bootstrap reconciles only frank-airbyte-preload.service and its exact private socket before and after every preload attempt. Socket cleanup seals the already-open parent, revalidates the parent and leaf identity, unlinks and fsyncs only the unchanged single-link socket, then restores exact engine ownership and 0700. An unknown unit state, failed stop/readback, wrong-type or concurrently substituted sentinel stops unchanged; a parent not already in that exact engine-owned state also stops without mutation. When the operation and cleanup both fail, both causes are preserved under one safe error code, and only a closed internal code allowlist can enter notes. There is no sleep, socket-only readiness, UID-wide kill, user-manager restart, or Docker-data cleanup fallback. Bootstrap also records the discovered runuser path and hash; the probe uses that exact evidence-bound executable and rejects drift. Both accept only /usr/bin/runuser or /usr/sbin/runuser with an executable root-owned leaf and root-owned, group/other-non-writable ancestors. The runtime probe requires the persistent user service to read back notify, all, active, and running, and the transient preload service to be collected.

The host probe negotiates the private engine through exact unversioned GET /version over its Unix socket. It does not use Docker CLI version, whose client negotiation emits a version-prefixed route that is deliberately denied by the executor policy. AuthZ admits only the byte-exact raw /version URI in both authorization phases. One absolute monotonic deadline covers the entire connect/request/response exchange; the body is size-bounded, must be HTTP 200 JSON with a numeric engine version, and never falls back to the host daemon. Later reviewed info, image inspection, and denied volume checks continue through absolute /usr/bin/docker against the same explicit private socket. Every invocation receives a minimal child environment with DOCKER_API_VERSION=1.51; inherited Docker host, context, TLS, custom-header, credential-config and executable-resolution settings are excluded. Auto-selecting a newer daemon API is a probe failure, not a reason to widen executor authorization.

Docker CLI image-attestation output must be exactly one --format record with one terminal LF. The probe removes only that terminator before exact field validation. Missing LF, CRLF, embedded LF and extra records fail closed; the probe never uses generic whitespace normalization.

Offline imports have their own fixed 180-second bound per authenticated archive. That bound applies only to the exact private-socket docker image load --input commands. All bootstrap control, readiness, inspection and cleanup commands keep the 15-second bound. An import timeout fails as approved-image-preload-failed, collects the exact transient preload unit, removes only its exact socket, and does not retry, pull from a registry, use the host daemon, or activate the AuthZ/engine/broker generation.

The tool writes a credential-safe deployment journal, but that journal and a green container health state are not live acceptance. Production acceptance also requires exact workflow/activity pollers, representative real discovery and extraction, allowed and denied broker requests, restart/recreation and byte-drift recovery evidence.

If a request reports connector_outcome_unknown, do not retry it manually. Claims with an exact stored container ID are reconciled by the next exact Temporal attempt only after Docker proves that ID absent or terminal. A no-ID claim requires the installed operator reconciler after the original request deadline:

sudo -u frank-airbyte-broker \
  /opt/frank-airbyte-executor/releases/<accepted-40-char-sha>/venv/bin/python \
  /opt/frank-airbyte-executor/releases/<accepted-40-char-sha>/app/scripts/reconcile_airbyte_executor_claim.py \
  --claim-id <exact-claim-uuid> \
  --docker-socket /run/frank-airbyte-executor/docker/docker.sock \
  --state /opt/frank-airbyte-executor/state/broker/claims.sqlite \
  --workspace-root /run/frank-airbyte-executor/claims

Use only the exact quarantined claim UUID from safe broker evidence and the exact installed release SHA. The command refuses to run before the original deadline. It reconciles only a missing deterministic container name or an exact-label container in created, exited, dead, or removing state; a running container, identity mismatch, Docker attestation failure, or transport ambiguity remains quarantined. A successful result is still not extraction acceptance; allow Temporal to issue attempt N+1 and observe the real outcome.

Target immutable production releases

This is the target/alternate release topology, not the current mutable box. The current base Compose deployment uses locally built images and does not use Harbor or docker-compose.release.yml. Activating this topology is a deliberate runbook cutover.

For the immutable topology, Frank application releases are built and verified by .github/workflows/harbor.yml. Pull requests run source-only, non-deploying release-contract tests; the only secret they consume is the established GH_PAT repository secret needed to fetch the private shared-utils submodule, and checkout does not persist it. This avoids a second undeployed checkout secret and does not authorize production access. Pull requests receive no Harbor, SSH, runtime, or production-environment credentials and cannot publish candidates, promote tags, or deploy. After merge, the protected main workflow builds the four exact candidates, verifies their digests against real PostgreSQL and Temporal, and may deploy through the protected production GitHub environment only when the repository variable PRODUCTION_DEPLOY_ENABLED is exactly true. Keep the opt-in disabled until that environment and the reviewed host boundary are provisioned. The exact source-worker digest also runs the credential JIT, legacy-payload redaction, and poller-health suites; an import-only smoke test is not sufficient.

The production input is a credential-free manifest plus a deterministic, hash-bound bundle containing only the two Compose files, the release deployer, and reviewed runtime assets. CI never sends a Git checkout, source overlay, environment file, or credential to the host. A first protected deployment may initialize the reviewed host launcher and operator paths. Every later release requires the launcher, config, and bootstrap evidence to be byte-identical and mode-correct; drift stops deployment rather than being overwritten.

The launcher reads operator-owned paths from /opt/frank-low-code/config/deploy.json:

{
  "env_file": "/operator/path/frank.env",
  "data_dir": "/operator/path/frank-data"
}

The environment file remains outside release artifacts and is used only for Compose interpolation. No resolved release service inherits it. API, migration, preflight, source-worker, and discovery-worker Vault identities use the same generic VAULT_ROLE_ID_FILE, VAULT_SECRET_ID_FILE, and VAULT_CACERT container paths but distinct host files; every unrelated service receives no Vault identity. Direct AppRole values and unsafe identity files fail release preflight. The stable data directory remains the same across releases.

Harbor is one optional immutable-image publication path. The canonical server runbook also defines the reviewed local-image #665 cutover, so Harbor is not a prerequisite for connector-execution isolation. Neither path copies source into running containers or bind-mounts the host checkout into workers.

The deployer validates all digests and OCI Git-revision labels, proves that the API AppRole has exactly create, update, and read on the reviewed Source credential path and the worker AppRole has exactly read, applies the additive schema migration, and migrates every structurally declared legacy credential as one database transaction. Any identity, schema, Vault readback, or migration failure stops before a new application process starts. The old API and source consumers are stopped during contraction; this is not described as a Temporal queue drain. New source and discovery workers sanitize queued legacy payloads and must prove exact workflow and activity pollers before admission reopens. The deployer then waits for the exact V2 ontology pollers before starting the API and remaining workers. Only a successful deployment writes sanitized evidence under /opt/frank-low-code/evidence/ and moves the current symlink. It never runs Compose down, prunes images or volumes, or executes an Alembic downgrade.

Verify a release with the health endpoint and persisted evidence:

curl --fail https://api.example.invalid/health

The response includes only the full release SHA and API image digest. Compare those values to the reviewed release manifest. Also verify the single Alembic head and current Temporal workflow/activity pollers; container health or a 2xx response alone is not release evidence.

On failure, inspect the persisted deployment journal to identify the last completed stage. The unchanged current symlink does not prove that partially started services reverted. Fix forward after a committed migration. A prior digest manifest may be redeployed only after its binaries are explicitly proven compatible with the current additive schema and live Temporal histories. Never run an automatic schema downgrade, compose down, volume deletion, database recreation, or image prune. Keep the V2 worker available while any V2 workflow history or retry remains non-terminal.

Changing the launcher or operator paths is not an application release. It requires a separate reviewed and explicitly approved re-bootstrap procedure; normal CI deploys intentionally fail on such drift.

Key environment variables

Area	Variables
API and auth	KEYCLOAK_URL, KEYCLOAK_REALM, KEYCLOAK_CLIENT_ID, KEYCLOAK_ISSUER, CORS_ALLOWED_ORIGINS
Database	POSTGRES_HOST, POSTGRES_PORT, POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD
Iceberg/S3	ICEBERG_CATALOG_URI, ICEBERG_CATALOG, AWS_ENDPOINT_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION
Temporal	TEMPORAL_HOST, TEMPORAL_PORT, TEMPORAL_NAMESPACE, task queue variables
Source worker	TEMPORAL_SOURCE_ACTIVITY_MAX_WORKERS, TEMPORAL_SOURCE_MAX_CONCURRENT_ACTIVITIES, TEMPORAL_SOURCE_MAX_CONCURRENT_WORKFLOW_TASKS, TEMPORAL_SOURCE_MAX_CONCURRENT_ACTIVITY_TASK_POLLS, TEMPORAL_SOURCE_MAX_CONCURRENT_WORKFLOW_TASK_POLLS, role-specific FRANK_*_BROKER_SOCKET_DIR
Source credential mode	SOURCE_CREDENTIAL_MODE is exactly legacy_inline on base Compose or vault on the immutable release overlay during Generation A
Dagster	DAGSTER_URL
Logs/traces	LOKI_URL, LOKI_AUTH_TOKEN, OTEL_EXPORTER_OTLP_ENDPOINT
AI	MARTHA_API_URL, MARTHA_KEYCLOAK_URL, MARTHA_CLIENT_ID, MARTHA_CLIENT_SECRET
Ontology	ONTOLOGY_ENABLED, ONTOLOGY_SERVICE_URL, ONTOLOGY_API_KEY, ONTOLOGY_TENANT_ID, ONTOLOGY_TENANT_BINDINGS_JSON
Source credential Vault	Shared VAULT_ADDR/CA plus separate service-specific file-backed AppRole identities on the immutable release overlay
Worker platform secrets	Service-specific PostgreSQL and object-store secret file paths; containers receive only generic POSTGRES_PASSWORD_FILE, AWS_ACCESS_KEY_ID_FILE, and AWS_SECRET_ACCESS_KEY_FILE
Pattern registry	PATTERN_WEBHOOK_SECRET, PATTERN_ADMIN_SECRET

Maintenance commands

make up
make down
make status
make logs
make build
make build-no-cache
make init-iceberg
make init-db
make init-sdm
make test-iceberg

For API route-level checks:

curl http://localhost:8002/health
curl http://localhost:8002/api/v1/status
curl http://localhost:8002/api/v1/services/health