Frank

Appearance

frankctl — CLI Reference ​

Terminal-first client for the Frank Low-Code Pipeline platform. The CLI wraps the same REST API as the dashboard, plus a declarative apply path for git-managed provisioning.

This guide covers every shipped frankctl verb. Run frankctl <cmd> --help for option details.

Authoring a feed end to end? See From External Source to Ontology for the full recipe (preview → author → sandbox-validate → apply → verify), with a real worked example.

Install ​

bash
npm install -g @aiaiai-pt/frankctl
frankctl --help

Requires Node.js >= 18. The published package is @aiaiai-pt/frankctl; the binary is frankctl.

From source ​

bash
cd frank-low-code-pipeline/frank-cli
npm install
npm run build
npm link
frankctl --help

Configuration ​

VariablePurpose
FRANKCTL_API_URLFrank API base URL. Default http://localhost:8000.
FRANKCTL_KEYCLOAK_URLKeycloak base URL.
FRANKCTL_KEYCLOAK_REALMKeycloak realm. Default frank.
FRANKCTL_CLIENT_IDOAuth client id.
FRANKCTL_USERNAMEUsername for headless login.
FRANKCTL_PASSWORDPassword input for headless login; prefer injected stdin where available.
FRANKCTL_CLIENT_SECRETService-account secret; prefer injected stdin where available.
FRANKCTL_PROFILEActive config profile.
FRANK_DEV_MODE=true + FRANKCTL_TENANT_ID=<uuid>Forward X-Tenant-ID header instead of an OIDC token. Local dev only.

frankctl auth login runs browser PKCE against the configured Keycloak realm and stores the token in the active profile's mode-0600 credential file. Headless and service-account secrets are never accepted in argv:

bash
printf '%s\n' "$FRANKCTL_PASSWORD" | \
  frankctl auth login --headless --username "$FRANKCTL_USERNAME" --password-stdin
printf '%s\n' "$FRANKCTL_CLIENT_SECRET" | \
  frankctl auth login --service-account --client-secret-stdin

Do not run these examples through a wrapper that prints stdin or environment values. Omitting the stdin flags makes the command read the corresponding environment variable directly.

Resources ​

frankctl templates ​

Discovers and renders curated source-to-ontology verticals. Rendering is local: it makes no API request and emits ordinary frank.platform/v1 multi-document YAML for the existing pipelines apply -f command.

bash
frankctl templates list
frankctl templates show cira
frankctl --json templates show cira

# Live-proven CIRA components (default)
frankctl templates render cira -o cira-proven.yaml

# Every declared CIRA component
frankctl templates render cira --all -o cira-all.yaml

# One or more explicit components and structural target parameters
frankctl templates render cira \
  --component cityfy-events \
  --set frank_tenant_prefix=00000000 \
  --set ontology_tenant_id=cira \
  -o cira-events.yaml

frankctl pipelines apply -f cira-proven.yaml --dry-run
frankctl pipelines apply -f cira-proven.yaml --dry-run-server --live-ontology
frankctl pipelines apply -f cira-proven.yaml --wait

Repeat --component to select an explicit subset; it cannot be combined with --all. show reports each component's live_proven or declared_pending status and any pre-existing Source names it requires. The two --set values rewrite only the Frank Iceberg tenant prefix and BackingDataset ontology tenant fields. There is no arbitrary template evaluation.

The live-proven default contains 50 declarations across seven components: four Sources, 23 Pipelines, and 23 BackingDatasets covering Cityfy events, Transdev GTFS, CICLOPE emergency data, regional transport, and CIRA air quality. It emits no pending-component warning. --all includes the pending regional stop-time component: one IdentityPolicy, one Pipeline, and one BackingDataset, for 53 declarations across eight components. It emits a pending warning because that three-document slice is declared and dry-runnable but not yet live-proven end to end.

The air-observation declaration explicitly casts its millisecond epoch to the persisted timestamp(6) with time zone precision. This is value-preserving and lets an unchanged full refresh retain the exact Silver snapshot; omitting the cast makes Trino infer timestamp(3) and causes a false schema-change rewrite.

The CIRA pack treats ontology schema as externally owned: rendered output contains no ensure_schema and cannot create or modify ontology types. Source prerequisites are ordinary Sources configured through the existing product; template rendering neither stores nor rotates their credentials. See the repository's pipeline_templates/cira pack for the inventory and exact repeatability contract.

frankctl sources ​

CRUD for data sources.

bash
frankctl sources list
frankctl sources get <id>
frankctl sources create -f source.yaml
frankctl sources update <id> -f patch.yaml
frankctl sources delete <id> --yes
frankctl sources discover <id>
frankctl sources sync <id>
frankctl sources logs <id> <run-id>
frankctl sources history <id>

# Vault mode only:
frankctl sources credentials set <id> --name <name> --values-file - < values.yaml
frankctl sources credentials rotate <id> --values-file - < rotated-values.yaml

frankctl sources streams list <source-id>
frankctl sources streams set <source-id> -f streams.yaml
frankctl sources streams refresh-schema <source-id> --from-discovery
frankctl sources streams refresh-schema <source-id> -f discovery-result.yaml

Credential behavior depends on the server's explicit deployment mode. Current base Compose uses SOURCE_CREDENTIAL_MODE=legacy_inline: Source create/update accepts the write-only credential_values field, persists it inside the redacted source_config, and preserves hidden values on ordinary updates. The sources credentials set|rotate commands above are unavailable in that mode.

The immutable release uses SOURCE_CREDENTIAL_MODE=vault. Only there do those credential commands accept a YAML/JSON object matching the pattern's declared fields exactly and write it to tenant-isolated Vault. Keep temporary value files untracked and mode 0600, or pipe them from the approved secret manager. A Vault-mode Git manifest contains ordinary source_config and the opaque credential_ref; pipelines export preserves that reference without values. In inline mode, export instead strips registered credential forms and emits no reference. Neither mode returns values through the API or CLI.

streams refresh-schema requires either --from-discovery or -f. The discovery form accepts --timeout <sec> and --poll-interval <ms>; the file form expects {streams: [...]} containing a complete discovery snapshot. For matched existing streams it updates only discovered schema and supported sync modes. It preserves namespace/destination routing, sync/write/cursor/enabled configuration, and runtime state. Streams omitted from a non-empty snapshot are removed as stale. See Sources for the namespace contract and full consequences.

Stream-level type-drift ops (issue #469):

bash
frankctl sources streams diff-types <source-id> <stream-id>
frankctl sources streams set-types  <source-id> <stream-id> \
    --field dt=integer --field pm25=number

See § Bronze type drift at the bottom of this guide for the playbook.

frankctl pipelines ​

CRUD for multi-step pipelines plus the declarative apply path (see below).

bash
frankctl pipelines list
frankctl pipelines get <id>
frankctl pipelines validate <id>          # sandbox dry-run
frankctl pipelines delete <id> --yes      # archive; retains data and history

# Pipeline-owned scheduling and lifecycle
frankctl pipelines schedule get <id>
frankctl pipelines schedule set <id> --type manual
frankctl pipelines schedule set <id> --type eager
frankctl pipelines schedule set <id> --type cron \
  --value "0 6 * * *" --timezone Europe/Lisbon
frankctl pipelines schedule set <id> --type interval \
  --value 15m --timezone UTC
frankctl pipelines pause <id>
frankctl pipelines resume <id>
frankctl pipelines trigger <id>

# Declarative apply
frankctl pipelines apply -f vertical.yaml
frankctl pipelines export <id> -o yaml > vertical.yaml

# Scaffold a starter vertical from a synced source's physical bronze schema
frankctl pipelines scaffold --source <id-or-name> > vertical.yaml
frankctl pipelines scaffold --source my_source --stream obs -o vertical.yaml

Pipeline scheduling controls the Dagster transforms generated from the current Pipeline version. It does not control Source extraction; Source schedules are owned separately by Temporal through frankctl schedules ....

eager is the normal continuous mode: roots react to new upstream materializations and downstream steps follow DAG dependencies. cron and interval run independently of Source freshness. Exact intervals are limited to minute values that divide 60, hour values that divide 24, or 1d; use a five-field cron expression for other cadences. Timezones are IANA names and default to UTC.

schedule get, schedule set, pause, and resume read back the owning Dagster location before exiting successfully. Human and --json output expose desired and effective policy, desired and observed fingerprints, root asset identities, and reconciliation state. Durable-but-unconfirmed pending_activation, drifted, or error state exits non-zero instead of reporting a false success. trigger is active-Pipeline only and returns one correlation ID plus a receipt for each root. Do not schedule a generated Transform directly: the API returns 409 and points to the owning Pipeline command. Pausing a Pipeline does not pause standalone Transforms or the shared Dagster sensor.

pipelines scaffold reads a synced source's physical bronze schema — the real nested Iceberg ROW columns, not discovery's flattened logical names — and emits an apply-ready Source + Pipeline + BackingDataset multi-doc YAML. Everything is derived from the real schema:

  • the silver custom_sql step is pre-filled with correct Trino ROW references (e.g. o."properties"."idEstacao"), with envelope _* lineage columns dropped;
  • the BackingDataset carries a derived property_mappings + ensure_schema (ontology property = column name; physical type → ontology type; primary key picked heuristically);
  • each stream gets write_disposition from its sync_mode (full_refresh → replace, so scheduled syncs don't re-append);
  • ensure_schema.tenant_scoped: false (reference datasets — tenant-scoped types need an X-Tenant-Id the sync path doesn't send).

So the only edit before pipelines apply -f --wait is naming the ontology entity (entity_type_id / entity_type_name). The source must have synced to bronze first (the ROW structure only exists after a sync); scaffolding an un-synced source errors with that hint. --stream selects which stream when a source has more than one enabled.

frankctl backing-datasets ​

CRUD for backing datasets — Iceberg tables that bind to an ontology entity type. Aliased as frankctl bd.

bash
frankctl backing-datasets list
frankctl backing-datasets get <id>
frankctl backing-datasets create -f bd.yaml [--allow-deprecate]
frankctl backing-datasets update <id> -f patch.yaml [--allow-deprecate]
frankctl backing-datasets sync <id> [--force] [--no-wait]   # push silver → ontology, poll exact run
frankctl backing-datasets entities <id> [--limit N]   # read entities back out of the ontology
frankctl backing-datasets delete <id> --yes

entities closes the authoring loop: after a sync it reads the actual entities back from the ontology (values, not just status), scoped by the BD's ontology_tenant_id — so verification never leaves the CLI.

When binding a BD to a pipeline, omit iceberg_namespace/iceberg_table: set pipeline: (or pipeline_id:) and the silver target is derived from the pipeline's terminal step.

Against a sync-contract-v2 API, sync triggers an ontology sync and polls the returned run ID until that exact run reaches synced, skipped, or error (exit 1 on error). Its output includes the replay reason, row count, exact snapshot ID, force flag, and the safe first 12 characters of both attempted and applied effective-SyncSpec fingerprints. Human output prefers the decimal-string snapshot_id_exact, avoiding JavaScript precision loss for 64-bit Iceberg IDs. Use --force to full-scan and REST-upsert all rows even when both the Iceberg snapshot and effective mapping spec are unchanged. --no-wait --json returns the run ID, full attempted fingerprint, and force flag from the accepted trigger without polling. Waited --json output preserves the flat BackingDataset fields and adds sync_trigger plus sync_run evidence; failed runs still exit 1. JSON preserves the legacy numeric snapshot_id and also exposes snapshot_id_exact for precision-safe consumers.

The CLI also supports an older API for an ordinary sync. If the trigger omits the exact run ID or fingerprint evidence, waited mode polls the BackingDataset status and labels replay evidence unavailable; --no-wait reports only the legacy trigger fields it actually received. This fallback does not claim an exact run. --force never uses it: the CLI first reads GET /api/v1/backing-datasets/capabilities and exits before the sync POST unless version 2, exact-run polling, and force replay are explicitly advertised. It also requires the accepted trigger to confirm force_replay: true.

A BD with sync_mode: on_materialization is pending when either its Iceberg snapshot or its effective mapping spec changes. An identical snapshot and spec creates a persisted skipped run and performs zero ontology entity writes. Mapping updates are rejected with 409 while a sync is running. Retry after that exact run is terminal; an exact declarative re-apply with no ensure_schema work remains a read-only no-op.

--allow-deprecate permits the ensure_schema: block to narrow the ontology entity type by deprecating fields. Without the flag, narrowing returns 409 (additive evolution only).

Publishing files to an ontology file field ​

A property mapping can carry a file: block, which turns a column into an ontology file field. type stays the physical Iceberg type.

From a URL column — the column holds HTTP(S) URLs the ontology fetches:

yaml
property_mappings:
  - column: code
    property: code
    type: string
    is_primary_key: true
  - column: image_url
    property: image
    type: string
    file:
      mode: url
      identity_column: image_id       # required — a URL is not an identity
      fingerprint_column: image_etag  # or: immutable: true
      filename_column: image_name     # optional
      content_type_column: image_mime # optional
      on_error: fail                  # or: omit (optional target fields only)

Frank sends the complete URL — presigned query strings included — and the ontology fetches, stores, and returns a managed key that Frank attaches to the entity.

From an inline binary column — the column holds the bytes, and Frank uploads them:

yaml
property_mappings:
  - column: code
    property: code
    type: string
    is_primary_key: true
  - column: attachment_bytes
    property: attachment
    type: binary
    file:
      mode: upload
      input: blob
      identity_column: attachment_id      # required — bytes are not an identity
      filename_column: attachment_name    # optional
      content_type_column: attachment_mime # optional
      on_error: fail

No fingerprint_column is needed: Frank computes SHA-256 from the bytes it reads, and that hash drives reuse and replacement. Declare one only when you already carry a trusted content hash.

When a run publishes files, sync adds asset counts to its output:

files_requested         3
files_uploaded          2
files_reused            1
files_attached          3
files_failed            0
files_omitted           0
files_outcome_unknown   0
files_orphan_candidate  1
files_bytes_sent        20480

Two need operator attention. files_outcome_unknown counts assets whose upload outcome was lost in transit: the ontology may hold bytes under a key Frank never saw, so those are quarantined and will not retry on their own. files_orphan_candidate counts superseded ontology keys — Frank never deletes ontology storage, so they stay visible for reconciliation.

files_bytes_sent counts only bytes Frank itself sent, so it stays 0 for URL-mode publication.

Republication is driven by identity_column and the fingerprint, never by the URL: a rotated presigned URL for an unchanged asset uploads nothing, and neither does a rerun over an unchanged blob.

See the ontology guide for the full contract.

frankctl ai compose-pipeline ​

AI-assisted pipeline authoring. Takes a spec describing user intent + source tables + (optionally) the target shape, fires the frank_compose_pipeline Martha workflow, and returns either the raw JSON proposal or a YAML envelope that pipes straight into pipelines apply -f.

bash
# JSON proposal (default) — useful for inspection / piping into other tools
frankctl ai compose-pipeline -f intent.yaml > proposal.json

# YAML envelope — pipes directly into declarative apply
frankctl ai compose-pipeline -f intent.yaml --output yaml > vertical.yaml
frankctl pipelines apply -f vertical.yaml

Spec file shape (JSON or YAML):

yaml
user_intent: "Land OWM air-quality data into the air_quality_observed entity type"
pipeline_name: air_quality_observed       # required for --output yaml
source_tables: [bronze.cm_ave.air_pollution]
target_description: "Promote raw OWM payload into a typed dataset"
target_schema:                            # optional
  - { name: external_id,  type: string }
  - { name: dt,           type: integer }
  - { name: pm2p5,        type: number }
target_sdm_id: fiware:Environment/AirQualityObserved   # optional; flips target_type to "sdm"

--output yaml emits a single kind: Pipeline doc. Source and BackingDataset docs are not auto-generated — the compose endpoint takes already-existing source tables, and the BD shape (iceberg_namespace / property_mappings) isn't yet part of the AI's proposal. Operators add Source and BD docs by hand or via pipelines export after applying.

frankctl ai suggest source-config ​

Paste in raw upstream config (.env, vendor docs, API doc snippets) and get back an inferred pattern_id, redacted source_config, and a suggested name. The other half of the authoring on-ramp — compose-pipeline handles the Pipeline; this handles the Source.

bash
# From a file (JSON/YAML with {raw_text, available_pattern_ids?})
frankctl ai suggest source-config -f spec.yaml

# Inline (short snippets only)
frankctl ai suggest source-config --text "feed_url: https://example.com/rss"

# Via stdin (paste-anywhere workflow)
pbpaste | frankctl ai suggest source-config

# Pipe straight into a Source YAML the operator can `sources create -f`
frankctl ai suggest source-config -f spec.yaml --output yaml > source.yaml

Spec file shape:

yaml
raw_text: |
  REDIS_HOST=cache.example.com
  REDIS_PORT=6379
  REDIS_PASSWORD=secret
available_pattern_ids: [rest_api, kafka, postgres, redis]   # optional hint

Response (JSON, default):

json
{
  "pattern_id": "redis",
  "confidence": 0.9,
  "config": {"host": "cache.example.com", "port": 6379},
  "rationale": "Redis env vars detected",
  "detected_fields": ["host", "port"],
  "suggested_name": "redis_cache",
  "suggested_description": "Redis cache instance",
  "available": true
}

--output yaml emits the kind: Source envelope using suggested_name, pattern_id, and config. When the AI can't infer a pattern_id (confidence < threshold), --output yaml falls back to JSON and exits 5 with the rationale on stderr — the operator picks the pattern manually.

frankctl transforms, frankctl runs, frankctl patterns, frankctl schedules, frankctl datasets, frankctl ai ​

Domain-specific verbs. Run frankctl <noun> --help for the full list.

Diagnosing a transform that won't run ​

When a transform isn't materializing on upstream change, three commands tell you why without touching the database.

frankctl transforms get <id> — runnability at a glance ​

get surfaces the fields the scheduler actually gates on:

lifecycle_stage    ready
last_run_outcome   running        # "running" with no real run in flight = stuck
can_run_now        false          # false here means triggers will 400

can_run_now=false with last_run_outcome=running is the classic zombie: the cached outcome says a run is in flight when none is, so POST /schedule/trigger returns 400 forever and the source-update sensor can never fire it. Heal it with frankctl admin reconcile-runs (below).

frankctl transforms gate-status — the change-gate (ADL-016) ​

Eager transforms fire only when their upstream_version (a hash of their input tables' Iceberg snapshot ids) advances. This lists it per transform:

bash
frankctl transforms gate-status
# ID    NAME          UPSTREAM_VERSION   FIRES_ON_CHANGE
# t1    fires         97d2ec9566272863   yes
# t2    stuck                            NO (cold/unresolved)

upstream_version is empty (the sensor skips) when either the source has never synced (cold start) or an input table FQN doesn't resolve (e.g. a pre-#513 bronze.<ns>.<table> identifier). A whole vertical showing NO usually means the latter.

frankctl admin reconcile-runs — heal stuck runs & zombies ​

Syncs stuck backend runs to their real terminal state and clears zombie last_run_outcome=RUNNING markers (transforms with no run actually in flight), restoring can_run_now.

bash
# Preview only — lists what would be healed, changes nothing:
frankctl admin reconcile-runs --dry-run
# ID    CURRENT   WOULD_HEAL_TO
# t1    running   failed

# Heal (stale runs + zombies):
frankctl admin reconcile-runs

# Also re-check every active run, not just stale ones:
frankctl admin reconcile-runs --force

The periodic reconciliation sweep performs the same zombie heal automatically; this command is the on-demand operator path.

Declarative apply (pipelines apply -f) ​

frankctl pipelines apply -f <file> provisions a complete Frank vertical (Source + optional IdentityPolicy + Pipeline + BackingDataset) from a single multi-doc YAML file. The file is the unit of truth: re-running apply against the same file is idempotent.

File format ​

Each document carries a kubectl-style envelope:

yaml
apiVersion: frank.platform/v1
kind: Source | IdentityPolicy | Pipeline | BackingDataset
metadata:
  name: <string>           # lookup key on apply
  labels: {...}            # optional, free-form
spec:
  # …the existing JSON request body for the resource…

spec is exactly what you'd POST to the corresponding REST endpoint — the envelope is a YAML-only convention. The same JSON wire format still works for clients that don't use YAML.

Worked example ​

air_quality_observed.yaml:

yaml
apiVersion: frank.platform/v1
kind: Source
metadata: { name: owm_air_pollution }
spec:
  name: owm_air_pollution
  pattern_id: rest_api
  source_config:
    base_url: https://api.openweathermap.org/data/2.5
    endpoint: /air_pollution
    query_params: { lat: "41.38", lon: "-8.20" }
  schedule_type: cron
  schedule_value: "0 * * * *"
  streams:
    - name: air_pollution
      sync_mode: full_refresh
      write_disposition: replace
      is_enabled: true
---
apiVersion: frank.platform/v1
kind: Pipeline
metadata: { name: air_quality_observed }
spec:
  name: air_quality_observed
  description: OWM air quality → silver
  # source_ids resolves Source names → UUIDs at apply time
  source_ids: [ owm_air_pollution ]
  schedule_config:
    type: cron
    value: "0 * * * *"
---
apiVersion: frank.platform/v1
kind: BackingDataset
metadata: { name: air_quality_observed }
spec:
  # `pipeline` references the Pipeline doc above by name; the CLI
  # resolves to pipeline_id before POSTing.
  pipeline: air_quality_observed
  iceberg_namespace: silver_air_quality
  iceberg_table: air_quality
  entity_type_id: air_quality_observed
  entity_type_name: Air Quality Observed
  ontology_tenant_id: ts_demo
  primary_key_column: external_id
  property_mappings:
    - { column: external_id,   property: external_id, is_primary_key: true, type: string }
    - { column: date_observed, property: date_observed, type: string }
    - { column: pm2p5,         property: pm2p5,         type: number }
    - { column: no2,           property: no2,           type: number }
  # ensure_schema: Frank manages the ontology entity-type schema for this BD.
  # Field keys must match ^[a-z_][a-z0-9_]*$ — snake_case only.
  ensure_schema:
    display_name: Air Quality Observed
    fields:
      - { field_key: external_id,   field_type: { type: string },   required: true, indexed: true }
      - { field_key: date_observed, field_type: { type: datetime }, required: true }
      - { field_key: pm2p5,         field_type: { type: number },   required: false }
      - { field_key: no2,           field_type: { type: number },   required: false }

An identity-backed field-mapping step adds an IdentityPolicy declaration. A same-file reference contains only its name; Frank reconciles the recipe first, then pins the Pipeline mapping to the exact returned policy ID and version:

yaml
apiVersion: frank.platform/v1
kind: IdentityPolicy
metadata: { name: cira-stop-time-external-id }
spec:
  name: cira-stop-time-external-id
  strategy: passthrough
  source_fields: [_record_id]
  normalizers: []
  emit_format: "cira:bdtransportes:stop_time:{resolved_key}"
  collision_policy: error
---
apiVersion: frank.platform/v1
kind: Pipeline
metadata: { name: cira_transport_stop_times }
spec:
  name: cira_transport_stop_times
  source_ids: [cira-bdtransportes-horarios]
  steps:
    - name: stop_time_rows
      kind: field_mapping
      params:
        field_mappings:
          - target_column: external_id
            mapping_kind: identity
            identity_policy: { name: cira-stop-time-external-id }
            field_order: 0

Use {name, tier, version} for a pre-existing policy that is not declared in the file. Floating external names and tier/version qualifiers on same-file references are rejected. IdentityPolicy declarations create tenant policies; read-only system policies can only be referenced externally and exactly.

Relationship mappings add an execution prerequisite on the BackingDataset metadata. The dependency must match the relationship target and ontology tenant:

yaml
metadata:
  name: air_quality_observed
  dependsOn:
    - entityTypeId: air_quality_station
      ontologyTenantId: ts_demo
spec:
  ontology_tenant_id: ts_demo
  sync_mode: manual
  property_mappings:
    - column: station_key
      property: ref_station
      is_relationship: true
      target_type: air_quality_station
      target_key: name

Apply:

bash
frankctl pipelines apply -f air_quality_observed.yaml --wait
# applying Source/owm_air_pollution...
# applying Pipeline/air_quality_observed...
# applying BackingDataset/air_quality_observed...
# applied 3 doc(s)

Apply semantics ​

  1. Dependency order. Docs are sorted client-side: Source → IdentityPolicy → Pipeline → BackingDataset. File order within a kind is preserved for provisioning. For execution, each relationship mapping must have a matching metadata.dependsOn entry and must explicitly set spec.sync_mode: manual. The dependency graph is currently enforced by the CLI invocation, not persisted as server policy, so omitted mode and on_materialization are rejected before any API request. --wait directly triggers the manual BackingDatasets in topological order and requires an external dependency to exist in the same ontology tenant with status synced. Missing declarations, dependency cycles, and self-referential relationships are also rejected.

  2. Name resolution. Cross-doc references use metadata.name. A Pipeline.spec.source_ids entry that's a name (not a UUID) is looked up against (a) Sources just POSTed in this apply, (b) Sources already in the calling tenant. An identity mapping uses name only when its IdentityPolicy is declared in the same file. A pre-existing policy requires an exact {name, tier, version} reference. External policy references are tenant-aware and all resolve before the first POST; a miss cannot leave an earlier Source partially applied.

  3. Idempotency. Each POST carries ?if-not-exists=true. The server handles existing rows per the matrix below:

    Existing row stateResponse
    Missing at the lookup key201 (create)
    Exists, mutable fields all match200, untouched (no PATCH fires)
    Exists, mutable fields differ200, PATCHed in place
    Exists, immutable fields differ409 with {error: "immutable_diff", existing_id, fields: [{name, existing, requested}]}

    Lookup keys: (tenant_id, name) for Source/Pipeline, (tenant_id, entity_type_id, ontology_tenant_id) for BackingDataset (NULLS NOT DISTINCT). Re-running apply -f against an unchanged file is a true no-op — no row writes, no schedule churn, operational state (cursors, file ledgers, snapshot ids, run counters, last_sync_at) is preserved.

    Source.spec.streams follows the same desired-state rule. Declared streams are matched by name, created or patched in place, and retain cursor/chunk/ counter/run state. Omitted streams are preserved as unmanaged drift and produce a warning; only explicit is_enabled: false disables one. On reapply, omitted fields on a declared existing stream are also preserved; model defaults are used only when creating a missing stream. For a migration, declare each known obsolete enabled stream as disabled so scheduled and direct Source syncs also converge—not just this command's --wait execution.

    IdentityPolicy uses a version-aware desired-state contract:

    Latest tenant/name policyActionResult
    missingcreatecreate version 1
    recipe equalnooppreserve the exact row/version
    recipe changed and never boundpatchupdate that version in place
    recipe changed after first bindingnew-versionappend the next version

    First successful binding freezes a policy monotonically. Policy reads report tenant-live used_in and transform counts, including for shared system policies. A system policy's shared cached count is not overwritten by one tenant's usage; its is_frozen state remains global and monotonic. Returning to zero never makes a frozen recipe mutable or deletable.

    Authored YAML always keeps identity references symbolic; raw policy UUID, version, or hash pins are rejected. Internally, frankctl sends the exact ID/version plus the complete recipe hash returned by reconciliation. The server rechecks that hash before any binding replacement. Pipeline activation locks the Pipeline, then every existing affected Transform in deterministic UUID order with refreshed mapping/source snapshots, then all old and new policy rows in canonical UUID/version pin order. Direct Transform PUT/DELETE locks the Transform first and then policies in that same order. This prevents policy/Transform lock inversion and ensures removed bindings participate in the final used_in refresh, avoiding deadlocks and stale counters. If another apply changed an unfrozen recipe in the gap between resource requests, Pipeline apply returns structured 409 identity_policy_recipe_changed. Re-run the complete file so the policy is reconciled again; do not retry a captured Pipeline request.

    Upgrade migration s43_identity_policy_frozen also recovers historical first-use state from retained PipelineVersion identity mappings. It examines only mapping_kind: identity, recognizes canonical, uppercase, hyphenless, braced, and urn:uuid: UUIDs plus normalized positive version spellings, and never casts malformed JSON values. Invalid historical values stay unmatched rather than aborting the additive migration.

  4. Stops on first failure; the file is not atomic. A 409 (typically immutable_diff) aborts the apply, but docs that succeeded remain in the database. Fix the failing declaration (or deliberately use --allow-recreate, below) and re-run; unchanged resources reconcile as no-ops.

  5. Immutable fields per kind (a change here triggers immutable_diff):

    • Source: pattern_id
    • IdentityPolicy: recipe changes patch an unfrozen version or create a new frozen-policy version as described above; they do not use immutable_diff
    • Pipeline: none beyond tenant_id (everything else PATCHes; step DAG changes go through /versions)
    • BackingDataset: iceberg_namespace, iceberg_table, entity_type_name, schema_library_ref, schema_version, sync_mode, transform_id, pipeline_id

  6. --allow-recreate opt-in escape hatch: on an immutable_diff 409, DELETE the existing row via the existing_id in the response body and re-POST. Operational state is lost on the DELETE — the recreated row starts from a clean cursor, empty file ledger, no run history, status DRAFT/PENDING. Default off; use only when an immutable field genuinely needs to change (e.g. swapping pattern_id on a Source).

  7. --allow-deprecate is passed through to BackingDataset POSTs to permit ensure_schema: narrowing. Default off.

  8. --dry-run prints the apply plan (sorted, with metadata names and the names/count of declared Source streams) without POSTing. It does not pretend to know whether a stream is a create or patch because that requires server state.

  9. --dry-run-server sends the complete document set to the server's DB/catalog preflight without persisting it. Existing rows, immutable diffs, catalog entries, and cross-document references are checked. IdentityPolicy results report create, noop, patch, or new-version plus the exact planned_version; Source results include a per-stream create, patch, noop, or unmanaged plan and the declarative fields that differ. The result describes current state, not a reservation: apply rechecks concurrent changes and can still return a recipe-conflict 409.

  10. --dry-run-server --live-ontology adds a GET-only compatibility check for every locally valid BackingDataset. It verifies the configured ontology tenant/type plus mapped properties, relationship targets, keys, and declared types. Credentials remain server-side. --live-ontology without --dry-run-server is rejected; ordinary server dry-runs never contact the ontology.

  11. --wait executes applied resources behind strict global barriers: all declared Sources must reach exact terminal success before Pipeline transforms start, and all transforms must succeed before any BackingDataset sync is triggered. Evidence is correlated to the exact Source workflow/SyncRun, Dagster/TransformRun, and ontology SyncRun IDs. When a Source document includes spec.streams, --wait passes only its explicitly enabled declared stream names to the Source sync. Preserved unmanaged legacy streams are not executed accidentally. A Source document that omits streams retains the legacy behavior of syncing every persisted enabled stream. Transform success additionally requires an output Iceberg snapshot and row count. Source Bronze main refs are observations taken immediately after the exact Source run completes; they are labelled as observations, not as workflow-to-snapshot provenance.

For machine-readable execution evidence, combine --wait --json. The command emits one JSON object after execution finishes:

json
{
  "applied": [{"kind": "Pipeline", "name": "example", "id": "..."}],
  "wait": {
    "ok": true,
    "sources": [],
    "transforms": [
      {
        "dagster_run_id": "...",
        "transform_run_id": "...",
        "snapshot_id": "123",
        "row_count": 50,
        "status": "completed"
      }
    ],
    "backing_datasets": [
      {
        "workflow_id": "...",
        "ontology_sync_run_id": "...",
        "correlated_transform_run_id": "...",
        "snapshot_id": "123",
        "row_count": 50,
        "bulk_jobs": [
          {
            "job_id": "...",
            "operation": "insert",
            "submitted_count": 50,
            "completed_count": 50
          }
        ],
        "status": "synced"
      }
    ]
  }
}

Snapshot IDs are signed 64-bit identifiers serialized as decimal strings. Treat them as opaque IDs; converting them to JavaScript numbers can round the value and invalidate execution evidence. For ontology SyncRuns, --wait prefers the API's snapshot_id_exact, accepts a string-valued snapshot_id only as an old-API fallback, and rejects numeric-only evidence rather than stringifying a value that JavaScript may already have rounded.

For REST bulk syncs, bulk_jobs comes from the exact persisted ontology SyncRun, not an ephemeral console message or a broad "latest run" query. Only job ID, operation, submitted count, and completed count are stored; request payloads and credentials are excluded. A non-bulk or historical run returns an empty list.

When a BackingDataset is linked to a Pipeline executed by the same command, --wait also requires the ontology SyncRun snapshot to match the exact TransformRun snapshot. An already-running ontology workflow is rejected when that provenance is missing or its snapshot differs.

status: skipped means the ontology was already converged at the same snapshot and no rows were replayed. A successful --wait proves correlated execution and sync completion; use frankctl bd entities (or the target API) for separate entity readback proof.

Adoption workflow (export → review → commit → re-apply) ​

For pipelines that started life in the wizard UI and need to move into git, the round-trip is:

bash
# 1. Export Sources, identity recipes, mappings, BDs, and safe credential metadata.
frankctl pipelines export <pipeline-id> -o yaml > vertical.yaml

# 2. Review locally. frankctl does NOT expand ${ENV_VAR} in Source config.
$EDITOR vertical.yaml

# 3. Commit the reviewed declaration; it contains no credential values.
git add vertical.yaml
git commit -m "adopt air-quality vertical"

# 4. Re-apply the complete declaration.
frankctl pipelines apply -f vertical.yaml

After step 4 the complete non-secret vertical is the Git source of truth. Wizard edits and apply edits coexist as long as nobody changes the files out of band: the next apply PATCHes mutable drift back to the declaration. In vault mode, rotate values separately with sources credentials rotate; the stable reference keeps the manifest and Source operational state unchanged. In current base legacy_inline mode, update credentials through the product's write-only Source create/update input; the dedicated credential verbs fail.

Applying the CIRA Transdev/AveiroBus historical GTFS snapshot ​

The Transdev Source example is intentionally non-applyable because its two S3 credential placeholders are null. Create the Source from a non-secret working copy with those keys removed. On current base legacy_inline, bootstrap through the product's write-only Source create/update input. The sources credentials set step below is only for a vault deployment. Never add values to the manifest:

bash
cp dev_docs/examples/sources/cira-transdev-aveirobus-gtfs/source.non-apply.example.yaml \
  .Codex/cira-transdev-aveirobus-gtfs.yaml
$EDITOR .Codex/cira-transdev-aveirobus-gtfs.yaml

frankctl pipelines apply \
  -f .Codex/cira-transdev-aveirobus-gtfs.yaml \
  --dry-run-server
frankctl pipelines apply \
  -f .Codex/cira-transdev-aveirobus-gtfs.yaml
# Vault mode only:
frankctl sources credentials set <source-id> \
  --name cira-transdev-gtfs --values-file - < credential-values.yaml
frankctl sources discover <source-id>
frankctl sources sync <source-id> \
  --streams agency,calendar,calendar_dates,fare_attributes,fare_rules,feed_info,routes,shapes,stops,stop_times,trips \
  --force-full-refresh --timeout 1800

This is a pinned historical snapshot, not a current publisher feed. The Source uses full_refresh plus atomic replace publication for all eleven declared streams. Empty fare_attributes and fare_rules files remain declared stream identities but may not materialize Iceberg tables and create no ontology instances.

Preflight and run the nine Pipeline/BackingDataset pairs through the existing CIRA ontology types:

bash
frankctl pipelines apply \
  -f pipeline_templates/cira/09_transdev_aveirobus_gtfs.yaml \
  --dry-run-server --live-ontology

frankctl --json pipelines apply \
  -f pipeline_templates/cira/09_transdev_aveirobus_gtfs.yaml \
  --wait --timeout 1800

The service projection deliberately unions identities from calendar and calendar_dates. GTFS permits exception-only services; this snapshot therefore publishes 11 gtfs_service entities, including one service absent from the 10 weekly calendar rows, so all 1,624 calendar-date relationships resolve. The calendar transforms also normalize DLT-inferred numeric YYYYMMDD fields through an integer cast before parsing, avoiding scientific-notation date failures. The bundle has no ensure_schema and never creates or changes ontology schema.

Applying the CIRA Ciclope vertical ​

The repository's Ciclope Source example is intentionally not applyable. This first command must exit with unsupported kind 'SourceExample' before any API call:

bash
frankctl pipelines apply \
  -f dev_docs/examples/sources/cira-ciclope/source.non-apply.example.yaml \
  --dry-run

Inspect the deployed profile, then materialize a non-secret working copy:

bash
frankctl --json patterns get ciclope_soap

cp dev_docs/examples/sources/cira-ciclope/source.non-apply.example.yaml \
  .Codex/cira-ciclope.yaml
$EDITOR .Codex/cira-ciclope.yaml

In the private copy, change apiVersion to frank.platform/v1, change kind to Source, and remove the null username and password keys. Do not add SOAP actions, tickets, raw bodies, or WSDL configuration; ciclope_soap is a closed read-only profile. On current base legacy_inline, bootstrap through the product's write-only Source create/update input. In a vault deployment, use sources credentials set; do not place values in the file or argv.

Declare spec.sync_mode: monolithic and spec.max_items_per_chunk: null on this Source. The historical ends use run_started_at, so their row counts grow as the workflow cutoff advances; the current CICLOPE connector cannot resume safely inside one of those snapshots. Provider-time incremental overlap and natural-key merge are tracked in #613.

Preflight and apply the private Source, capture its returned ID, then prove all nine declared streams through discovery and one explicit Source sync:

bash
frankctl pipelines apply -f .Codex/cira-ciclope.yaml --dry-run-server
frankctl pipelines apply -f .Codex/cira-ciclope.yaml
# Vault mode only:
frankctl sources credentials set <source-id> \
  --name cira-ciclope --values-file - < credential-values.yaml

frankctl sources discover <source-id>
frankctl sources sync <source-id> \
  --streams towers,cameras,meteo_sensors,camera_status,current_afd_alarms,true_afd_alarms,occurrence_entries,realtime_meteo,meteo_history \
  --timeout 1800

frankctl sources get <source-id>
frankctl sources history <source-id>
frankctl datasets list --layer bronze
frankctl datasets preview <dataset-id> --limit 20

The tracked bundle contains five Pipelines and five BackingDatasets. Its live preflight performs GET-only compatibility checks against the existing ontology types. The subsequent --wait run materializes Silver and publishes entity values; the bundle has no ensure_schema, so it does not create or change ontology schema:

bash
frankctl pipelines apply \
  -f pipeline_templates/cira/10_ciclope_emergency.yaml \
  --dry-run-server --live-ontology

frankctl --json pipelines apply \
  -f pipeline_templates/cira/10_ciclope_emergency.yaml \
  --wait --timeout 1800

frankctl backing-datasets entities <emergency-resource-bd-id> --limit 20
frankctl backing-datasets entities <emergency-incident-bd-id> --limit 20

Readback should show tower Points only on emergency_resource. The emergency_incident entities come only from true_afd_alarms and contain code, incident_type, and reported_at, with no inferred location. Use frankctl sources logs <source-id> <run-id> when a Source run reports a transport or provider failure.

pipelines export <id> ​

Round-trips an existing Pipeline, its Sources, referenced tenant IdentityPolicies, and linked BackingDatasets back to multi-doc YAML. Export always strips every registered credential form from source_config. In vault mode it preserves the applicable opaque credential_ref; in legacy_inline there is no reference to emit. The reviewed output is therefore safe to keep in Git:

bash
frankctl pipelines export <pipeline-id> -o yaml > vertical.yaml

Server-set fields (id, created_at, runtime counters, status) are stripped. Source stream declarations are included, but stream IDs, dataset IDs, cursors, iterator chunk state, counters, timestamps, and discovery-learned schema/capabilities are stripped. Complete field-mapping steps preserve their typed mapping kind, source/target fields, transforms, typed literal/context configuration, and field order. Persisted identity UUID/version pins are made portable: tenant pins export as an IdentityPolicy document plus a same-file {name} reference; system pins remain exact {name, tier, version} external references. Credential values and Vault paths are never included in either mode.

ensure_schema: is not included on export — the operator adds the block only if they intentionally want Frank to manage the ontology schema going forward. Reapplying an exported CIRA-style declaration without that block cannot create or edit ontology types.

Direct YAML on any apply-able route ​

The YAML envelope works on the bare REST routes too — the same shape the CLI emits:

bash
curl -X POST https://api.frank.example/api/v1/sources \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/yaml" \
  --data-binary @source.yaml

Request: Content-Type: application/yaml on POST/PUT/PATCH to /api/v1/{sources, identity-policies, pipelines, backing-datasets} is accepted and unwrapped before the route handler sees it.

Response: Accept: application/yaml or ?format=yaml on GET against the same routes returns a YAML-wrapped envelope. JSON is still the default for both directions — the envelope is opt-in.

YAML parsers may resolve an unquoted ISO date or timestamp to a native temporal value. Frank recursively converts those values to ISO-8601 strings before the existing JSON request model validates them, including values nested in objects and arrays. Existing strings, numbers, booleans, and nulls retain their types. YAML-only values that cannot be represented in JSON, such as !!set, fail with a sanitized HTTP 400 that does not echo the submitted spec or credentials.

Exit codes ​

CodeMeaning
0Success
1Generic error
2Usage / config error
3Authentication error
4API / server error
5Validation error (e.g. sandbox failed)
6Config-file error

See also ​

Bronze type drift ​

A sync that fails with Cannot change column type: <field>: <bronze-type> -> <new-type> means Iceberg's union_by_name rejected the latest batch because the extracted value's type doesn't match what the bronze column was first written with. This usually happens when:

  1. A source synced for a while, established the bronze column type (e.g. long), and
  2. The extract pipeline later started emitting a different type for the same field (e.g. double — see the defensive int → float promotion in backend/services/extraction/coercers.py).

Two cures, your pick depending on what you want bronze to look like going forward.

Cure A — declare the field type so the coercer matches bronze ​

Pin the field's JSON-Schema type on the stream so the extract-time coercer fires on the way in (backend/services/extraction/coercers.build_field_coercers). Pinning dt: integer keeps dt as int64 end-to-end and the write matches the existing long column.

bash
# 1. See where bronze and the stream disagree.
frankctl sources streams diff-types <source-id> <stream-id>

# bronze: bronze.tenant_..._<source>.<stream>  (exists=true)
# FIELD  BRONZE  DECLARED  STATUS
# dt     long    (none)    bronze_only       ← will fail next sync
# value  double  (none)    bronze_only

# 2. Declare types matching bronze.
frankctl sources streams set-types <source-id> <stream-id> \
    --field dt=integer

# 3. Verify.
frankctl sources streams diff-types <source-id> <stream-id>
# dt   long  integer  coerce              ← now safe

The PATCH is partial-merge: it only touches the properties you pass, so an existing 50-field schema from discovery stays intact when you pin one field.

Cure B — widen the bronze column (deferred, see #469 slice B) ​

When you actually want bronze to take the wider type going forward (e.g. an integer field truly needs to become a float), the right fix is a bronze-side widen via PyIceberg update_schema().promote(...) for legal Iceberg promotions or a CTAS-and-atomic-swap for the others. Not shipped yet — tracked in issue #469.

Cure C — preventive (deferred, see #469 slice C) ​

The defensive int → float promotion in coercers.py is an extraction-side default that prevents an unrelated PyArrow inference bug. Teaching it to skip the promotion when bronze already has a narrower type pinned would prevent this class of incident at the source. Also tracked in #469.

Frank — low-code EL/T for the lakehouse.