Operations

Frank is built to be operated as a set of clear runtime surfaces: API, UI, source worker, transform worker, Temporal workflows, Dagster assets, Iceberg tables, logs, traces, and run records.

Services

Service	Purpose
api	FastAPI application, route registration, pattern sync, schema library access, AI endpoints, and admin APIs.
ui	SvelteKit application for source, transform, pipeline, model, ontology, and settings workflows.
source-worker	Temporal worker for source discovery and extraction.
transform-worker	Temporal worker for transform lifecycle and reconciliation work.
worker	General Temporal worker for AI and platform workflows.
Dagster	Asset materialization, schedules, sensors, and pipeline execution visibility.
Temporal	Durable workflow execution for async source, transform, and orchestration jobs.
Iceberg REST + MinIO/S3	Lakehouse catalog and object storage.
Trino	Query engine for transform execution and previews.
Loki	Persistent log querying for run details.
OpenTelemetry collector	Trace export for API and worker paths.

Local stack

cd ../common-infra
docker-compose up -d

cd ../frank-low-code-pipeline
make up
make status

Common access points:

Surface	URL
API docs	http://localhost:8002/docs
UI	http://localhost:5175
Health	http://localhost:8002/health
Dagster	http://localhost:3000 or configured Dagster URL

Startup work

API startup performs the platform initialization that should happen once per deploy:

• Runs through FastAPI lifespan setup.
• Initializes Iceberg client.
• Ensures raw namespace.
• Initializes AI transformer services.
• Initializes FIWARE SDM registry and schema libraries.
• Syncs source patterns from JSON files into the database.
• Syncs SQL transforms and transform patterns from filesystem config.
• Registers transform event listeners for Dagster code location reloads.
• Initializes OpenTelemetry and Langfuse instrumentation.
• Registers the complete API router set.

Run lifecycle

Frank stores lightweight run summaries in Postgres and sends detailed work to the relevant runtime.

Work	Runtime	User-facing records
Source discovery	Temporal source worker	Discovery workflow status.
Source sync	Temporal source worker + Iceberg	Sync run history, logs, source status.
Transform materialization	Dagster + API callback	TransformRun records, Dagster run ID, logs, lineage.
Pipeline sandbox	API + worker orchestration	Sandbox workflow status and step results.
Ontology sync	Temporal / Dagster sensor path	OntologySyncRun records and backing dataset history.
AI assistance	Martha workflow execution	AI trace/execution IDs and structured response payloads.

Logs

Useful CLI commands:

frankctl sources logs <source-id> <run-id> -f
frankctl transforms logs <transform-id> <run-id> -f
frankctl runs get <workflow-id>
frankctl runs wait <workflow-id>

Useful Compose commands:

make logs
make logs-api
make logs-ui
docker-compose logs -f source-worker
docker-compose logs -f transform-worker

The API and workers use structured JSON logging so Loki queries can filter by fields such as workflow ID, Dagster run ID, transform ID, source ID, and trace ID.

Traces

OpenTelemetry is initialized in the API and workers. Dagster-triggered transform paths, source worker paths, and AI paths include trace context where available.

Relevant env:

OTEL_EXPORTER_OTLP_ENDPOINT=alloy:4317

Source operations

Operational checklist:

1. Source is ready or active.
2. Discovery schema is current.
3. Streams are enabled and configured.
4. Incremental streams have cursor fields.
5. Merge streams have primary keys.
6. Target config matches the desired Bronze namespace/table convention.
7. Sync history shows successful runs.

CLI:

frankctl sources get <source-id>
frankctl sources streams list <source-id>
frankctl sources history <source-id>
frankctl sources sync <source-id>

Transform operations

Operational checklist:

1. Transform is hydrated.
2. can_run_now is true in the API/UI.
3. Current artifact runtime matches the expected execution engine.
4. Dagster code location has loaded the asset.
5. Last run outcome is not already running.
6. Logs are available for the run.
7. Output table and lineage edges match expectations.

CLI:

frankctl transforms get <transform-id>
frankctl transforms trigger <transform-id>
frankctl transforms runs <transform-id>
frankctl transforms logs <transform-id> <run-id>

Pipeline operations

Pipeline deployment path:

1. Draft or update pipeline.
2. Validate DAG.
3. Run sandbox.
4. Review step results.
5. Activate.
6. Monitor runs.

CLI:

frankctl pipelines get <pipeline-id> --include-version
frankctl pipelines validate <pipeline-id> --timeout 600

Ontology operations

Before syncing a backing dataset:

1. Entity type exists and is the intended version.
2. Iceberg table exists and has expected columns.
3. Property mappings include the primary key column.
4. Relationship mappings include target type and target key.
5. Health check passes.
6. Sync history is reviewed after trigger.

API:

GET  /api/v1/backing-datasets/{id}/health
POST /api/v1/backing-datasets/{id}/sync
GET  /api/v1/backing-datasets/{id}/sync-history

Key environment variables

Area	Variables
API and auth	KEYCLOAK_URL, KEYCLOAK_REALM, KEYCLOAK_CLIENT_ID, KEYCLOAK_ISSUER, CORS_ALLOWED_ORIGINS
Database	POSTGRES_HOST, POSTGRES_PORT, POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD
Iceberg/S3	ICEBERG_CATALOG_URI, ICEBERG_CATALOG, AWS_ENDPOINT_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION
Temporal	TEMPORAL_HOST, TEMPORAL_PORT, TEMPORAL_NAMESPACE, task queue variables
Dagster	DAGSTER_URL
Logs/traces	LOKI_URL, LOKI_AUTH_TOKEN, OTEL_EXPORTER_OTLP_ENDPOINT
AI	MARTHA_API_URL, MARTHA_KEYCLOAK_URL, MARTHA_CLIENT_ID, MARTHA_CLIENT_SECRET
Ontology	ONTOLOGY_ENABLED, ONTOLOGY_SERVICE_URL, ONTOLOGY_API_KEY, ONTOLOGY_TENANT_ID
Pattern registry	PATTERN_WEBHOOK_SECRET, PATTERN_ADMIN_SECRET

Maintenance commands

make up
make down
make status
make logs
make build
make build-no-cache
make init-iceberg
make init-db
make init-sdm
make test-iceberg

For API route-level checks:

curl http://localhost:8002/health
curl http://localhost:8002/api/v1/status
curl http://localhost:8002/api/v1/services/health