Architecture Overview

Frank is a low-code EL/T platform for building governed lakehouse data products and publishing them into an ontology.

It combines four layers:

1. Experience layer -- SvelteKit UI, frankctl, Python pattern CLI, and API.
2. Control plane -- FastAPI, metadata models, pattern registries, schema libraries, and auth.
3. Execution plane -- Temporal workers, Dagster assets, Trino execution, Kubernetes Python runner, and Martha AI workflows.
4. Data plane -- Iceberg catalog, S3/MinIO object storage, Bronze/Silver/Gold tables, and ontology-core-v2.

The picture

                              Builders and operators
                UI              frankctl             API / CI
                 |                  |                   |
                 +------------------+-------------------+
                                    |
                                    v
                          FastAPI control plane
         +--------------------------+---------------------------+
         |                          |                           |
         v                          v                           v
  Source registry            Transform registry           Pipeline registry
  Pattern catalog            Artifact hydration           Versioned DAGs
  Stream config              Runtime metadata             Sandbox/activation
         |                          |                           |
         v                          v                           v
  Temporal source worker     Dagster + transform worker    Dagster / sandbox
  Airbyte / dlt              Trino / dbt / Python runner   Step execution
         |                          |                           |
         +--------------------------+---------------------------+
                                    |
                                    v
                         Apache Iceberg lakehouse
                    Bronze -> Silver -> Gold datasets
                                    |
                                    v
                      Backing datasets and ontology sync
                                    |
                                    v
                            ontology-core-v2

Control plane

The FastAPI application owns product state:

• Sources and streams.
• Source pattern definitions.
• Transform specs, sources, mappings, artifacts, runs, and lineage.
• Transform pattern registry.
• Pipeline versions, steps, edges, and sandbox results.
• Schedules.
• Dataset browsing.
• Schema libraries.
• Ontology entity type proxy.
• Backing datasets and ontology sync history.
• Identity policies.
• AI routes backed by Martha.

Postgres stores metadata. Iceberg stores data. The API derives tenant scope from auth and passes service identity where workers need to call back into protected endpoints.

Source execution

Source execution is asynchronous and handled by Temporal source workers.

Source pattern -> Source -> Discovery -> Streams -> Sync -> Bronze tables

Airbyte patterns use Frank's native protocol adapter and separately owned, digest-approved connector containers behind the typed execution broker. The source and discovery workers have no Docker API access. dlt patterns use Python-native source builders for REST, GraphQL, filesystem, Kafka, and related lightweight sources.

Both engines implement the same extraction contract:

• Discover stream schemas.
• Extract selected streams.
• Apply data envelope metadata.
• Track cursors.
• Batch writes.
• Write to Iceberg through shared naming helpers.
• Return structured sync status and logs.

Transform execution

Transforms separate design-time specs from runtime artifacts.

Transform spec -> Hydration -> Artifact -> Materialization -> TransformRun

The transform spec stores sources, mappings, pattern params, target schema, materialization, and incremental mode. Hydration renders executable content:

• Trino SQL.
• dbt-style SQL.
• Python-runner files and container config.
• Future runtime renderers such as Flink SQL.

The recommended execution path is Dagster-first:

API trigger -> Dagster materialization -> asset calls API /execute -> run status sync

This gives operators a Dagster run, asset key, logs, and consistent scheduling behavior.

Pipeline execution

Pipelines are versioned DAGs over transform steps.

Pipeline -> PipelineVersion -> PipelineStep + PipelineStepEdge

Frank validates DAGs with topological sorting and cycle detection. Versions are immutable and content-hashed. Sandbox runs execute a version before activation. Activation links or creates transforms for each step and promotes the pipeline to an operational state.

AI execution

Frank owns workflow definitions for AI assistance and seeds them into Martha.

Frank API -> Martha workflow -> LLM / tools -> structured result -> Frank spec

AI routes return typed payloads for schema matching, field mapping, pattern params, SQL review, code generation, CI fixes, transform publishing, and pipeline composition.

Ontology execution

Ontology integration turns curated Iceberg tables into semantic entities.

Silver/Gold table -> BackingDataset -> OntologySyncRun -> ontology-core-v2 entities

Backing datasets map columns to entity properties and relationships. Sync runs track independent data-snapshot and effective-SyncSpec checkpoints, replay reasons, cursors, rows synced, workflow IDs, logs, and drift signals. Mapping or target changes therefore replay even when the Iceberg snapshot is unchanged; only identical data and spec become a zero-write no-op.

Temporal keeps the pre-existing V1 workflow, activity names, and ontology-sync-task-queue immutable while old histories drain. Dual-checkpoint syncs use the separate OntologySyncWorkflowV2, *_v2 activities, ontology-sync-v2-task-queue, and ontology-worker-v2. Deployment expands the database first, proves both pollers, then routes new API starts to V2. Rollback routes new starts to V1 before draining V2; the expanded schema remains until no V2 code or history can use it.

Observability

Frank uses:

• Structured logs for API and worker events.
• Loki for persistent run logs.
• OpenTelemetry for API and worker traces.
• Dagster run IDs for materialization tracking.
• Temporal workflow IDs for async operations.
• TransformRun, SyncRun, and OntologySyncRun records for fast UI/API history.

Design principles

EL and T are separate

Sources sync raw data. Transforms model data. A source can be active without a transform; a transform can remain usable while an upstream source needs attention.

Specs and artifacts are separate

Users edit transform specs. Hydration produces artifacts. Execution uses artifacts. This keeps design iteration away from runtime stability.

Patterns are product surface

Source and transform patterns are not hidden implementation details. They are how Frank expands connector coverage and transform capability while keeping UI and API behavior consistent.

Ontology is a publication layer

Frank does not treat ontology sync as an afterthought. Backing datasets, identity policies, entity type browsing, mapping suggestions, health checks, and sync history are first-class product features.