Pattern catalog

Patterns are Frank's connector templates. Each one declares how to extract data from a particular kind of source. The pattern catalog is the lookup table for Source.pattern_id in your apply YAML — pick a pattern, fill in its source_config, and Frank wires up the extraction engine, discovery, and incremental cursor logic for you.

# Live list of patterns on your deployment:
frankctl patterns list

# Per-pattern config schema (what source_config keys it accepts):
frankctl --json patterns get <pattern_id>

This page is curated reference. The authoritative files are backend/config/patterns/*.json for framework/legacy patterns and backend/connectors/*/pattern.json for co-located provider profiles.

At a glance

Pattern	Category	Engine	Typical use
rest_api	API	dlt	Any HTTP REST endpoint with pagination.
rss	API	dlt	RSS / Atom feeds.
graphql	API	dlt	GraphQL queries.
sftp_bulk	File	dlt	Directory of files on SFTP.
s3	File	dlt	S3 / S3-compatible buckets.
archive	File	dlt	Downloadable archives (ZIP / TAR / .gz).
gtfs_static	Transport	dlt	GTFS Static ZIP feeds with one Bronze stream per table.
ciclope_soap	API	dlt	Read-only INOV Ciclope tower, alarm, occurrence, and meteo streams.
filesystem	File	dlt	Local or NFS-mounted filesystem.
postgres	Database	dlt	PostgreSQL with WAL CDC.
mysql	Database	dlt	MySQL with binlog CDC.
mongodb	Database	dlt	MongoDB collections.
mssql	Database	dlt	SQL Server.
bigquery	Warehouse	dlt	BigQuery query / table extract.
snowflake	Warehouse	dlt	Snowflake query / table extract.
redshift	Warehouse	dlt	Redshift query / table extract.
databricks	Warehouse	dlt	Databricks Delta table extract.
kafka	Stream	dlt	Kafka topic consumption.
google_sheets	SaaS	dlt	Google Sheets cells.
salesforce	SaaS	airbyte	Salesforce objects.
hubspot	SaaS	airbyte	HubSpot CRM.
stripe	SaaS	airbyte	Stripe payments + customers.
zendesk	SaaS	airbyte	Zendesk tickets.
jira	SaaS	airbyte	Jira issues + projects.
github	SaaS	airbyte	GitHub repos + issues.
slack	SaaS	airbyte	Slack messages + channels.
notion	SaaS	airbyte	Notion pages + databases.
airtable	SaaS	airbyte	Airtable bases.
google_ads	SaaS	airbyte	Google Ads reporting.
google_analytics	SaaS	airbyte	GA4 reporting.

Patterns commonly used in prod

These are the patterns most prod sources route through today. Worth knowing in detail.

rest_api

The workhorse. Use this for any HTTP endpoint that returns JSON, including OWM, ipma, ogc_api_wfs, custom ministry APIs, etc.

Minimum source_config:

source_config:
  base_url: https://api.example.com
  endpoint: /v1/observations

Common extensions:

Key	Purpose
query_params	Map of fixed, literal query params. frankctl does not expand environment variables in Source config.
headers	Map of fixed request headers (e.g. Authorization).
record_path	Dotted path into the JSON response to find the records array. Default data or records. Use list for OWM-shaped responses.
pagination	`{type: page

Auth shapes the engine knows: Basic, Bearer, API key in header or query param, Keycloak passthrough. See the auth: examples in dev_docs/examples/pipelines/.

rss

Dead simple. Just feed_url:

source_config:
  feed_url: https://feeds.feedburner.com/example

Items land in bronze with title, link, published_at, summary, guid, plus whatever extension fields the feed includes. Sync mode is full_refresh; the GUID is the natural dedup key.

sftp_bulk

Directory of files on SFTP. Use sync_mode: iterator on the Source so the workflow checkpoints between chunks (default monolithic will time out on directories of >~5k files).

Credential-bearing examples in this reference show the immutable-release vault manifest shape. On current mutable/base Compose (SOURCE_CREDENTIAL_MODE=legacy_inline), omit credential_ref and provide the declared values through the write-only Source create/update input. Inline mode persists them in Source.source_config while read/export stay redacted; Vault lifecycle commands are unavailable. Neither mode makes credential values safe for tracked YAML.

# vault mode only
credential_ref: <tenant-owned-opaque-reference>
source_config:
  host: sftp.example.com
  port: 22
  username: ingestion-user
  path_prefix: /inbox/2026
  file_glob: "*.csv"
  file_format: csv
  fetch_concurrency: 5

Frank's file-ledger v2 skips files it's already committed, but re-reads any file whose mtime OR size changed. See the sftp-file-walk/ example for the full shape.

s3

S3 or S3-compatible bucket. Same shape as sftp_bulk but with bucket / prefix instead of host / path_prefix.

# vault mode only
credential_ref: <tenant-owned-opaque-reference>
source_config:
  bucket: my-bucket
  prefix: incoming/2026/
  file_glob: "*.parquet"
  file_format: parquet
  region: eu-west-1

postgres

PostgreSQL with WAL CDC for incremental sync.

# vault mode only
credential_ref: <tenant-owned-opaque-reference>
source_config:
  host: pg.example.com
  port: 5432
  database: production
  username: frank_reader
  schemas: [public, analytics]
  cdc_enabled: true
  replication_slot: frank_repl

These are Git-safe vault shapes. Values for fields declared with credential: true or type: password are written separately through the Vault-only frankctl sources credentials set|rotate --values-file - commands. In legacy_inline, the write-only create/update input persists those fields in source_config, but readback and export redact them. frankctl performs no ${VAR} interpolation in either mode.

Discovery enumerates tables in the listed schemas. The Stream's primary_key_path becomes the dedup key on merge writes.

archive

Downloadable archive (ZIP / TAR / TAR.gz). Use this for sources like GTFS-static where the publisher dumps everything in one big bundle.

source_config:
  archive_url: https://example.com/gtfs/static.zip
  format: zip
  file_glob: "*.txt"      # which files inside the archive to extract
  file_format: csv

The archive itself is the unit of progress — there's no incremental-within-an-archive concept. To re-fetch, increment the archive_url query string (e.g. ?version=2).

gtfs_static

gtfs_static discovers a GTFS Static ZIP and publishes one full_refresh/replace Bronze stream per *.txt table. It accepts ordinary HTTP(S) download authentication or an S3/MinIO object:

# vault mode only
credential_ref: <tenant-owned-opaque-reference>
source_config:
  auth_type: s3
  bucket: frank-seed
  object_key: transport/operator/feed.zip
  region_name: us-east-1
  file_glob: "*.txt"
  file_format: csv

For a current publisher feed, use the default feed_freshness_policy: current. If feed_valid_through is present and already past, Frank rejects the Source before download rather than presenting stale schedules as current data.

An intentionally historical or test snapshot must declare immutable provenance:

source_config:
  feed_freshness_policy: historical
  feed_valid_through: "2025-12-31"
  snapshot_label: operator-evaluation-2024-10-24
  source_git_commit: 0123456789abcdef0123456789abcdef01234567
  artifact_sha256: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
  artifact_size_bytes: 293657

Historical mode requires all four provenance fields. The commit must be a full 40-character lowercase hexadecimal Git commit, the digest a full lowercase SHA-256, and the size a positive integer. Frank verifies the downloaded bytes against the declared size and digest for both HTTP and S3. A mismatch is terminal and no table is published.

In legacy_inline, S3 access keys are write-only Source values persisted in source_config but redacted on read/export. In vault, they are stored separately and represented by credential_ref. There is no credential_source or environment-variable interpolation mode. Keep transient credential input private, apply the Source before a secret-free tracked Pipeline/BackingDataset bundle, and never place access keys in Git, issue text, logs, or CLI arguments.

ciclope_soap

ciclope_soap is Frank's registered, read-only profile for the CIRA INOV Ciclope WCF service. It reuses an internal SOAP 1.1 core for bounded HTTP, envelope, XML, and Fault handling. That core is not a Source pattern: operators cannot select it or configure an action, WSDL, raw body, envelope, ticket, or arbitrary header. The profile owns the closed read-operation allow-list and cannot invoke provider Set* or image operations.

The profile exposes exactly nine full_refresh/replace Bronze streams:

• towers, cameras, meteo_sensors, and camera_status;
• current_afd_alarms and true_afd_alarms;
• occurrence_entries; and
• realtime_meteo and meteo_history.

Realtime and history resolve tower selectors differently:

• realtime_meteo: non-empty tower_ids is an exact call list. Empty tower_ids dynamically derives unique sensor-bearing inventory towers.
• meteo_history: both selectors empty means all exact inventory (tower_id, parameter_type) pairs; only one selector means filter those discovered pairs; both selectors non-empty means their strict Cartesian product without inventory discovery.

Discovery records the single algorithm active for the parsed Source config under x-ciclope.target_resolution, including its mode, target_fields, and config_inputs. Inventory-backed modes also expose inventory_source as GetInstalledTowers/MeteoSensorsInstalled; explicit realtime and both-explicit history omit it because they bypass inventory. The sibling x-ciclope.source_operation is the sole value-operation field and remains the emitted row's source_operation.

Failure-to-empty conversion is deliberately narrow. Provider DataNotAvailable means zero rows only for true_afd_alarms and occurrence_entries. A bounded SearchMeteoValues2 window means zero rows only when every configured attempt ended in a connection reset. A timeout, mixed failure sequence, unbounded history call, or reset from another operation remains an error. SOAP POST is one attempt by default; only trusted profile code can mark a fixed operation retry-safe.

For continuously advancing historical snapshots, set occurrence_end_datetime and/or history_end_datetime to run_started_at. Frank freezes one timezone-aware cutoff when the Source workflow starts, sends it as the meteo-history upper bound, and applies it as the occurrence upper bound. Retries reuse that exact cutoff. A literal ISO-8601 end remains available for reproducible fixed backfills.

The tracked example at dev_docs/examples/sources/cira-ciclope/source.non-apply.example.yaml uses the unsupported frank.platform/example/v1 plus SourceExample envelope, so frankctl pipelines apply rejects it before any API call. Copy it to a working file, change the envelope to frank.platform/v1 plus Source, remove the null username and password keys, and apply the non-secret Source. On current legacy_inline, provide the two declared values through the product's write-only Source create/update input; the credential lifecycle endpoints are unavailable. In vault, use frankctl sources credentials set and keep only the returned opaque credential_ref in the Git-owned Source manifest. There is no username_env/password_env mode or ${VAR} expansion. Apply the tracked pipeline_templates/cira/10_ciclope_emergency.yaml bundle only after the credential-backed Source is ready.

The tracked bundle maps tower equipment to emergency_resource, including a Point derived from valid tower coordinates. It maps only provider-confirmed true_afd_alarms to emergency_incident (code, incident_type, and reported_at) and deliberately omits incident location: a tower identifies the detector, not the fire coordinates. The other seven streams remain Bronze-only. The bundle publishes entities through existing ontology types; it does not create or change ontology schema.

Less-common patterns

Patterns from the at-a-glance table not detailed above (graphql, google_sheets, all of the SaaS connectors, etc.) follow the same declare-source_config pattern. Run frankctl --json patterns get <pattern_id> on your deployment to see the exact schema, or inspect backend/config/patterns/<pattern_id>.json and backend/connectors/<pattern_id>/pattern.json in the repository.

Patterns NOT in the public catalog

Frank ships provider profiles that this curated page does not describe in detail (for example eredes_sgl, bpstat, ren_datahub, and ipma). Their co-located packages under backend/connectors/ are the code reference. Use frankctl patterns list to confirm whether a particular deployment has seeded the profile before declaring a Source.

Authoring a new pattern

New provider-specific connectors should be one co-located package under backend/connectors/<id>/: __init__.py exports its ConnectorSpec and lazy builder/discovery references, while pattern.json, builder.py, and discovery.py stay beside it. Frank's scanner registers that package without manual registry edits. Framework and older split connectors remain under backend/config/patterns/, backend/services/dlt/builders/, and backend/services/discovery/.

Reusable protocol machinery is not itself a pattern. Keep it under backend/services/ and expose only closed provider profiles that own their operation, authentication, retry, and response semantics. See dev_docs/solutions/custom-connector-guide.md for the complete contract and tests.