Ontology Integration

Frank turns curated Iceberg tables into semantic entities. The ontology integration lets teams publish pipeline outputs into ontology-core-v2 so applications can consume typed, versioned, relationship-aware data.

The model

text

Gold / Silver Iceberg table
        |
        v
Backing dataset
        |
        v
Ontology entity type
        |
        v
Ontology entities

A backing dataset says: this Iceberg table backs this ontology entity type, using these column-to-property mappings and this primary key.

Entity types

Entity types are schemas served by ontology-core-v2. Frank proxies the entity type surface so data builders can work inside the same UI and API:

http

GET    /api/v1/ontology/status
GET    /api/v1/ontology/entity-types
GET    /api/v1/ontology/entity-types/domains
GET    /api/v1/ontology/entity-types/{code}
POST   /api/v1/ontology/entity-types
POST   /api/v1/ontology/entity-types/{code}/versions
PATCH  /api/v1/ontology/entity-types/{code}
DELETE /api/v1/ontology/entity-types/{code}
GET    /api/v1/ontology/entity-types/{code}/versions

Entity types can include fields and relationships. Frank synthesizes relationship references into field-like mapping targets so users can map station_name or route_id style columns into relationship refs during backing dataset setup.

Backing datasets

A backing dataset contains:

Field	Meaning
`iceberg_namespace` / `iceberg_table`	The materialized table to publish.
`entity_type_id` / `entity_type_name`	The ontology type being backed.
`schema_library_ref`	Optional source schema reference such as `fiware:Transportation/Vehicle`.
`property_mappings`	Column-to-property mapping array.
`primary_key_column`	Stable entity key column.
`title_key_column`	Human-readable entity label column.
`sync_mode`	When the dataset should publish.
`cursor_column`	Optional incremental sync cursor.
`transform_id` / `pipeline_id`	Optional lineage back to the producer.

Backing dataset lifecycle:

text

pending -> syncing -> synced
synced -> syncing
synced -> needs_remapping
needs_remapping -> pending
error -> pending | syncing

Property mappings

Mappings are explicit and reviewable:

json

[
  {
    "column": "vehicle_id",
    "property": "id",
    "is_primary_key": true,
    "type": "string"
  },
  {
    "column": "observed_at",
    "property": "dateObserved",
    "type": "datetime"
  },
  {
    "column": "station_name",
    "property": "ref_station",
    "is_relationship": true,
    "target_type": "station",
    "target_key": "name"
  }
]

Relationship mappings let the sync activity resolve business keys into ontology entity UUIDs.

File publication

An Iceberg column can populate an ontology file field in one of two ways: the column holds a URL the ontology fetches, or it holds the bytes and Frank uploads them. Add a nested file block to the mapping. type stays the physical Iceberg type — the file block is what selects file semantics.

URL mode

A string column holding an HTTP(S) URL:

yaml

property_mappings:
  - column: image_url
    property: image
    type: string
    file:
      mode: url
      identity_column: image_id
      fingerprint_column: image_etag
      filename_column: image_name
      content_type_column: image_mime
      on_error: fail

Frank passes the complete URL — including the query string a presigned URL needs — to the ontology, which fetches and stores the bytes and returns a managed key. Frank writes that key into the entity. Frank never fetches, rewrites, strips, or stages the bytes or the URL itself, and it never writes a raw URL into a file field.

Upload mode (inline binary)

A binary column holding the file's bytes:

yaml

property_mappings:
  - column: attachment_bytes
    property: attachment
    type: binary
    file:
      mode: upload
      input: blob
      identity_column: attachment_id
      filename_column: attachment_name
      content_type_column: attachment_mime
      on_error: fail

Frank reads the bytes, computes their SHA-256, and uploads them to the same ontology endpoint as multipart. The bytes stay in the sync activity's own memory for the length of one row: they never appear in a workflow input or result, a Temporal search attribute, memo or heartbeat, a log, a span, an error message, a persisted sync spec, or a ledger row.

fingerprint_column is optional here. With no column declared, the SHA-256 Frank computes from the bytes it read is the fingerprint, so an unchanged blob reuses the existing key and a changed one publishes a new asset. Declare a column only when you already carry a trusted content hash.

filename_column and content_type_column travel on the multipart part itself, which is where the ontology reads them; with neither declared, the upload defaults to upload and application/octet-stream. They are metadata hints and do not bypass the ontology's own controls.

Block reference

Key	Required	Meaning
`mode`	yes	`url` (the ontology fetches) or `upload` (Frank sends the bytes).
`input`	upload only	`blob` — the mapped column holds the bytes. Not accepted for `mode: url`. Object references are rejected until their own release lands.
`identity_column`	yes	Stable asset identity. Neither an expiring URL nor the bytes themselves is an identity.
`fingerprint_column`	conditional	Content/version fingerprint, e.g. an ETag. Required in URL mode unless `immutable: true`. Optional in upload mode, where Frank computes SHA-256 from the bytes instead.
`immutable`	url only	Declares that the bytes never change for a given identity. Mutually exclusive with `fingerprint_column`, and rejected for upload mode — Frank always has a real content hash there.
`filename_column`	no	Original filename passed to the ontology.
`content_type_column`	no	MIME type passed to the ontology.
`on_error`	no	`fail` (default) surfaces a publication failure. `omit` drops the value; allowed only against an optional ontology field, and always visible in run evidence.

The physical type must match the mode: a string type (string, varchar, char, text) for url, and a binary one (binary, varbinary, blob, bytes, bytea) for upload with input: blob. The mapped file column can never double as identity_column, fingerprint_column, filename_column or content_type_column — those are persisted, logged and exported, and neither a credential-bearing URL nor raw bytes belongs on those surfaces.

A NULL column value means the row has no asset: the property is dropped and the entity keeps whatever it already had. In upload mode a zero-length blob is a data error, not an absent asset, and follows on_error.

Republication and retry

Whether a file is republished depends on identity_column and the fingerprint, never on the URL or the bytes' position. A rotated presigned URL for the same identity and fingerprint uploads nothing and keeps the existing key; so does a rerun over an unchanged blob.

The ontology's file upload has no idempotency key and mints a new key per accepted upload, so Frank keeps a durable per-asset ledger:

The returned key is persisted before the entity is written. If the entity write fails, the retry attaches the same key with zero re-uploads.
Concurrent runs serialize per asset, so one claim yields at most one acknowledged upload.
A changed fingerprint uploads once, attaches the new key, and leaves the previous key visible as an orphan candidate. Frank never deletes ontology storage.
An unknown transport outcome — a lost response, a gateway error — is quarantined rather than retried, because bytes may already be stored under a key Frank never saw. Quarantine is not softened by on_error: omit. It needs an operator to inspect the ontology target.

File-bearing rows always take the per-row REST path. The gRPC bulk path is insert-only and the REST bulk path has no sequencing point between upload and entity write, so neither can carry this contract.

Each sync run reports asset outcomes: requested, uploaded, reused, attached, failed, omitted, outcome-unknown, orphan-candidate, and bytes-sent. frankctl backing-datasets sync prints them as files_* rows when a run published files. files_bytes_sent counts only bytes Frank itself sent, so it is always 0 for URL-mode publication.

Provenance

Frank-owned persistence, logs, spans, errors, and exports never contain a URL query, fragment, or credential, and never contain file bytes. Every published asset carries a credential-free provenance record instead:

Mode	Scheme	Host	Path
`url`	`https`	URL host	URL path, no query
`upload`	`iceberg`	Iceberg namespace	`table.column`

Both are stored with a SHA-256 digest of the locator, so one query answers "where did these bytes come from" across both modes, and the scheme tells you which mode produced the row.

SSRF, redirect, MIME, size, and egress controls remain the ontology's. Private and internal addresses, s3://, worker-local paths, inline blobs, and object references are not URL mode.

Mapping assistance

Frank can suggest backing dataset mappings:

http

POST /api/v1/backing-datasets/suggest-mappings

The suggestion request includes the Iceberg table and target entity type. Frank uses table schema, target property names, and AI assistance to propose column-to-property matches.

Sync

Backing datasets sync rows from Iceberg into ontology-core-v2. The sync path tracks:

Workflow ID and workflow run ID.
Status: pending, running, synced, error, skipped.
Started and completed timestamps.
Rows synced.
Snapshot ID, including an exact decimal-string representation.
Full vs incremental sync.
Error message.
Trigger source.
Replay reason and whether force replay was requested.
Attempted and successfully applied effective-SyncSpec fingerprints.

Sync-run JSON keeps numeric snapshot_id for existing clients and adds snapshot_id_exact so JavaScript and other number-limited consumers can read the full 64-bit Iceberg snapshot ID without rounding.

Frank treats the Iceberg snapshot and effective SyncSpec as independent checkpoints. A change to mappings, resolved runtime transforms, entity target, REST/gRPC endpoint, or effective ontology tenant replays the unchanged snapshot through REST upsert. The next run skips with zero entity writes only when both checkpoints match. Failed runs retain the previously applied fingerprint. Mapping or ensure_schema changes return 409 while a sync is running, so a later spec cannot overtake and regress the frozen run's ontology/checkpoint. An exact declarative re-apply without schema convergence remains a read-only 200 and does not disturb the running sync.

REST bulk and per-row safety

When a compatible gRPC endpoint is configured, Frank keeps using the gRPC batch path. Otherwise, REST selection is cardinality-aware:

Environment variable	Default	Meaning
`ONTOLOGY_REST_BULK_THRESHOLD_ROWS`	`1000`	Estimated rows at or above this value require REST bulk. A missing estimate is treated as high-cardinality.
`ONTOLOGY_REST_BULK_MAX_ITEMS`	`10000`	Maximum entities in one bulk request.
`ONTOLOGY_REST_BULK_MAX_PAYLOAD_BYTES`	`52428800`	Maximum UTF-8 bytes in the exact serialized `{items: [...]}` request.
`ONTOLOGY_REST_BULK_PAGE_SIZE`	`200`	Target enumeration page size.
`ONTOLOGY_REST_BULK_JOB_TIMEOUT_SECONDS`	`300`	Maximum wait for one asynchronous bulk job.
`ONTOLOGY_REST_BULK_POLL_INTERVAL_SECONDS`	`0.5`	Bulk-job polling interval.

Before a REST write, Frank pins the selected Iceberg snapshot, validates every source primary key, and builds duplicate-checked target and relationship-key indexes. Missing/duplicate source keys or unresolved/ambiguous relationships therefore fail before source-entity mutation. Bulk inserts use POST /api/v1/{type}/bulk; updates use PATCH with target UUIDs. Every request, enumeration page, and job poll carries the configured ontology tenant.

A bulk acknowledgement must contain one valid job ID. Frank polls that exact job until completed and requires its completed count to equal the submitted count. Completed job ID, operation, submitted count, and completed count are persisted on the exact SyncRun. Payloads and credentials are not evidence and are never stored there.

A lost or malformed submission acknowledgement is ambiguous because the CIRA REST API has no caller-supplied idempotency key or actor-scoped job lookup. Frank therefore does not resubmit that batch automatically. The run fails closed for operator inspection. A retry may poll a pending job only when its exact ID was checkpointed. HTTP 404/405 means bulk is unsupported; a required high-cardinality bulk sync fails rather than falling back to per-row writes.

Smaller REST datasets retain stable-key per-row upsert. Business-key GET and UUID-addressed PATCH retry bounded transport failures, HTTP 408/429, and 5xx responses because those operations are idempotent. POST is never blindly retried. After an ambiguous POST result, Frank re-reads by stable key and confirms or patches the accepted entity when found; otherwise the row remains unconfirmed for Temporal retry. Retry logs exclude URLs, row values, bodies, and credentials. Neither REST strategy deletes ontology entities.

Useful endpoints:

http

GET  /api/v1/backing-datasets/capabilities
POST /api/v1/backing-datasets/{id}/sync?force=true
GET  /api/v1/backing-datasets/{id}/sync-history
GET  /api/v1/backing-datasets/{id}/sync-history/{run_id}
GET  /api/v1/backing-datasets/{id}/sync-history/{run_id}/logs
GET  /api/v1/backing-datasets/{id}/health

The capabilities response advertises sync contract version 2, exact-run polling, and force-replay support. Clients must confirm those capabilities before requesting force replay; absence is not evidence that an older API will honor force=true.

For rolling upgrades, the historical OntologySyncWorkflow and its three V1 activities remain on ontology-sync-task-queue. New starts use the isolated OntologySyncWorkflowV2 activities on ontology-sync-v2-task-queue, served by ontology-worker-v2. This separation lets V1 histories and retries drain without receiving V2 replay semantics. It is a deployment contract, not a claim that a particular environment has completed the rollout.

The health endpoint checks the mapping, table state, ontology status, and schema drift signals that matter before publication.

Schema libraries

Schema libraries provide target schemas for transforms and backing datasets:

http

GET  /api/v1/schema-libraries
GET  /api/v1/schema-libraries/{library_id}/domains
GET  /api/v1/schema-libraries/{library_id}/domains/{domain}/schemas
GET  /api/v1/schema-libraries/{library_id}/schemas/{schema_id}
GET  /api/v1/schema-libraries/schema/{full_id}
GET  /api/v1/schema-libraries/search
POST /api/v1/schema-libraries/validate/{full_id}

The registry combines FIWARE Smart Data Models and custom schemas behind one browsing and validation surface.

Identity policies

Identity policies define stable keys for semantic entities. Strategies include:

passthrough: use the normalized source field.
composite: concatenate normalized fields.
hash: hash the composite key.
uuid: generate a UUID-form key from normalized values.

Policies can normalize values with operations such as trim, upper/lower, space stripping, and NFC normalization. They can be system-level or tenant-level.

Important endpoints:

http

GET    /api/v1/identity-policies
GET    /api/v1/identity-policies/{id}
POST   /api/v1/identity-policies
PUT    /api/v1/identity-policies/{id}
DELETE /api/v1/identity-policies/{id}
POST   /api/v1/identity-policies/{id}/dry-run

Use dry runs to verify identifier output before a transform or backing dataset depends on it.

Recommended workflow

Build and run a transform into a Silver or Gold table.
Choose or create an ontology entity type.
Register a backing dataset for the table.
Use mapping suggestions, then review field and relationship mappings.
Pick primary key and title key columns.
Run a health check.
Trigger sync.
Monitor sync history and logs.

Ontology Integration ​

The model ​

Entity types ​

Backing datasets ​

Property mappings ​

File publication ​

URL mode ​

Upload mode (inline binary) ​

Block reference ​

Republication and retry ​

Provenance ​

Mapping assistance ​

Sync ​

REST bulk and per-row safety ​

Schema libraries ​

Identity policies ​

Recommended workflow ​

Ontology Integration

The model

Entity types

Backing datasets

Property mappings

File publication

URL mode

Upload mode (inline binary)

Block reference

Republication and retry

Provenance

Mapping assistance

Sync

REST bulk and per-row safety

Schema libraries

Identity policies

Recommended workflow