Databricks resource management

TechFabric Harness lets an agent (or a user driving one) manage the Databricks resources it operates — create a Job, define a pipeline, build an AI Search index, size a serving endpoint, create a Genie Agent, grant access — through the same governed, typed seams the consumption tools already use.

Resource management is opt-in per surface and fails closed: a consumption-only databricks() bundle keeps its existing tools, and any model-exposed write surface refuses to initialize unless approval routing is configured. Stable services run through Databricks' official modular TypeScript SDK. Fabric owns the agent layer: approval binding, policy, lineage, durable retries, managed-resource fingerprints, and cleanup. Protocols absent from the SDK stay in a small, explicit raw protocol adapter. This does not replace Databricks Asset Bundles, Terraform, or checked-in pipeline source: Harness never generates that IaC. But the lifecycle of a checked-in bundle — validate, deploy, run, destroy — is itself a governed agent surface; see Checked-in Asset Bundles.

The docs use resource management as the developer-facing term. “Authoring” remains in a few flag and certification names where it distinguishes create/update/delete from consumption; it does not mean a second infrastructure-as-code system.

Quickstart: author your first Job

Enable one surface, bound its compute, and route writes to an approval audience:

import { init } from '@fabric-harness/sdk';
import { databricks } from '@fabric-harness/databricks';

const dbx = databricks({
  host: process.env.DATABRICKS_HOST!,
  principal: {
    kind: 'service-principal',
    host: process.env.DATABRICKS_HOST!,
    clientId: process.env.DATABRICKS_CLIENT_ID!,
    clientSecret: process.env.DATABRICKS_CLIENT_SECRET!,
  },
  jobAuthoring: true,
  computePolicy: {
    maxConcurrentRuns: 1,
    maxRunTimeoutSeconds: 1800,
    requiredTags: { 'cost-center': 'agents' },
    // schedules stay disabled unless you opt in with `schedules: true`
  },
  governance: {
    stewardAudience: 'data-platform',
    catalogs: ['main'],
    onLineage: (record) => audit(record),
  },
});

const fabric = await init({ modelProvider: dbx.modelProvider, tools: dbx.tools, policy: dbx.policy });

What happens when the model calls databricks_create_job:

1. The call routes to the data-platform approval audience. Execution does not start without a grant bound to this exact call (see approvals).
2. The job spec is validated against computePolicy before any API call — schedules off, timeout and concurrency capped, required tags present.
3. The created job is stamped with the fabric-harness:managed tag and a canonical spec fingerprint, so later updates and deletes can prove ownership.
4. The operation is recorded in lineage with the executing principal and every approver.

For local development only, allowUnapprovedAuthoring: true bypasses the fail-closed approval requirement. It is the single escape hatch and should never be set in a deployed app.

Runnable end-to-end versions of this flow live in the authoring examples.

How approval and governance work

An approval grant binds the logical tool-call id, a canonical digest of the input, the executing principal, the approver identities, and an expiry. A durable retry may replay the same operation, but a grant can never be reused for different input or a different identity. Grant TTL starts when the approval is decided, not when it was requested.

The governance block controls routing and audit:

Read tools remain approval-free when an admin surface is enabled. The typed clients (DatabricksJobsAuthoring, DatabricksGenieAdmin, …) can always be used without model approval policy — "the SDK may author" is deliberately separate from "a model may autonomously author". Native Databricks/Unity Catalog authorization is authoritative in either path.

Narrow by service only when that service boundary is explicit in the tool inventory. For example, use separate deploy_dev (service: "dev") and deploy_prod (service: "prod") tools, then set approvalServices: ["prod"]. The governance wrapper and approval policy use the same selection, so the dev tool runs without a grant while the production tool fails closed until approved.

Genie conversations, lifecycle management, and Agent Mode are configured together under genie. Supplying genie.manage is the explicit write opt-in; no separate authoring boolean is required.

Request-scoped user identity

Use a principal returned by forwarded-token validation/current-user inspection, never a label supplied by the request:

const inspection = await inspectDatabricksAppUserAuthorization({ headers, host });
const user = dbx.forPrincipal({ tokenProvider: forwardedTokenProvider, principal: inspection.principal });

The scoped bundle has isolated generated SDK/model/embedding/credential clients and does not mutate the parent. Persistence wiring and policy/lineage sinks remain app-scoped. Approval records and lineage distinguish the executing principal from all approvers.

Native SDK boundary

databricks() exposes the official principal-bound clients at bundle.sdk. Application code can also create them directly:

import { databricksSdk, databricksJobs } from '@fabric-harness/databricks';

const sdk = databricksSdk({ host, principal });
const jobs = databricksJobs(sdk.jobs);

Generated clients own stable Jobs, Statement Execution, Lakeflow, Vector Search, Model Serving, Files, Secrets, Genie, Access Management, Unity Catalog, MLflow Experiments, and Lakebase request/response models. Fabric's private raw protocol transport has an exhaustive method-and-path allowlist for endpoints that the modular SDK does not yet expose. Anything else is denied before credentials are resolved or network I/O starts. Source-boundary and runtime-denial tests enforce that quarantine. Every generated module is exact-pinned to SDK 0.21.0, and CI verifies the package pins, runtime constant, compatibility page, README, generated serialization, and packaged import together.

Genie Agent permissions use the generated @databricks/sdk-accessmanagement client; they are not a raw-protocol exception. The only stable-API fallbacks are the exact Genie query-result and AI Search floating-score decoder mismatches documented on the compatibility page.

Resource-management surfaces

Each surface is an independent opt-in. Enable only what the agent needs:

A bundle with every surface enabled looks like this — treat it as a reference for field names, not a starting point:

const dbx = databricks({
  host,
  principal,
  jobAuthoring: true,
  oneOffCompute: true,
  computePolicy: {
    allowedPolicyIds: ['policy-id'],
    allowedExistingClusterIds: ['explicit-shared-cluster-id'],
    maxWorkers: 2,
    maxConcurrentRuns: 1,
    maxRunTimeoutSeconds: 1800,
    allowedNodeTypes: ['Standard_D4ds_v5'],
    allowedRuntimeVersions: ['15.4.x-scala2.12'],
    serverlessPerformance: 'STANDARD',
    schedules: false,
    requiredTags: { 'cost-center': 'agents' },
  },
  lakeflowAuthoring: true,
  lakeflowEvents: true,
  assetBundles: { bundleDir: '.', target: 'dev' },
  genie: {
    manage: {
      resourceStore: managedResourceStore,
      delete: 'managed-only',
    },
  },
  aiSearchAdmin: {
    allowedOperations: ['createIndex', 'syncIndex', 'describeIndex'],
    allowedIndexes: ['main.kb.docs_idx'],
    allowedEndpoints: ['kb-search'],
    allowedEmbeddingModelEndpoints: ['databricks-bge-large-en'],
    allowedSourceTables: ['main.kb.docs'],
  },
  servingAdmin: true,
  ucAdmin: true,
  ucDestructiveAdmin: true,
  workspaceWrite: true,
  secretsWrite: true,
  secretProvider,
  warehouseId,
  governance: {
    stewardAudience: 'data-platform',
    approvalTtlSeconds: 900,
    catalogs: ['main'],
    onLineage: audit,
  },
});

Compute policy reference

computePolicy bounds every Jobs-authoring and one-off-run spec before any API call:

Allowlist fields must be non-empty when configured; numeric caps must be non-negative. Violations fail at bundle initialization or before the write executes — never after.

computePolicy does not bound execution of an existing job. databricksRunJobTool and databricksNotebookTool carry their own required runPolicy / notebookPolicy, enforced on the run path itself. See Bound what a model can run.

Managed resources and safe deletion

Destructive tools only touch resources TechFabric Harness can prove it created. Because Databricks services expose different metadata capabilities, three marker schemes exist:

• Jobs — created jobs carry the tags fabric-harness:managed=true and fabric-harness:fingerprint=<canonical spec hash>. ifExists: 'reuse' adopts only a single job matching name, managed tag, and fingerprint; anything ambiguous is refused. Updates refuse to remove the ownership tags and re-stamp the fingerprint.
• Unity Catalog — created catalogs and schemas carry the property fabric-harness.managed="true"; volumes (which have no properties) carry a comment prefixed [fabric-harness:managed]. deleteManaged() reads the object first and refuses to delete anything not carrying its marker. No force or cascade options exist.
• Genie Agents — the Genie management API supports neither tags nor properties, so ownership lives in a durable DatabricksManagedResourceStore manifest recording fingerprint, creator, and tool-call provenance. Deletion requires the exact managed id and expected fingerprint; an already-trashed managed id is treated as an idempotent success. MemoryDatabricksManagedResourceStore is only for tests and local development; use LakebaseDatabricksManagedResourceStore or another durable implementation in production.
• Asset Bundles — bundle ownership lives in the same DatabricksManagedResourceStore manifest (resourceType: "databricks-bundle"), recording a sha256 fingerprint of the sorted bundle source tree (excluding .databricks/ CLI state, node_modules/, and dist/). Deploy records the fingerprint after a successful CLI run; destroy requires the record and refuses to tear down a drifted tree unless force is set. See Checked-in Asset Bundles.

A create whose API response is ambiguous is never retried automatically, so a transient failure cannot orphan an unmarked resource. Schedules default off. Serverless jobs take the serverless branch; classic VM job compute requires a non-empty allowedPolicyIds and a matching policyId on every new cluster. Existing all-purpose clusters are denied unless their exact id appears in allowedExistingClusterIds.

Checked-in Asset Bundles

A job/pipeline bundle authored in YAML — by hand, by databricks bundle init, or by exporting a Lakeflow Designer canvas as a .designer.ipynb referenced from a notebook_task — stays checked-in IaC. What Harness governs is its lifecycle. The assetBundles option points the bundle at a directory containing databricks.yml and adds four model tools plus a typed client at bundle.assetBundle:

const dbx = databricks({
  host,
  principal,
  assetBundles: { bundleDir: './bundle', target: 'dev' },
  governance: { stewardAudience: 'data-platform' },
});

The fingerprint is a deterministic sha256 over the sorted bundle source tree (relative path plus content), excluding .databricks/, node_modules/, and dist/. The lifecycle shells out to the Databricks CLI (executable defaults to databricks, so it must be on PATH) and authenticates through the CLI's normal environment or --profile resolution; target, profile, and env pass through unchanged. CLI failures surface with a bounded stderr tail and are never retried automatically.

Several bundles in one agent

Passing a name-keyed map instead of one entry configures several bundles — a separate lifecycle per name, exposed as bundle.assetBundles:

const dbx = databricks({
  host,
  principal,
  assetBundles: {
    jobs: { bundleDir: './bundles/jobs', target: 'dev' },
    pipelines: { bundleDir: './bundles/pipelines', target: 'dev' },
  },
  governance: { stewardAudience: 'data-platform' },
});

await dbx.assetBundles?.jobs?.validate();

The tool names stay the same four. Every one of their input schemas gains a required bundle selector enumerating the configured names (jobs, pipelines), so a call must always say which bundle it means — including when only one bundle is configured, which keeps the schema and its approval digests stable if a second bundle is added later. Names must match ^[A-Za-z0-9][A-Za-z0-9_-]*$, and an empty map is rejected. Governed lineage names only the bundle the call selected, never the whole configured set, because the descriptor is a /bundle input path rather than a static value.

An approval grant binds the tool-call id, the executing principal, and the canonical tool input. bundle is part of that input, so a grant approved for { bundle: 'jobs' } cannot be replayed against { bundle: 'pipelines' } — distinct names cannot cross-approve. A name's target, however, is configuration rather than input: retargeting jobs from dev to prod does not invalidate outstanding grants for jobs, so a durable approved call replayed after that change would hit the new target. When dev and prod need separate approvals, model them as separate named entries (jobs_dev, jobs_prod) instead of editing one name's target.

Managed-resource identity is scoped per name and target — <bundle name from databricks.yml>#<configured name>[#<target>], where the #<target> suffix is present only when that entry configures a target — so several lifecycles can share one durable DatabricksManagedResourceStore without overwriting each other's fingerprint records. The single-bundle shape keeps the bare manifest name, so existing records stay addressable.

One limitation: approval routing is per tool name, so every named bundle routes to the same governance.stewardAudience. Per-bundle audiences are not expressible — if jobs and pipelines need different approvers, build two agents.

Like every authoring surface, assetBundles fails closed: without governance.stewardAudience (or the local-only allowUnapprovedAuthoring: true), bundle initialization throws instead of exposing ungated deploy/run/destroy tools. Application code can also drive the same lifecycle directly through databricksAssetBundleLifecycle() / databricksAssetBundleTools() without a model. See examples/with-databricks-bundle-deploy for a runnable steward-gated flow and fh add databricks bundle for the scaffold.

Genie Agents

Databricks renamed Genie spaces to Genie Agents, while the stable management and Conversation API paths retain /genie/spaces. Fabric consistently calls the resource identifier agentId.

The stable consumption path uses the typed DatabricksGenieClient. It starts or continues a conversation, requires a terminal COMPLETED state, retrieves every query result from its attachment-specific endpoint, and returns typed text, query, result, suggested-question, and visualization metadata. Missing ids, terminal failure, polling exhaustion, cancellation, oversized responses, and malformed attachments fail explicitly. Application code can also list conversation history and message comments or delete an exact conversation id; only ask is model-facing by default.

Genie lifecycle management is stable for the certified create/query/update/trash scope:

const dbx = databricks({
  host,
  principal,
  genie: {
    manage: {
      resourceStore: new LakebaseDatabricksManagedResourceStore({ client: lakebase }),
      delete: 'managed-only',
    },
  },
  governance: { stewardAudience: 'data-platform', catalogs: ['main'] },
});

const salesAnalyst = await dbx.genieAdmin?.create({
  version: 2,
  title: 'Sales analyst',
  parentPath: '/Shared/agents',
  warehouseId: 'warehouse-id',
  dataSources: [{ table: 'main.sales.orders' }],
});

DatabricksGenieAdmin provides list/get/create/update/delete, export/import, and permissions APIs. The serializer emits Databricks serialized_space version 2, deterministic 32-character ids, and the required sorted collections. Updates compare the caller's expected canonical fingerprint and send the last observed Databricks ETag. The management API has no general field mask: description is the only field that supports explicit clearing via remove; other fields can only be set to new values. Ambiguous creates, updates, and deletes are not retried. Raw serialized_space is accepted only by typed-client import/export; no built-in ToolDef accepts it.

Model-facing create, inspect, update, and optional delete tools use a normalized schema containing tables, columns, sample questions, and one text instruction. SQL examples, functions, joins, filters, expressions, measures, and benchmarks become available only when genie.manage.sqlPolicy is configured. The policy runs server-side for every SQL fragment, must return the complete parsed catalog.schema.object set, and denies any resource outside dataSources. An allowlisted prepared statement is also valid; returning an empty or partial parse for dynamic SQL is not. Mutation requires steward approval and a durable ownership record.

const certifiedSql = new Map([
  ['SELECT COUNT(*) FROM main.sales.orders', ['main.sales.orders']],
]);

const dbx = databricks({
  // ...identity, ownership store, governance...
  genie: {
    manage: {
      resourceStore: managedResourceStore,
      sqlPolicy: ({ sql }) => {
        const resources = certifiedSql.get(sql);
        if (!resources) throw new Error('SQL is not in the certified statement set');
        return { resources };
      },
    },
  },
});

Fabric's ordinary Genie client uses the Conversation API. The separately configured Agent Mode client supports Databricks' Beta SSE endpoint:

const dbx = databricks({
  host,
  principal,
  genie: {
    agentMode: {
      agentId: '0123456789abcdef0123456789abcdef',
      acknowledgeBeta: true,
      timeoutMs: 5 * 60_000,
      idleTimeoutMs: 2 * 60_000,
      modelTool: { maxOutputBytes: 512 * 1024 },
      onEvent: (event) => auditAgentModeProgress(event),
    },
  },
});

for await (const event of dbx.genieAgentMode!.respond('Summarize revenue by region')) {
  if (event.type === 'response.completed') console.log(event.response.output);
}

It validates the preview's 32-hex agent id, never retries the response-creating POST, propagates cancellation, enforces idle and overall deadlines, parses arbitrarily chunked SSE, preserves unknown Beta events, enforces monotonic sequence numbers and event/stream limits, and requires response.completed or response.failed. modelTool adds the governed databricks_genie_agent_mode tool; progress goes only to onEvent, while the model receives one bounded terminal response. listConversationItems() exposes the cursor-based history endpoint. A failed terminal event is yielded and then raises DatabricksGenieAgentModeResponseError on the next iterator step.

The workspace must be enrolled by the Databricks account team and a workspace admin must enable Genie Agents Agent Mode API in Previews. Agent Mode allows one in-flight response per conversation and can keep an SSE connection open for up to 90 minutes. Fabric does not silently substitute it for ordinary Genie conversations. See the Genie Agents API and Beta Agent Mode API for the native contracts.

Bind a Genie Agent to a Databricks App

export default {
  target: 'databricks-app',
  databricks: {
    app: {
      genie: {
        agentId: '0123456789abcdef0123456789abcdef',
        // permission defaults to CAN_RUN
      },
    },
  },
};

The generated bundle attaches a genie_space resource and injects DATABRICKS_GENIE_AGENT_ID through valueFrom; it does not embed the id in model context. CAN_EDIT and CAN_MANAGE are rejected unless the deployment block also sets authoring: true.

Genie Ontology export

planModuleOntologyExport is the first piece of the Genie Ontology exporter: it consumes a governed Fabric module manifest — the JSON-serializable ModuleManifest / ModuleManifestBundle contract from @fabricorg/platform/manifest, consumed type-only — and produces a deterministic, fingerprinted DatabricksOntologyExportPlan describing how the module's declared ontology maps onto Unity Catalog Semantics (metric views, glossary terms, domain assignments) and a Genie Agent spec:

import { planModuleOntologyExport } from '@fabric-harness/databricks';

const plan = await planModuleOntologyExport({
  manifest, // ModuleManifest or ModuleManifestBundle produced by the vertical's Platform process
  catalog: 'main',
  schema: 'lending_semantics',
  // domain defaults to the manifest namespace; genie/feed options are optional
});

The plan step is pure — no network, no clock — so its output is safe to render for review and diffs byte-identically across runs for an unchanged manifest; plan.fingerprint changes when the manifest or plan inputs change. The manifest is an export source only: UC Metric Views remain the runtime semantic source of truth, export is one-directional (Platform → UC/Genie), and Genie stays read-only over Fabric-governed domains.

plan.policyClassifications records the class (threshold-ratio-metric, gate-compliance-rule, or code-evaluator) plus the matched evidence for every manifest policy, and plan.metricViewDdl carries one CREATE OR REPLACE VIEW ... WITH METRICS LANGUAGE YAML statement per module (<namespace>_metrics), tagged with the fabric-harness.managed marker and a manifest-fingerprint comment so drift is detectable by inspection and re-export is byte-identical. Threshold/ratio data policies become metric-view measures; compliance gates (FCRA/TCPA-style) become glossary business rules and never MEASURE(...) — a code policy becomes a metric seed only through the explicit metricSeedPolicyIds opt-in. Measures from countable action volumes arrive with the operational-truth feed tables in a later release.

When the feed option is present, plan.feedTables carries the operational-truth feed: three Delta/UC tables per module — <schema>.<namespace>_asset_events (domain events with schema_version, tenant, correlation, and causation), _action_invocations (action id, version, terminal status, failure classification, duration, and redacted parameters only), and _policy_evaluations (decision, reasons, evaluator identity/version, and failure mode). The generated DDL is CREATE TABLE IF NOT EXISTS ... USING DELTA with 'delta.appendOnly' = 'true', the managed tag, and a feed-fingerprint comment. Each table is replay-safe by construction: the hydration pipeline merges on the Platform id column (spec.replayKey — event_id, invocation_id, evaluation_id) with MERGE ... WHEN NOT MATCHED THEN INSERT, so re-hydrating recorded history is idempotent. Tenant scoping is a tenant_id column enforced by UC grants per consumer, never filtered at export, and the feed reads durable history records only — it is orthogonal to the vertical's host execution runtime. Events land at their declared schema_version; Harness performs no upcasting at append time (the conservative resolution of design question Q3 — a vertical that wants a current-shape stream upcasts in its own producer). applyModuleOntologyExport executes the feed DDL through the same approval-gated statement seam as the metric views, ahead of them (the metric view sources the asset-events feed), with one provenance record per table.

Every feed table spec carries an explicit publishableFields allowlist, and row materialization projects each record onto exactly that list before Delta append — anything not allowlisted is dropped, so PII/PHI never reaches Genie's training surface. Redaction ownership stays with the vertical: Harness defines no redaction policy of its own. Feed rows are materialized only through the caller's redacted-record producer (the fabric-autorefi redactActionParameters pattern); materialization fails closed without it, so raw Platform records can never be appended unredacted. Harness's role is limited to the schema contracts, the publish-time allowlist check, and absence tests proving raw parameters and PII-bearing fields never land. Retention for the append-only feed is defined in docs/databricks-genie-ontology-feed-retention.md: a 90-day hot retention window by default (per-table configurable, keyed to each table's recorded domain time), no Harness-side archive by default, and a scheduled expiry job that lands with the first dogfood deployment — re-hydration through planOntologyFeedHydration can always restore expired rows from Platform history.

planOntologyFeedHydration is the replay-safe append pipeline (design §3.7). It takes the plan's feed table specs, recorded Platform history grouped by kind (assetEvents, actionInvocations, policyEvaluations), and the required redacted-record producer, and returns merge-ready rows plus one deterministic MERGE INTO ... USING (VALUES ...) WHEN NOT MATCHED THEN INSERT statement per table, keyed by the spec's replay key. Rows are redacted, allowlist-projected, deduplicated on the replay key within the batch (first occurrence wins, drops counted), and sorted by replay key, so identical history yields a byte-identical plan and re-applying a statement inserts no duplicate rows — the idempotent re-run contract. The pipeline is pure (no network, no credentials): the scheduled feed pipeline executes the statements through runStatement with a DatabricksStatementClient constructed for the per-tenant workspace service principal (§5), while tenant scoping stays a tenant_id column enforced by UC grants. For asset events, assetEventRecordFromEnvelope maps the shipped AssetEventEnvelope shape from @fabricorg/platform/events (consumed type-only, exactly like the manifest) onto the feed record fields; Platform ships no durable ActionInvocation/PolicyEvaluation record contract, so the minimal DatabricksOntologyActionInvocationRecord / DatabricksOntologyPolicyEvaluationRecord types define what the vertical's history reader hands over. The runnable examples/with-databricks-ontology-export workspace demonstrates the full story credential-free — plan, hydration with a demo redaction producer, idempotent re-run, and a PII-absence check — with exact expected output in its README.

plan.glossaryTerms and plan.domainAssignments carry the Unity Catalog glossary upsert payloads: one term per object type (displayName names the term; idPrefix and the module idPrefixes registry are preserved as metadata for identifier parsing), subject-vocabulary terms, one term per event type with schemaVersion metadata, lifecycle terms for actions with mutatesDomain: true (read-side phrasing only), business-rule terms referencing policyId@policyVersion for gate/compliance and code policies (a dataDefinition summary for data policies; code-evaluator logic is Platform-internal and never exported), and one term per state-machine state label. Every term carries manifestVersion metadata. Domain assignments bind one UC domain per module namespace — the domain option overrides the default — with the module description as the domain comment. Adapter steps, code-policy bodies, and requiredRoles / requiredPermissions / eventPhase are deliberately excluded. These payloads are plan data until apply; the governed publish path goes through applyModuleOntologyExport (see below), and the glossary/domain UC REST surface maturity is still being tracked while Genie Ontology is in public preview — apply accepts an injected DatabricksOntologyGlossaryPublisher so the publish client can evolve without changing the plan contract.

When the genie option is present, plan.genieSpec carries a fully derived DatabricksGenieAgentSpecV2 — the same spec shape the Genie authoring tools accept, normalized and fingerprinted through the identical canonicalization DatabricksGenieAdmin applies. Each module in the bundle contributes a kind: "metric-view" data source (<catalog>.<schema>.<namespace>_metrics), plus its declared feed tables when feed options are present; threshold/ratio policies become sqlSnippets.measures reusing the exact metric-view measure names and expressions; object-type displayNames and state labels land as column synonyms; bundle dependencies become joins on the shared object_type key (only when both manifests are in the bundle); sample questions come from action and state vocabulary with read-side phrasing only; and the single permitted instruction carries the namespace description, the event lifecycle vocabulary, and the read-only constraint — Genie never mutates Fabric-governed domains. Action parameter names are treated as untrusted text: they appear only as read-side slot hints in sample questions, and only after passing a denylist filter (password, secret, token, ssn, …) plus the vertical's own genie.sensitiveFieldPatterns. plan.genieSpecFingerprint is the canonical spec fingerprint the governed apply path will use as expectedFingerprint, so a plan-time diff keys off the same conflict contract as the shipped authoring path:

const plan = await planModuleOntologyExport({
  manifest: bundle, // ModuleManifestBundle — joins derive from cross-module dependencies
  catalog: 'main',
  schema: 'vertical_semantics',
  genie: {
    agentId: existingAgentId, // omit to create; used for the conflict-check fingerprint
    admin: genieAdmin, // existing fingerprinted authoring client (apply only; wiring, not an input)
    parentPath: '/Workspace/Genie',
    warehouseId: 'wh-1',
    // sensitiveFieldPatterns: ['drivers_license'], // vertical-declared additions to the denylist
  },
});
// plan.genieSpec — reviewable, deterministic DatabricksGenieAgentSpecV2
// plan.genieSpecFingerprint — desired post-apply fingerprint for no-op/conflict detection

applyModuleOntologyExport is the governed write side of the plan/apply split. It takes the plan plus already-constructed clients (it never resolves credentials — interactive export should build them with OBO identity so UC enforces the curator's own grants) and applies the artifacts in dependency order: feed tables → metric views → glossary/domains → Genie Agent:

import { applyModuleOntologyExport } from '@fabric-harness/databricks';

const result = await applyModuleOntologyExport(plan, {
  resourceStore, // shared managed-resource provenance store
  principal: 'user:curator@example.com', // exporting principal label, never a token
  toolCallId, // stable correlation id for this apply invocation
  statements, // statement-execution client (metric-view DDL seam)
  warehouseId: 'wh-1',
  approval: async (request) => {
    // Required. Called once per write, before it happens: feed-table DDL,
    // metric-view DDL, domain assignment, glossary publish, Genie Agent
    // create/update — each
    // with effect: 'write', the governed resource, and the content fingerprint.
    // Route to the steward approval flow; throw to deny.
  },
  sqlPolicy, // required when the derived spec carries SQL fragments (fail-closed)
  genie: { admin: genieAdmin, agentId: existingAgentId }, // admin must share resourceStore
  glossary, // optional DatabricksOntologyGlossaryPublisher
});

Every authoring mutation is approval-gated: the approval callback fires per write with effect: "write" and is the caller's hook into the same steward approval flow that gates the authoring tools — apply never approves itself. Genie spec changes go through DatabricksGenieAdmin.update compare-and-swapped against the fingerprint recorded at last apply in the managed-resource store: a live fingerprint that drifted (another author's edit) surfaces the typed DatabricksManagedResourceConflictError — no blind overwrite, no silent retry. Updates also require the agent to be recorded as Harness-managed, exactly like the model-facing authoring tool. SQL fragments in the derived spec (measure snippets, join SQL) are admitted only through sqlPolicy, and apply throws before any write when the spec carries fragments but no policy was supplied. Exporter-generated metric-view and feed-table DDL is deterministic plan output; it executes through runStatement with approval rather than a raw fetch.

On success the managed-resource store records provenance for every artifact — one record per metric view (ontology-metric-view/<catalog.schema.view>), per glossary domain (ontology-glossary-domain/<domain>), the Genie Agent ownership record written by DatabricksGenieAdmin itself, and an append-only export record keyed by the plan fingerprint (ontology-export/<planFingerprint>) — each carrying the exporting principal, toolCallId, timestamps, and the fabric-harness.managed tag. A changed manifest yields a new plan fingerprint and therefore a new export record; historical records are never rewritten.

Re-export against an unchanged manifest produces an empty diff: when every recorded fingerprint matches the plan and the managed Genie Agent still matches live, apply performs no writes and returns result.noChanges: true with the per-artifact applied: false markers. The result also carries the applied artifacts (view names, agent id, glossary term counts, domain assignments), provenance record ids, and post-apply fingerprints. Apply is not transactional across artifacts — a conflict on a later artifact leaves earlier ones applied and recorded — but every artifact is independently idempotent, so re-running apply converges.

Drift detection and steward attestation

Genie Ontology's knowledge store infers expressions, synonyms, concepts, and experts; the manifest declares semantics. They will diverge, and the divergence is surfaced as evidence — never applied back. compareOntologyDrift is the pure declared-vs-inferred comparison: it takes the export plan's declared side (glossary terms and measure-seed policy classifications) plus the knowledge store's inferred concepts as plain data, and classifies every finding:

Severity encodes the conflict policy: the manifest is authoritative for mutation semantics, so an inference whose label collides with a different declared definition than its target is a conflict routed to steward review; a tolerated inference mapped onto its declared target is analytics-only; an exact restatement of the declared name is agreement and produces no finding. The comparison is deterministic — no network, no clock — and runs credential-free against plain fixtures. The knowledge-store read goes through the injected DatabricksGenieKnowledgeStoreReader interface (one read method returning plain data), so the preview/unsettled knowledge-store APIs (design question Q1) stay behind the caller's adapter and a preview-API change never reshapes the module.

attestOntologyDrift wraps the comparison into the steward-surfacing path: it reads inferred concepts through the injected reader, compares, and appends exactly one attestation record per run to the caller-supplied DatabricksOntologyDriftEvidenceSink:

import { attestOntologyDrift } from '@fabric-harness/databricks';

const record = await attestOntologyDrift(plan, {
  stewardAudience: 'lending-stewards', // required — routes the record to steward review
  knowledgeStore, // injected read-only client (design Q1 preview caveat)
  evidence, // append-only audit sink
  agentId: existingAgentId, // optional Genie Agent scope for the read
});
// record.attestation — ExecutionAttestation via databricksExecutionAttestation
// record.comparison — the classified findings

The record's attestation is built through the same databricksExecutionAttestation helper the Platform bridge uses, with runId (a content fingerprint of plan + findings + audience) as the external operation id and the steward audience in its metadata. Records are append-only audit evidence: each run appends, historical records are never rewritten. Nothing auto-applies back — the drift module exposes no apply path, the knowledge-store interface is read-only by construction, and the only side effect is the single evidence append. A fingerprint mismatch on the Genie side still blocks silent re-export through the existing DatabricksManagedResourceConflictError contract on the next applyModuleOntologyExport; nothing flows back into the module without a human changing governed code.

Current status: the genie-ontology-export capability is registered at beta in the capability registry while Genie Ontology is in public preview. Plan, apply, and drift carry credential-free contract coverage; retained protected-workspace apply evidence is still pending and is the remaining P0 exit item. Promotion to stable follows the same rule as genie-authoring: retained protected-workspace evidence on non-preview APIs.

Serving and AI Search constraints

aiSearchAdmin takes a resource policy, not a boolean. allowedOperations decides which of the six tools are registered at all — an agent that only needs to sync an index never sees databricks_delete_search_endpoint:

Names are pinned into the tool inputSchema (const for one, enum for several) and re-checked at call time; an out-of-policy name throws DatabricksResourceNotAllowedError before the Vector Search API is reached.

AI Search direct-access creation requires a full schema and vector column dimension. The model-facing serving tools reject environmentVars; applications that deliberately need raw values must use the typed client so those values never enter model context. See the native Serving Endpoints API, AI Search Indexes API, and Unity Catalog Grants API.

Secrets and lineage

The databricks_put_secret input is { scope, key, secretRef: { kind: 'secret', name } }. The configured SecretProvider resolves material during server-side execution and passes it directly to Databricks. The value never appears in model context, tool input, or lineage. The raw-value method exists only on the typed client and is an application-level trust decision.

Structured governance metadata uses extended JSON Pointer paths, including * for arrays. Bundle initialization validates every authoring descriptor against its input schema. At execution, a required zero-match fails closed, every catalog-qualified resource is allowlist-checked, and every resource is recorded in lineage.

Examples and failure behavior

• examples/with-databricks-jobs-authoring: policy-bound multi-task Job lifecycle.
• examples/with-databricks-bundle-deploy: governed checked-in Asset Bundle validate/deploy/run/destroy with fingerprint drift detection.
• examples/with-databricks-dataeng: Lakeflow operation, authoring, and event expectations.
• examples/with-databricks-rag-admin: AI Search endpoint/index lifecycle and query verification.
• examples/with-databricks-authoring-admin: serving, non-destructive UC, workspace, and secret-reference writes.
• examples/with-databricks-genie-authoring: normalized Genie Agent lifecycle with a durable Lakebase ownership manifest.
• examples/with-databricks-ontology-export: credential-free Genie Ontology export plan, feed hydration, idempotent re-run, and PII-absence check.

All examples document exact credentials and expected output in their README. Missing approval routing fails at bundle initialization. Missing warehouse/secret provider dependencies, governance descriptor errors, grant/principal mismatches, catalog denial, native Databricks authorization, timeouts, ambiguous creates, and cleanup leaks fail explicitly.

Stability

Jobs, Lakeflow, AI Search administration, non-preview custom-model serving, managed-only UC administration, workspace writes, secret-reference writes, and Genie Agent management (for its create/query/update/trash scope) are stable for their documented scope, backed by retained protected-workspace lifecycle evidence. Jobs evidence covers serverless STANDARD and policy-bound classic definitions; AI Search evidence covers direct-vector and Delta Sync lifecycles. Provisioned Throughput, AI Gateway administration, Agent Mode streaming, and Databricks App OBO authentication retain their separate Databricks preview constraints. The full release gate, destructive certification lifecycle, and retained evidence are documented in Authoring certification and release evidence.

Upgrade note

Approvals persisted before grant provenance existed cannot authorize a post-upgrade tool call. An idempotent session detects the missing grant and requests approval again under a new provenance id. This is intentionally fail-closed; operators may see one new approval after upgrading a durable session.

Pair compute limits with databricksTenantCostLimit() and System Tables reconciliation when authoring Jobs or serving resources. Compute lifecycle outside governed Jobs remains deferred until budget guardrails can be enforced server-side.