# TechFabric Harness complete documentation
Canonical index: https://harness.techfabric.com/llms.txt
License: Apache-2.0
TechFabric Harness is a TypeScript framework for governed, durable autonomous agents. The complete corpus includes Databricks Responses API and MLflow ResponsesAgent interoperability, Unity AI Gateway, Unity Catalog Agent Services, SQL, AI Search and RAG evaluation, Lakeflow, Lakebase, Apps deployment and certification, plus enterprise identity/RBAC, channels, databases, sandboxes, MCP, operator controls, and every public API reference.
---
# What is TechFabric Harness
Canonical: https://harness.techfabric.com/docs
A TypeScript framework for building durable, deployable autonomous agents.
## The 30 second answer
**Databricks (or any host platform) decides who can touch data, models, and compute. TechFabric Harness
decides how an autonomous agent runs.**
Fabric is a **headless** TypeScript framework for **durable, deployable agents**. Finite jobs live
in `.fabricharness/jobs/`; persistent, addressable agents live in `.fabricharness/agents/`. Run them
locally and build for Node, Docker, Temporal, Cloudflare, Azure-oriented targets, or Databricks Apps
and Model Serving proxy deployments.
Its distinctive boundary is **not** the prompt loop, threads, or request-time tool lists. Fabric
keeps execution recoverable and governance enforceable when work:
- outlives a request or process,
- waits minutes or days for an exact-operation approval,
- crosses infrastructure (shell, sandboxes, channels, other clouds),
- or must leave correlated operator evidence behind.
| You need… | Use |
| --- | --- |
| Interactive Databricks-only agent with request/thread lifecycle | [Databricks AppKit](https://developers.databricks.com/docs/appkit/v0/plugins/agents) or native SDK |
| Recoverable runs, durable approvals, portable policy, isolated work, multi-target deploy | **TechFabric Harness** (on Databricks or elsewhere) |
| Deterministic short application code (no agent loop) | Native Databricks TypeScript SDK |
For the full overlap analysis, see
[Why TechFabric Harness on Databricks](/docs/databricks/why-fabric).
> **Headless runtime, optional clients.** Agents run through the SDK, HTTP, schedules, channels, or
> Temporal without requiring a UI. Use `fh fiber` for terminal interaction or
> `@fabric-harness/react` for an application UI; both consume the same authenticated public protocol.
New to the framework? Follow [Learn TechFabric Harness](/docs/learning-paths). It gives every shipped
feature family a progressive guide, runnable practice workspace, failure/cleanup expectation, and
production proof instead of asking you to discover the framework from the API reference.
## Why a framework, not just an SDK
Most agent libraries leave you to wire up the runtime, the build, the dev server, the deployment story, and the durability story yourself. TechFabric Harness is opinionated about those things so the agent code stays focused on the work:
- **Workspace conventions** — `.fabricharness/jobs`, `agents`, `roles`, `skills`, `policies`, `sandboxes`, plus a project `AGENTS.md`.
- **CLI** — discover, run, build, deploy, inspect, replay, and verify everything from one binary (`fabric-harness` or `fh`).
- **Runtime adapters** — local Node, Docker sandbox, Temporal worker, Cloudflare Workers, Foundry-hosted, more on the way.
- **Headless by default** — agents complete autonomously. Approvals are an explicit hook, not a default user prompt.
- **Durable by design** — bounded runs, persistent sessions, submissions, cancellation,
checkpoints, retry classification, and optional Temporal workflows survive process and worker
failure.
- **Capability-scoped security** — definition policy remains a security floor across tools,
commands, filesystems, networks, sandboxes, and connectors; secrets stay out of model context.
- **Durable governance** — approvals bind the exact operation and principal, survive long waits,
and remain correlated with lineage, cost, artifacts, and terminal state.
- **Portable contracts** — agent, session, tool, policy, source, sandbox, store, and deployment
contracts do not belong to one cloud or data platform.
## What is distinctive
Many platforms now provide TypeScript agent definitions, threads, scoped tools, streaming, and
request-time approval. Fabric includes those features, but does not present them as unique. Its
strongest advantages begin where a request-scoped agent host stops:
| Requirement | TechFabric Harness contract |
| --- | --- |
| Work must survive process or worker failure | Durable submissions, leases, bounded retries, terminal-state classification, checkpoints, and optional deterministic Temporal workflows |
| Approval may take minutes or days | Approval records remain bound to the operation and executing identity instead of living only in one HTTP stream |
| Policy must follow the workload | Definition policy is a security floor across local, Databricks, cloud, edge, cluster, connector, and sandbox execution |
| The agent must perform isolated work | Tasks, shell commands, mounted sources, artifacts, checkpoints, forks, and portable sandbox references share the session contract |
| Operators need recovery evidence | Events, traces, replay views, lineage, cost, artifacts, cleanup records, and distinct cancelled, exhausted, retryable, permanent, and terminal outcomes stay correlated |
| The deployment platform may change | The same finite or persistent definition builds for Node, Docker, Temporal, Cloudflare, Azure-oriented targets, Databricks Apps, and other supported backends |
## How TechFabric Harness works
Definitions declare what an agent can do. The runtime creates an isolated session, assembles roles,
skills, tools, policy, and context, then drives the model loop. Every tool or shell action passes
through capability policy before a sandbox or provider adapter executes it. Events, approvals,
artifacts, cost, and results remain correlated to the session and submission.
```mermaid
flowchart LR
INPUT[CLI, HTTP, schedule, or channel] --> DEF[Job or persistent agent]
DEF --> SESSION[Session runtime]
subgraph Context[Context assembly]
ROLE[Roles]
SKILL[Skills]
MEMORY[Memory and mounted sources]
end
ROLE --> SESSION
SKILL --> SESSION
MEMORY --> SESSION
SESSION --> MODEL[Model provider]
MODEL --> ACTION{Next action}
ACTION -->|Tool| POLICY[Capability policy]
ACTION -->|Shell or file| POLICY
ACTION -->|Final result| RESULT[Typed result and events]
POLICY -->|Approval needed| APPROVAL[Durable approval]
POLICY -->|Allowed| EXEC[Sandbox or connector]
APPROVAL --> EXEC
EXEC --> SESSION
SESSION --> STORE[(Session and submission store)]
classDef entry fill:#f4f4f5,stroke:#71717a,color:#18181b
classDef fabric fill:#dbeafe,stroke:#2563eb,color:#172554
classDef control fill:#fef3c7,stroke:#d97706,color:#422006
classDef state fill:#dcfce7,stroke:#16a34a,color:#052e16
class INPUT,DEF entry
class SESSION,MODEL,RESULT fabric
class ACTION,POLICY,APPROVAL control
class EXEC,STORE,ROLE,SKILL,MEMORY state
```
On Databricks, the same runtime can use Model Serving, SQL Warehouses, Unity Catalog, AI Search,
Genie, Feature Serving, Lakeflow, Lakebase, MLflow, and Databricks Apps under one propagated identity.
It can also author Jobs, pipelines, indexes, endpoints, grants, workspace objects, and secret
references through opt-in approval-bound tools. Start with
[TechFabric Harness on Databricks](/docs/databricks).
## Build governed agents around Databricks workloads
TechFabric Harness is designed for enterprises that already use Databricks as their governed data and
AI platform and need an application runtime around it. Data, models, compute, and authorization stay
native to Databricks. Fabric adds durable agent sessions, model-safe tools, approval routing,
identity propagation, deployment contracts, lineage, and cost correlation.
| Workload | Databricks foundation | TechFabric Harness adds |
| --- | --- | --- |
| Analytics copilot | Genie, SQL Warehouses, Unity Catalog | Safe `SELECT` path, steward approval for broader SQL, OBO identity, durable conversations |
| Knowledge assistant | AI Search, AI Gateway, MLflow | Citation-validated RAG, session state, evaluation export, release evidence |
| Data operations agent | Jobs, Lakeflow, notebooks | Approval-bound execution, durable receipts, retries, cleanup, status collection |
| Stateful Databricks App | Apps, Lakebase, UC Volumes | Persistent sessions and streams, restart recovery, attachments, tenant deletion |
| Agent API and discovery | Responses API, MLflow ResponsesAgent, Agent Services | One interoperable endpoint, deployment artifacts, discovery, grants, and policy |
Start with the [Databricks workload map](/docs/databricks/workloads), use the
[enterprise adoption guide](/docs/databricks/enterprise-adoption) to plan a production rollout, and
review [workspace compatibility evidence](/docs/databricks/compatibility) before making a cloud,
region, or preview claim.
Databricks now also provides a beta TypeScript
[AppKit agent host](https://developers.databricks.com/docs/appkit/v0/plugins/agents) with threads,
tools, subagents, cancellation, human approval, and Responses-compatible routes. That is real
overlap. Use AppKit directly for a Databricks-only interactive agent when its request and thread
lifecycle is sufficient. Use Fabric when the workload needs recoverable execution, durable waits,
portable policy, isolated shell/filesystem work, cross-system orchestration, or the option to move
the same contract beyond Databricks. The approaches also compose: Fabric can use Databricks Apps,
Agent Bricks, Unity AI Gateway, Lakebase, Jobs, and Unity Catalog as native services.
## Complete capability map
| Area | Included TechFabric Harness features |
| --- | --- |
| Agent lifecycle | Finite agents, persistent addressable agents, per-interaction initialization, sessions, submissions, tasks, subagents, dispatch, schedules, webhooks, and channels |
| Context | Prompts, roles, Markdown-first skills, session memory, mounted filesystem sources, attachments, compaction, typed input, and validated output |
| Execution | Typed tools, commands, shell, MCP, connectors, isolated sandboxes, sandbox capability discovery, snapshots, forks, and portable references |
| Durability | Bounded retries and loops, leases, cancellation propagation, checkpoints, replay views, idempotent operations, recovery, and deterministic Temporal workflows |
| Governance | Capability policy for tools, commands, filesystems, networks, credentials, connectors, exact-operation approvals, identity binding, budgets, redaction, and tenant isolation |
| State | File, SQLite, Postgres, Redis, Cloudflare, Lakebase, session streams, submissions, memory, artifacts, attachments, and deletion contracts |
| Models and data | OpenAI-compatible providers, Anthropic, Bedrock, Vertex, Azure, Databricks, Workers AI, custom providers, databases, retrieval, and governed data tools |
| Delivery | CLI, HTTP, Responses API interoperability, typed client, React hooks, Fiber console, queues, schedules, channels, portable build artifacts, and deployment targets |
| Operations | Structured events, OpenTelemetry, MLflow integration, metrics, token and cost attribution, logs, release evidence, certification, and supply-chain attestations |
## Standard agent terminology
TechFabric Harness keeps the terms that are converging across the agent ecosystem. You will find these everywhere in the docs:
| Term | Meaning |
| --- | --- |
| **Agent** | A configured autonomous runtime. |
| **Session** | A persisted message/context thread. |
| **Skill** | A reusable Markdown- or code-backed procedure. |
| **Role** | A scoped instruction/model profile. |
| **Sandbox** | An isolated execution environment with filesystem/shell/tools. |
| **Task** | A child or delegated agent run. |
| **Tools** | Model-callable functions. |
| **Commands** | Shell-level capabilities exposed to the sandbox. |
| **Build** | A compiled, deployable workspace artifact. |
## Agent functions for conversations, finite jobs for results
Use a **persistent agent function** when people or systems will message the same address over time.
Hooks keep its model, tools, skills, sandbox, and instructions together in one readable definition:
```ts title=".fabricharness/agents/triage.ts"
import {
createAgent,
useModel,
useSandbox,
useSkill,
useTool,
} from '@fabric-harness/sdk';
import { reviewChecklist } from '../skills/review';
import { proposeFix, searchIssues } from '../tools/github';
function TriageAgent() {
useModel('anthropic/claude-sonnet-4-6');
useSandbox('docker', { cwd: '/workspace' });
useSkill(reviewChecklist);
useTool(searchIssues);
useTool(proposeFix);
return `Triage the reported issue end to end.
Reproduce it, identify the root cause, propose a fix,
and verify the result before you finish.`;
}
export default createAgent(TriageAgent, {
durability: { maxAttempts: 5, timeoutMs: 2 * 60 * 60_000 },
});
```
The function renders before each interaction. Its return value is the instruction, while hooks compose
capabilities in call order. Static policy, durability, triggers, and initial-data validation belong in
`createAgent()`'s second argument so the host can enforce them even when the function cannot render.
Use a **finite agent** when one typed invocation should return one bounded result:
```ts title=".fabricharness/jobs/echo.ts"
import { defineAgent, schema } from '@fabric-harness/sdk';
export default defineAgent({
name: 'echo',
input: schema.object({ message: schema.string() }),
output: schema.string(),
run: ({ input, prompt }) => prompt(input.message),
});
```
The bare `@fabric-harness/sdk` import gives finite agents headless defaults (`runtime: 'stateless'`,
`sandbox: 'virtual'`, the Pi loop runtime, and automatic compaction). Definition-level instructions,
tools, policies, cost budgets, and approval timeouts become runtime defaults; invocation policy and
budgets cannot weaken the definition's security floor.
For **Temporal-backed finite agents** or **compliance/audit workloads** where implicit defaults are
undesirable, switch to `@fabric-harness/sdk/strict`. The call shape stays the same, but every runtime
option is explicit. Runtime (`stateless` / `inline` / `temporal`) and deployment target remain separate
choices.
```ts title="Strict — every option declared, Temporal-safe"
import { defineAgent, schema } from '@fabric-harness/sdk/strict';
export default defineAgent({
name: 'triage',
model: 'openai/gpt-5.5',
input: schema.object({ issueNumber: schema.number(), title: schema.string() }),
output: schema.object({ severity: schema.enum(['low','medium','high']), summary: schema.string() }),
triggers: { webhook: true, schedule: '*/15 * * * *' },
run: async ({ init, input }) => {
const fabric = await init({
runtime: 'temporal',
sandbox: 'local',
compaction: { enabled: false },
policy: triagePolicy,
});
const session = await fabric.session();
return await session.prompt(`Triage issue #${input.issueNumber}: ${input.title}`);
},
});
```
Persistent and finite definitions deploy through the same CLI and share the same session, tool, skill,
sandbox, policy, and model contracts. See [Dynamic agents and hooks](/docs/building/dynamic-agents),
[Persistent agents](/docs/building/persistent-agents), and
[SDK entrypoints, runtimes, and targets](/docs/reference/sdk-entrypoints-runtimes-targets) for the full
distinction.
## Public API surface
```ts
const fabricAgent = await init(options);
const session = await fabricAgent.session(id?, options?);
await session.prompt(text, options?);
await session.skill(name, options?);
await session.task(text, options?);
await session.shell(command, options?);
```
## Where to go next
- **Want the 10-line agent?** [Headless agents with the minimal entrypoint](/docs/getting-started/headless-mode).
- **Confused by entrypoints, stateless, inline, or Temporal?** [SDK entrypoints, runtimes, and targets](/docs/reference/sdk-entrypoints-runtimes-targets).
- **New here?** [Use cases](/docs/use-cases) → [Installation](/docs/getting-started/installation) → [Your first agent](/docs/getting-started/first-agent).
- **Want CLI specifics?** [CLI reference](/docs/cli).
- **Ready to deploy?** [Deployment](/docs/deployment).
- **Building on Databricks?** [Architecture, integrations, quickstart, and validation](/docs/databricks).
- **Curious about scope?** See the [capability matrix](/docs/reference/capability-matrix).
---
# Learn TechFabric Harness
Canonical: https://harness.techfabric.com/docs/learning-paths
A progressive route through every TechFabric Harness capability, from the first finite agent to durable production and Databricks certification.
TechFabric Harness has a broad surface, but you should not learn it as a list of exports. Start with one
agent lifecycle, add only the capabilities the workload needs, and prove each boundary before moving
to the next stage. This page is the canonical map for the complete framework.
## Choose the first outcome
| You want to build | Start here | You are done when |
| --- | --- | --- |
| One bounded task with typed input and output | [Your first agent](/docs/getting-started/first-agent) | `fh run` validates the input, returns a typed result, and records a terminal run. |
| A durable conversational agent | [Persistent agents](/docs/building/persistent-agents) | Two interactions use the same address without sharing state with another identity. |
| An agent that safely takes action | [Enterprise controls](/docs/building/enterprise-controls) | An allowed action succeeds, a denied action fails before execution, and an approval is bound to the exact operation. |
| An agent that works in files or a shell | [Sandboxes](/docs/building/sandboxes) | Capability discovery, timeout, cancellation, and cleanup pass for the selected backend. |
| A recoverable long-running workflow | [Runtime modes](/docs/reference/runtime-modes) | A restart or worker replacement resumes accepted work without duplicating its external effect. |
| A web, chat, voice, or channel application | [HTTP applications](/docs/building/http-applications) | Authenticated ingress reaches the intended agent and preserves actor and tenant identity. |
| A Databricks workload | [Databricks quickstart](/docs/databricks/quickstart) | Mock authoring, workspace preflight, identity, deployment, and the target-workspace smoke test pass. |
| A production deployment | [Production readiness](/docs/reference/production-readiness) | Build verification, auth, stores, recovery, observability, deletion, and supply-chain gates pass, and [validation status](/docs/reference/validation-status) matches the enabled capability set. |
## The learning sequence
### 1. Learn the two agent lifecycles
Read [Agent APIs](/docs/getting-started/api-consistency), then build both forms:
- `defineAgent()` is a finite, bounded invocation that returns one result.
- `createAgent()` is a persistent, addressable agent initialized for each interaction.
Use [Agent anatomy](/docs/building/anatomy) for finite definitions and
[Dynamic agents and hooks](/docs/building/dynamic-agents) for persistent composition. Then inspect
the definition with `fh agents` and `fh describe` before running it.
Practice with `hello-world`, `minimal`, `finite-jobs`, and `dynamic-agent` in the
[example catalog](/docs/examples). Do not proceed until you can explain where termination bounds,
static policy, interaction state, and initialization belong.
### 2. Build context deliberately
Add context one layer at a time:
1. [Sessions and prompts](/docs/building/sessions-prompts) for the conversation boundary.
2. [Roles](/docs/building/roles) for instruction overlays.
3. [Skills](/docs/building/skills) for Markdown-first procedures.
4. [Session memory](/docs/building/session-memory) and
[context compaction](/docs/reference/compaction) for bounded continuity.
5. [Filesystem sources](/docs/reference/filesystem-sources) and
[artifacts](/docs/building/artifacts) for mounted knowledge and retained output.
Practice with `with-skill`, `with-packaged-skills`, `with-s3-source`, `data-analyst`, and
`support-agent`. Verify what enters model context, what remains server-side, and what is retained
after deletion.
### 3. Give the agent capabilities
Use [Tools](/docs/building/tools) for typed model calls, [Commands](/docs/building/commands) for
shell capabilities, [Tasks](/docs/building/tasks) and [Subagents](/docs/building/subagents) for
delegation, and [MCP](/docs/reference/mcp) for remote tool discovery.
Every capability needs:
- a schema and effect classification;
- a definition-level policy floor;
- bounded output, timeout, cancellation, and concurrency;
- explicit secret resolution outside model context; and
- deterministic failure behavior that callers can distinguish from cancellation or exhaustion.
Practice with `with-tools`, `with-finish-tool`, `with-task`, `with-local-shell`,
`with-signal-messages`, and `application-routes`.
### 4. Isolate execution
Start with the [sandbox guide](/docs/building/sandboxes), choose a backend using the
[sandbox matrix](/docs/reference/sandboxes-matrix), and learn the portable
[sandbox lifecycle](/docs/reference/sandbox-lifecycle). Use
[sandbox connectors](/docs/building/sandbox-connectors) when the environment is provider-managed.
The full path covers virtual, local, Docker, Kubernetes, Cloudflare Computer, Cloudflare Sandbox,
Databricks SQL, Daytona, E2B, Modal, and Vercel Sandbox. Provider pages under
[Ecosystem → Sandboxes](/docs/ecosystem/sandboxes) document authentication and unsupported
operations. Practice with the matching `with-*` workspace and verify create, exec, files, timeout,
cancellation, and cleanup—not only a successful command.
### 5. Add governance before mutation
Read [Policies and approvals](/docs/reference/policies-approvals), then the guided
[Approvals](/docs/building/approvals) and [Enterprise controls](/docs/building/enterprise-controls)
pages. Continue with:
- [Authentication and RBAC](/docs/operating/auth);
- [Multi-tenancy](/docs/operating/multi-tenancy);
- [Rate limiting](/docs/operating/rate-limiting);
- [Private networking](/docs/operating/private-networking);
- [Secrets, retention, and residency](/docs/operating/data-governance); and
- [Security hardening](/docs/reference/security-hardening).
Practice with `with-approval`, `private-networking`, `with-analytics-copilot`, and the Databricks
authoring workspaces. Test one allowed action, one policy denial, one expired or rejected approval,
and one cross-tenant access attempt.
### 6. Make accepted work durable
Read [Runtime modes](/docs/reference/runtime-modes), [Session stores](/docs/reference/session-stores),
and [Build and run artifacts](/docs/deployment/build-artifacts). Then add:
- checkpoints and replay with `with-checkpoint`;
- unified database persistence with `database-persistence`;
- durable child work with `with-task` and `finite-jobs`;
- Temporal activities and deterministic workflows with
[Temporal Worker](/docs/deployment/temporal-worker); and
- recovery objectives with [Backup and disaster recovery](/docs/operating/disaster-recovery).
The durability proof is a restart, cancellation, retry-classification, and idempotency test. A
successful uninterrupted run is not durability evidence.
### 7. Choose models, data, and retrieval
Use [Model providers](/docs/building/model-providers) for OpenAI-compatible, Anthropic, Bedrock,
Vertex, Azure, Databricks, Workers AI, and custom providers. Use
[Databases](/docs/ecosystem/databases) for SQLite, libSQL/Turso, Postgres, Lakebase, MySQL,
MongoDB, Redis/Valkey, and Supabase patterns.
For knowledge workloads, choose mounted files, a database tool, MCP, or provider-native retrieval
based on the data boundary. The [use-case guide](/docs/use-cases) compares those approaches.
Databricks AI Search and governed RAG have their own [RAG path](/docs/databricks/rag).
### 8. Deliver through the right interface
The same runtime can be reached through:
- the [CLI](/docs/cli), `fh fiber`, schedules, and webhooks;
- the authenticated [HTTP server](/docs/reference/http-server);
- `@fabric-harness/client` and [React hooks](/docs/building/react);
- [Channels](/docs/building/channels), including every adapter in
[Ecosystem → Channels](/docs/ecosystem/channels); and
- [Voice](/docs/building/voice) with a selected [voice provider](/docs/building/voice-providers).
Practice with `react-chat`, `scheduled-jobs`, `with-channel-adapters`, `with-slack-channel`, and
`voice-data-collector`. Validate signature/authentication failures, duplicate delivery, retries,
stream reconnection, and identity propagation.
### 9. Build and deploy portably
Learn `fh build`, `fh deploy`, build manifests, package verification, and target-specific preflight
from [Deployment overview](/docs/deployment). The maintained paths cover Node, Docker, Temporal,
Cloudflare, Azure/ACI/AKS, Foundry Hosted Agents, Databricks Apps and serving proxies, AWS/SST,
Fly.io, Railway, Render, GitHub Actions, and GitLab CI. Remote sandbox providers are execution
backends and do not replace the deployment runtime.
For each target, follow its page through prerequisites, build, validation, expected health result,
failure behavior, and cleanup. Use [Portable agent packages](/docs/deployment/portable-packages)
when another repository consumes the artifact.
### 10. Operate and improve the system
Use [Agent events](/docs/reference/events), [Telemetry](/docs/reference/telemetry), and
[Cost attribution](/docs/operating/cost-attribution) before production traffic. Add:
- [Evaluations](/docs/building/evals) and the [eval library](/docs/reference/eval-library);
- [Operator Console](/docs/operating/operator-console);
- [Operational SLOs](/docs/operating/operational-slos);
- [Audit export](/docs/operating/audit-export);
- [Supply-chain evidence](/docs/operating/supply-chain); and
- the [production readiness checklist](/docs/reference/production-readiness) and current
[validation status](/docs/reference/validation-status).
Practice with `with-observability` and `operational-slos`. Prove redaction, correlation, alerting,
retention, deletion, and recovery instead of treating emitted logs as sufficient observability.
### 11. Learn the Databricks path in order
Databricks owns data, models, identity, compute, governance, and hosting. Harness owns the portable,
durable agent-control boundary around those services.
1. [Choose a workload](/docs/databricks/workloads) and read
[why Harness](/docs/databricks/why-fabric).
2. Complete the [quickstart](/docs/databricks/quickstart) and
[local naming/authentication](/docs/databricks/local-naming-auth).
3. Choose native services from [Databricks integrations](/docs/databricks/integrations): AI
Gateway and Model Serving, SQL, Unity Catalog, AI Search, RAG, Genie, Jobs, Lakeflow, Feature
Serving, Lakebase, MLflow, managed MCP, Agent Services, Supervisor Agents, Knowledge Assistants,
managed memory, Apps, and ResponsesAgent.
4. Learn approval-bound writes in [resource management](/docs/databricks/authoring), including
Jobs, Lakeflow, AI Search, Serving, Unity Catalog, Workspace, secret references, Asset Bundles,
and Genie Agents.
5. Use the dedicated [Genie Agent Mode path](/docs/databricks/genie-agent-mode) for the explicit
Beta create, permission, streaming, model-tool, failure, and cleanup journey.
6. Deploy through the [Databricks App tutorial](/docs/deployment/databricks-app) and add Lakebase or
Temporal durability when the workload requires it.
7. Check [workspace compatibility](/docs/databricks/compatibility) and
[live certification](/docs/databricks/live-certification), then compare the exact package and
target against [validation status](/docs/reference/validation-status) before making a production
claim.
Preview availability is workspace-dependent. Contract support, a successful optional live probe,
and release-blocking certification are different claims and remain labeled separately.
## Complete feature-family map
| Feature family | Primary learning guide | Runnable practice | Production proof |
| --- | --- | --- | --- |
| Finite agents, schemas, middleware, nested invocation | [Agent anatomy](/docs/building/anatomy) | `minimal`, `finite-jobs` | Bounded terminal result and idempotent child invocation |
| Persistent agents, hooks, state, finish guards | [Dynamic agents](/docs/building/dynamic-agents) | `dynamic-agent` | Identity-isolated interactions and restart continuity |
| Sessions, prompts, roles, skills, memory, compaction | [Sessions and prompts](/docs/building/sessions-prompts) | `with-skill`, `with-packaged-skills` | Context bounds, redaction, retention, and deletion |
| Tools, commands, tasks, subagents, finish tools, signals | [Tools](/docs/building/tools) | `with-tools`, `with-task`, `with-finish-tool`, `with-signal-messages` | Schema denial, effect policy, timeout, cancellation |
| Filesystem sources, artifacts, attachments | [Filesystem sources](/docs/reference/filesystem-sources) | `with-s3-source`, `data-analyst` | Mount bounds, artifact integrity, cleanup |
| MCP and connector recipes | [MCP](/docs/reference/mcp) | `with-databricks-managed-mcp` | Authenticated discovery, allowlist, effect preservation |
| Sandboxes and isolated execution | [Sandbox matrix](/docs/reference/sandboxes-matrix) | Docker, Kubernetes, Cloudflare, Daytona, E2B, Modal, Vercel workspaces | Capability, isolation, timeout, cancellation, reclamation |
| Policy, approvals, budgets, secrets | [Enterprise controls](/docs/building/enterprise-controls) | `with-approval`, `with-analytics-copilot` | Allow, deny, approval, expiry, secret-redaction tests |
| Stores, checkpoints, replay, Temporal | [Runtime modes](/docs/reference/runtime-modes) | `with-checkpoint`, `database-persistence`, `with-temporal` | Crash recovery and no duplicate external effect |
| HTTP, client, React, Fiber, schedules | [HTTP applications](/docs/building/http-applications) | `application-routes`, `react-chat`, `scheduled-jobs` | Auth, reconnect, duplicate delivery, graceful shutdown |
| Channels and voice | [Channels](/docs/building/channels) | `with-channel-adapters`, `voice-data-collector` | Signature, identity, retry, deduplication tests |
| Models, databases, retrieval | [Model providers](/docs/building/model-providers) | `support-agent`, `data-analyst`, provider examples | Credential, authorization, data-boundary tests |
| Evals, telemetry, SLOs, audit, cost | [Evaluations](/docs/building/evals) | `with-observability`, `operational-slos` | Redacted correlated evidence and release thresholds |
| Registry, boundary, portable packages | [Agent registry](/docs/building/agent-registry) | `agent-registry` | Immutable definition, grants, provenance, compatibility |
| Deployment targets and supply chain | [Deployment overview](/docs/deployment) | Target-specific workspaces | Verified artifact, health, rollback, cleanup |
| Databricks data, AI, authoring, Apps, durability | [Databricks](/docs/databricks) | All `with-databricks-*` workspaces | Target-workspace preflight and retained certification |
## Public package map
| Package | Learn it here |
| --- | --- |
| `@fabric-harness/sdk` | [Agent APIs](/docs/getting-started/api-consistency), [building agents](/docs/building/anatomy) |
| `@fabric-harness/cli` | [CLI overview](/docs/cli) |
| `@fabric-harness/node` | [Node deployment](/docs/deployment/node), [HTTP server](/docs/reference/http-server) |
| `@fabric-harness/client` | [HTTP applications](/docs/building/http-applications) |
| `@fabric-harness/react` | [React applications](/docs/building/react) |
| `@fabric-harness/vite` | [Vite integration](/docs/reference/vite-integration) |
| `@fabric-harness/temporal` | [Temporal Worker](/docs/deployment/temporal-worker) |
| `@fabric-harness/cloudflare` | [Cloudflare deployment](/docs/deployment/cloudflare) |
| `@fabric-harness/azure` | [Azure deployment](/docs/deployment/azure) |
| `@fabric-harness/databricks` | [Databricks learning path](/docs/databricks) |
| `@fabric-harness/connectors` | [Connector catalog](/docs/building/connector-catalog) |
| `@fabric-harness/channels` | [Channels](/docs/building/channels) |
| `@fabric-harness/databases` | [Databases](/docs/ecosystem/databases) |
| `@fabric-harness/evals` | [Evaluations](/docs/building/evals) |
| `@fabric-harness/agent-registry` | [Agent registry and governance](/docs/building/agent-registry) |
| `@fabric-harness/agent-boundary` | [Portable packages](/docs/deployment/portable-packages) |
## How to know a feature is learned
For every feature you adopt, keep six things together:
1. **Prerequisites:** runtime, package, provider, authentication, permissions, and preview/SKU needs.
2. **Minimal path:** the smallest public-API configuration that demonstrates the capability.
3. **Expected result:** observable output, event, artifact, state transition, or health response.
4. **Failure path:** at least one denial, invalid input, cancellation, timeout, or unavailable-provider test.
5. **Cleanup:** the exact resource, store, session, sandbox, or deployment reclamation action.
6. **Evidence:** a deterministic test locally and a provider-backed smoke or certification record where
the feature depends on managed infrastructure.
The [example catalog](/docs/examples) is indexed by these expectations. API signatures remain in the
[generated reference](/docs/reference/api); the reference is a lookup tool, not the learning order.
Use [Runnable examples](/docs/examples/runnable) for synchronized commands,
expected evidence, provider limitations, failure behavior, and cleanup across every workspace.
---
# Runnable examples
Canonical: https://harness.techfabric.com/docs/examples/runnable
Runnable commands, expected proof, failure behavior, and cleanup for every maintained TechFabric Harness example.
{/* Generated by scripts/generate-example-learning-guide.mjs. Edit the example package, README, track contract, or generator; do not edit this file directly. */}
This guide is synchronized with every top-level workspace under `examples/`. Use it after the
[complete framework learning path](/docs/learning-paths) to choose a runnable proof for
a feature. Commands assume an authorized source checkout from the repository root.
For a public project without source access, initialize a workspace with:
```sh
npx --yes @fabric-harness/cli@latest init
```
Then copy the public API pattern from the linked feature guide. An example build proves local
composition; it does not establish readiness for a managed provider.
## How to run a row
Run the first command as the deterministic or build proof. Run the second command, when present,
for the workload or development server. `pnpm --dir` executes the script from the example's own
workspace, so the commands work when copied from the repository root.
## Framework foundations
Learn finite and persistent lifecycles, state, schemas, configuration, tools, approvals, tasks, schedules, application routes, and browser clients before adding a managed runtime.
- **Prerequisites and authentication:** Node.js 22 or newer and the repository pnpm version. Use the mock commands where shown; model-backed commands additionally need the credential for the model configured by that workspace.
- **Expected evidence:** The selected check exits successfully or the run returns the typed result, durable state transition, event, or browser behavior described in the proof column.
- **Failure behavior:** Invalid input, policy denial, approval rejection, exhaustion, cancellation, timeout, and unknown capabilities remain distinct terminal outcomes. A mock path never establishes provider readiness.
- **Cleanup:** Stop development servers with Ctrl-C. Retain local run records only when they are needed for logs or replay; remove example-local databases and artifacts when the proof is complete.
| Example | Command | What it proves |
| --- | --- | --- |
| [`agent-registry`](/docs/reference/source-access) | `pnpm --dir examples/agent-registry run check` `pnpm --dir examples/agent-registry run run` | This credential-free example validates a vertical-neutral agent definition, pins an immutable version and model budget, separates proposal authority from execution authority, and proves that an `approval-required` autonomy ceiling prevents direct execution. |
| [`application-routes`](/docs/reference/source-access) | `pnpm --dir examples/application-routes run check` `pnpm --dir examples/application-routes run dev` | Runs authenticated custom Fetch routes and middleware, including a report route and durable delivery to an addressable customer agent. |
| [`with-mounted-agent-router`](/docs/reference/source-access) | `pnpm --dir examples/with-mounted-agent-router run build` `pnpm --dir examples/with-mounted-agent-router run run` | Mounts the persistent-agent HTTP surface inside an application that already owns its own server, under its own prefix and behind its own middleware. |
| [`database-persistence`](/docs/reference/source-access) | `pnpm --dir examples/database-persistence run check` `pnpm --dir examples/database-persistence run dev` | Uses one persistence contract for sessions, submissions, conversation offsets, attachments, finite runs, budgets, and cascade deletion across local libSQL and optional managed databases. |
| [`dynamic-agent`](/docs/reference/source-access) | `pnpm --dir examples/dynamic-agent run build` `pnpm --dir examples/dynamic-agent run dev` | This example composes a persistent agent with hooks. It stores milestone state durably, unlocks `advanced_analysis` after the first milestone, upgrades its model after the second, streams a named `data-progress` part, attaches response metadata, and uses an agent-finish guard to keep the same response working until `complete_milestone` runs. |
| [`finite-jobs`](/docs/reference/source-access) | `pnpm --dir examples/finite-jobs run check` `pnpm --dir examples/finite-jobs run dev` | Runs a typed parent job, admits an idempotent child run through middleware, and exposes the child receipt and event stream through the public run protocol. |
| [`hello-world`](/docs/reference/source-access) | `pnpm --dir examples/hello-world run build` | Defines the functionally named `hello` and `ask` finite agents, then proves config defaults, credential-free mock execution, and optional durable Temporal execution. |
| [`minimal`](/docs/reference/source-access) | `pnpm --dir examples/minimal run build` | The smallest possible TechFabric Harness agent. No input schema, no output schema, no capability policy, no session store, no artifacts. Just `defineAgent({...})` + `session.prompt` against a model of your choice. |
| [`react-chat`](/docs/reference/source-access) | `pnpm --dir examples/react-chat run check` `pnpm --dir examples/react-chat run dev` | Connects a React client to the public Harness protocol, streams transient SSE updates, reconciles them with durable messages, and exercises responsive light and dark layouts. |
| [`scheduled-jobs`](/docs/reference/source-access) | `pnpm --dir examples/scheduled-jobs run check` `pnpm --dir examples/scheduled-jobs run dev` | Declares a weekday report schedule and demonstrates local Node execution plus Cloudflare Cron generation, tenant-aware admission, and public run inspection. |
| [`with-approval`](/docs/reference/source-access) | `pnpm --dir examples/with-approval run build` | Demonstrates policy-gated tool execution. |
| [`with-checkpoint`](/docs/reference/source-access) | `pnpm --dir examples/with-checkpoint run build` `pnpm --dir examples/with-checkpoint run run` | The `workspace-recovery` finite agent proves that a local sandbox can return to a named checkpoint. It writes one file, takes a checkpoint, writes a second file, restores the checkpoint, and reports the recovered filesystem state. |
| [`with-config`](/docs/reference/source-access) | `pnpm --dir examples/with-config run build` `pnpm --dir examples/with-config run run` | Demonstrates central workspace configuration through `.fabricharness/config.ts`. |
| [`with-finish-tool`](/docs/reference/source-access) | `pnpm --dir examples/with-finish-tool run build` `pnpm --dir examples/with-finish-tool run run` | Demonstrates the `finish` / `give_up` result tool pattern for structured output. |
| [`with-imported-skills`](/docs/reference/source-access) | `pnpm --dir examples/with-imported-skills run build` `pnpm --dir examples/with-imported-skills run run` | Demonstrates **build-time skill imports**: a skill authored as a `SKILL.md` directory anywhere in the project, imported directly by the agent that uses it. |
| [`with-packaged-skills`](/docs/reference/source-access) | `pnpm --dir examples/with-packaged-skills run build` `pnpm --dir examples/with-packaged-skills run run` | Demonstrates packaged skills with lazy resource loading. |
| [`with-signal-messages`](/docs/reference/source-access) | `pnpm --dir examples/with-signal-messages run build` `pnpm --dir examples/with-signal-messages run run` | Demonstrates `session.createSignalEntry()` for injecting typed signal entries into the session history. |
| [`with-skill`](/docs/reference/source-access) | `pnpm --dir examples/with-skill run build` `pnpm --dir examples/with-skill run run` | The `personalized-greeter` finite agent loads a Markdown-first `hello-skill` and the `friendly` role from the embedded `.fabricharness/` workspace, then invokes the skill with typed arguments. |
| [`with-task`](/docs/reference/source-access) | `pnpm --dir examples/with-task run build` `pnpm --dir examples/with-task run run` | Demonstrates parent/child task orchestration with shared durable session storage. |
| [`with-tools`](/docs/reference/source-access) | `pnpm --dir examples/with-tools run build` `pnpm --dir examples/with-tools run run` | Demonstrates built-in sandbox file tools with durable session history. |
## Runtimes, sandboxes, sources, and stores
Move the same public lifecycle across local, container, remote-sandbox, durable workflow, object-source, and database boundaries.
- **Prerequisites and authentication:** Start with the local or Docker proof. Provider-backed workspaces require the provider credential, target account, supported region or plan, and any runtime-specific CLI named in its guide.
- **Expected evidence:** A build proves adapter composition. A live run must additionally prove capability discovery, bounded command or file behavior, cancellation, and resource reclamation on the selected backend.
- **Failure behavior:** Missing credentials, unsupported operations, isolation escape attempts, network denial, process failure, timeout, cancellation, and cleanup failure surface explicitly; Harness does not redirect work to the host or another backend.
- **Cleanup:** Every live sandbox or deployment must stop its exact remote resource. Source mounts are read-only and detach with the session; durable stores retain only the records selected by the example retention policy.
| Example | Command | What it proves |
| --- | --- | --- |
| [`coding-agent-lite`](/docs/reference/source-access) | `pnpm --dir examples/coding-agent-lite run build` | Clones a repo into a Docker sandbox, installs dependencies, then runs an arbitrary prompt against the cloned source. This demonstrates TechFabric Harness lite mode for coding-agent workflows in about 13 lines. |
| [`remote-coding-agent`](/docs/reference/source-access) | `pnpm --dir examples/remote-coding-agent run build` | Provider-neutral remote sandbox example for coding agents. |
| [`with-azure`](/docs/reference/source-access) | `pnpm --dir examples/with-azure run build` `pnpm --dir examples/with-azure run build:foundry` | Minimal TechFabric Harness agent that runs through an Azure OpenAI deployment with Azure Key Vault-resolved secrets. |
| [`with-cloudflare-sandbox`](/docs/reference/source-access) | `pnpm --dir examples/with-cloudflare-sandbox run build` `pnpm --dir examples/with-cloudflare-sandbox run build:cloudflare` | Builds a Cloudflare Worker artifact that uses Cloudflare Sandbox containers for shell and filesystem operations. It includes a finite job and a persistent agent backed by Durable Object admission, FIFO leases, offset streams, attachment storage, abort, reconciliation, and cascade deletion. |
| [`with-cloudflare-shell-workspace`](/docs/reference/source-access) | `pnpm --dir examples/with-cloudflare-shell-workspace run build` `pnpm --dir examples/with-cloudflare-shell-workspace run build:cloudflare` | This example targets Cloudflare Workers with the early-preview `@cloudflare/computer` Workspace backend. |
| [`with-cloudflare-workers-ai`](/docs/reference/source-access) | `pnpm --dir examples/with-cloudflare-workers-ai run build` `pnpm --dir examples/with-cloudflare-workers-ai run dev` | Builds and optionally deploys a Cloudflare Worker that resolves Workers AI through its native binding without an API key in agent state. |
| [`with-daytona`](/docs/reference/source-access) | `pnpm --dir examples/with-daytona run build` `pnpm --dir examples/with-daytona run run` | Runs a bounded shell workload in a Daytona-managed remote development sandbox. |
| [`with-docker`](/docs/reference/source-access) | `pnpm --dir examples/with-docker run build` | Runs standard Harness file and shell tools inside a Docker container with a scoped workspace mount. |
| [`with-e2b`](/docs/reference/source-access) | `pnpm --dir examples/with-e2b run build` `pnpm --dir examples/with-e2b run run` | Runs a finite job in E2B, streams bounded command output, and removes the sandbox during session cleanup. |
| [`with-kubernetes`](/docs/reference/source-access) | `pnpm --dir examples/with-kubernetes run build` `pnpm --dir examples/with-kubernetes run run` | The `kubernetes-system-inspector` finite agent attaches a Harness sandbox to a Kubernetes pod via the `kubernetesSandbox()` adapter from `@fabric-harness/connectors/k8s` and runs a bounded diagnostic command in `/workspace`. |
| [`with-local-shell`](/docs/reference/source-access) | `pnpm --dir examples/with-local-shell run build` `pnpm --dir examples/with-local-shell run run` | The `node-version-inspector` finite agent uses the local sandbox and `session.shell()` to report the Node.js version available to a Harness workload. |
| [`with-modal`](/docs/reference/source-access) | `pnpm --dir examples/with-modal run build` `pnpm --dir examples/with-modal run run` | Runs a finite Harness workload in a Modal serverless sandbox through the portable sandbox contract. |
| [`with-postgres-store`](/docs/reference/source-access) | `pnpm --dir examples/with-postgres-store run build` `pnpm --dir examples/with-postgres-store run run` | Demonstrates configuring one Postgres bundle for sessions, submissions, conversation streams, attachments, finite runs, and cost budgets. |
| [`with-s3-source`](/docs/reference/source-access) | `pnpm --dir examples/with-s3-source run build` `pnpm --dir examples/with-s3-source run run` | The `s3-report-browser` finite agent mounts a bounded S3 prefix as a read-only Harness source and lists its files with the built-in `glob` tool. It needs no embeddings, retrieval service, or vector database. |
| [`with-temporal`](/docs/reference/source-access) | `pnpm --dir examples/with-temporal run build` `pnpm --dir examples/with-temporal run run:mock` | Runs the `temporal-agent` through a real Temporal worker with deterministic workflow boundaries, plus a credential-free Node-target mock path for local verification. |
| [`with-vercel-sandbox`](/docs/reference/source-access) | `pnpm --dir examples/with-vercel-sandbox run build` `pnpm --dir examples/with-vercel-sandbox run run` | Runs a finite job through Vercel Sandbox, forwards timeout cancellation, and stops the sandbox during cleanup. |
## Delivery, security, and operations
Prove authenticated ingress, channel identity, private networking, redacted telemetry, evaluations, and operational objectives around the agent lifecycle.
- **Prerequisites and authentication:** The deterministic checks need only Node.js and pnpm. Live channel and voice paths require their signing secrets, provider credentials, callback URLs, and a store suitable for duplicate-delivery and restart tests.
- **Expected evidence:** Evidence includes verified ingress identity, governed replies, redacted correlation fields, evaluated SLO decisions, or an allowed network path paired with a denied bypass.
- **Failure behavior:** Bad signatures, stale timestamps, duplicate delivery, tenant mismatch, redaction violations, unavailable exporters, and disallowed network destinations fail closed or enter the documented retry path.
- **Cleanup:** Stop local listeners, remove temporary provider callbacks and test subscriptions, revoke short-lived credentials, and retain only redacted operational evidence required by policy.
| Example | Command | What it proves |
| --- | --- | --- |
| [`operational-slos`](/docs/reference/source-access) | `pnpm --dir examples/operational-slos run test` | Exports stable Harness operational metrics, evaluates the reference SLO policy, and validates importable Prometheus and Grafana assets. |
| [`private-networking`](/docs/reference/source-access) | `pnpm --dir examples/private-networking run test` `pnpm --dir examples/private-networking run start` | Proves that an allowed request succeeds through the configured proxy while a direct network bypass fails. |
| [`voice-data-collector`](/docs/reference/source-access) | `pnpm --dir examples/voice-data-collector run build` | Uses OpenAI Realtime to collect functionally named fields, submits typed `submit_field` calls, and persists the resulting field map in session memory. |
| [`with-channel-adapters`](/docs/reference/source-access) | `pnpm --dir examples/with-channel-adapters run check` `pnpm --dir examples/with-channel-adapters run dev` | This runnable workspace mounts all 18 first-party channel adapters on one persistent agent. |
| [`with-observability`](/docs/reference/source-access) | `pnpm --dir examples/with-observability run test` | This example emits the same redacted Fabric observability record to Braintrust, Jetty, and Sentry adapter callbacks, with stable job, agent, session, submission, and tenant correlation. The `observability.eval.ts` fixture runs through `@fabric-harness/evals` and Vitest. OpenTelemetry's hierarchical span adapter is covered by `packages/sdk/test/otel-observer.test.ts` and accepts the same correlation fields. |
| [`with-slack-channel`](/docs/reference/source-access) | `pnpm --dir examples/with-slack-channel run build` `pnpm --dir examples/with-slack-channel run dev` | An end-to-end **channel** example: a Slack app mention drives a persistent agent, which replies in the same thread. Demonstrates the channels workstream (design) — webhook ingress → dispatch → outbound tool, with exactly-once delivery and per-user identity. |
## End-to-end workload examples
Apply the framework to recognizable engineering and support work without treating the workload prompt as a substitute for lifecycle, policy, and cleanup controls.
- **Prerequisites and authentication:** Most workloads require the model credential declared by the example. Shell or repository workloads also need their named local tools, fixtures, and read or write permissions.
- **Expected evidence:** The result should match the typed workload artifact—review, reproduction, documentation, analysis, migration, support answer, test, or release note—and preserve the run evidence used to reproduce it.
- **Failure behavior:** Missing inputs, unavailable models, malformed typed output, denied commands, unsafe mutations, and exhausted limits fail without silently returning an unvalidated artifact.
- **Cleanup:** These examples should leave external systems unchanged unless an approval-bound action explicitly says otherwise. Remove temporary clones, generated fixtures, and local run records after reviewing the result.
| Example | Command | What it proves |
| --- | --- | --- |
| [`api-docs-generator`](/docs/reference/source-access) | `pnpm --dir examples/api-docs-generator run build` `pnpm --dir examples/api-docs-generator run run` | Generate MDX docs pages for HTTP routes. Demonstrates `fumadocsSource` to give the agent your existing docs as a tone/structure reference. |
| [`bug-reproducer`](/docs/reference/source-access) | `pnpm --dir examples/bug-reproducer run build` `pnpm --dir examples/bug-reproducer run run` | Convert a free-form bug report into a minimal failing test. Pure inference — no shell required, runs anywhere. |
| [`changelog-writer`](/docs/reference/source-access) | `pnpm --dir examples/changelog-writer run build` `pnpm --dir examples/changelog-writer run run` | Generate a Keep-a-Changelog markdown section between two git refs. |
| [`code-review`](/docs/reference/source-access) | `pnpm --dir examples/code-review run build` | Provides separate read-only `code-review` and `pr-review` agents that return typed findings and a final recommendation for local source or a GitHub pull request. |
| [`data-analyst`](/docs/reference/source-access) | `pnpm --dir examples/data-analyst run build` | Analyzes a CSV inside a network-isolated Docker sandbox and stores reproducible `analysis.md` and `summary.json` artifacts in the Harness session store. |
| [`dependency-auditor`](/docs/reference/source-access) | `pnpm --dir examples/dependency-auditor run build` `pnpm --dir examples/dependency-auditor run run` | Run `npm` / `pnpm` audit and ask the model to prioritize the findings. |
| [`incident-runbook`](/docs/reference/source-access) | `pnpm --dir examples/incident-runbook run build` `pnpm --dir examples/incident-runbook run run` | Match an alert to a mounted runbook (Fumadocs/Mintlify MDX) and walk through diagnostic steps. Read-only — never executes. |
| [`issue-triage-ci`](/docs/reference/source-access) | `pnpm --dir examples/issue-triage-ci run build` | Runs a cautious CI issue-triage agent that is read-only by default, emits inspectable artifacts, denies publishing operations, and approval-gates issue comments. |
| [`release-notes`](/docs/reference/source-access) | `pnpm --dir examples/release-notes run build` `pnpm --dir examples/release-notes run run` | Customer-facing release notes from merged PRs between two tags. |
| [`schema-migration`](/docs/reference/source-access) | `pnpm --dir examples/schema-migration run build` `pnpm --dir examples/schema-migration run run` | Draft a SQL migration with `up`/`down` statements. Apply commands are gated behind approval in complete entrypoint. |
| [`support-agent`](/docs/reference/source-access) | `pnpm --dir examples/support-agent run build` | A customer support agent that answers questions by searching a knowledge base. The knowledge base is just a directory of markdown files mounted into the agent's sandbox — no vector store, no embeddings, no retrieval pipeline. |
| [`support-agent-cloudflare-r2`](/docs/reference/source-access) | `pnpm --dir examples/support-agent-cloudflare-r2 run build` | Support-agent example that mounts a Cloudflare R2 bucket prefix into the Fabric sandbox as normal files. |
| [`support-agent-foundry`](/docs/reference/source-access) | `pnpm --dir examples/support-agent-foundry run build` | Support-agent example shaped for Microsoft Foundry Hosted Agents. |
| [`test-generator`](/docs/reference/source-access) | `pnpm --dir examples/test-generator run build` `pnpm --dir examples/test-generator run run` | Generate a Vitest spec for a TypeScript source file. Includes a `fixtures/src/util.ts` so `run` works clean. |
## Databricks workloads
Progress from credential-free composition to target-workspace identity, governed data and AI access, native resource authoring, App bindings, durability, and protected certification.
- **Prerequisites and authentication:** Run the mock or local contract first. Live paths require Databricks OAuth, least-privilege workspace and Unity Catalog permissions, the documented SKU or preview, and the exact resource identifiers named by the workspace.
- **Expected evidence:** A local check proves Harness composition only. A live proof must record target-workspace preflight, identity, native service evidence, bounded agent behavior, cleanup, and the certification tier that actually ran.
- **Failure behavior:** Authentication, authorization, preview, policy, stale-fingerprint, ambiguous-write, cancellation, timeout, and native terminal failures remain explicit. Non-idempotent or ambiguous writes are not retried automatically.
- **Cleanup:** Trash or delete only the exact Harness-owned temporary resource in a finally path, wait for the native terminal state, then remove its ownership record. Never remove a pre-existing customer resource as example cleanup.
| Example | Command | What it proves |
| --- | --- | --- |
| [`with-analytics-copilot`](/docs/reference/source-access) | `pnpm --dir examples/with-analytics-copilot run build` `pnpm --dir examples/with-analytics-copilot run start` | Separates safe single-statement SQL reads from approval-bound arbitrary SQL and Genie management, with local validation before any Databricks call. |
| [`with-databricks`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks run build` `pnpm --dir examples/with-databricks run run` | Lighthouse use case #1 from the Databricks platform plan: a governed analytics copilot that answers questions over the lakehouse, with **Unity Catalog enforcing access** and Fabric adding approval routing, lineage, and an egress allowlist on top. |
| [`with-databricks-agent-service`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-agent-service run test` `pnpm --dir examples/with-databricks-agent-service run start` | This example registers an externally hosted TechFabric Harness agent in Unity Catalog, verifies discovery and metadata updates, reads its grants, optionally exercises grant/revoke, and deletes the temporary registration. It defaults the external route to the Harness `/responses` endpoint. Agent Services is not yet represented in the modular TypeScript SDK, so Fabric keeps its preview transport private and exposes this lifecycle through the typed `databricks()` bundle. |
| [`with-databricks-app-resources`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-app-resources run check` `pnpm --dir examples/with-databricks-app-resources run build` | Build a TechFabric Harness agent as a Databricks App while keeping Databricks Apps, Declarative Automation Bundles, Unity Catalog, Jobs, Model Serving, SQL Warehouses, and Databricks Secrets authoritative. |
| [`with-databricks-appkit-interop`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-appkit-interop run test` `pnpm --dir examples/with-databricks-appkit-interop run start` | This runnable spike pins `@databricks/appkit` 0.53.0 and proves the safe additive boundary: an AppKit beta agent discovers and invokes a read-only tool exposed by TechFabric Harness over MCP. The Harness server remains authoritative for authenticated identity, tenant context, policy version, canonical argument digest, cancellation signal, and tool-call evidence. |
| [`with-databricks-authoring-admin`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-authoring-admin run build` `pnpm --dir examples/with-databricks-authoring-admin run run` | Combines custom-model serving endpoint management, non-destructive Unity Catalog administration, workspace notebook writes, and secret-reference writes. All surfaces are opt-in and route to the `platform-admin` approval audience. The secret tool receives `{kind:'secret', name}` and resolves `APP_RUNTIME_SECRET` server-side; raw material never enters model tool input or lineage. |
| [`with-databricks-bundle-deploy`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-bundle-deploy run build` `pnpm --dir examples/with-databricks-bundle-deploy run run` | Validates, deploys, and runs a checked-in Databricks Asset Bundle (DAB) through governed model tools. Harness does not generate the bundle YAML — `bundle/databricks.yml` and its notebooks stay checked-in infrastructure-as-code. The agent owns only the lifecycle: `databricks_bundle_validate` is read-only, while `databricks_bundle_deploy`, `databricks_bundle_run`, and `databricks_bundle_destroy` are mutations that require `data-platform` steward approval. Deploy records a sha256 fingerprint of the bundle source tree in a managed-resource store, so drift and foreign deployments are detected instead of stomped. After deploy, the agent can kick off a run of a bundle-defined job or pipeline by its `databricks.yml` resource key with `databricks_bundle_run`; submission is always `--no-wait`, and run status is polled through the bounded jobs/lakeflow status tools rather than waited on inside the tool call. |
| [`with-databricks-compute`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-compute run test` `pnpm --dir examples/with-databricks-compute run start` | Keeps SQL Warehouses, SQL-backed portable sandboxes, asynchronous Jobs, one-off notebook submission, and Databricks App hosting as distinct execution choices. |
| [`with-databricks-cost-attribution`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-cost-attribution run build` `pnpm --dir examples/with-databricks-cost-attribution run start` | Demonstrates `databricksTenantCostLimit` with a mocked generated Statement Execution client, so the example runs **without any environment variables** or a live warehouse. |
| [`with-databricks-dataeng`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-dataeng run build` `pnpm --dir examples/with-databricks-dataeng run run` | Lighthouse use case #3 from the Databricks platform plan: a data-engineering agent that manages **Lakeflow Declarative Pipelines** (DLT) and inspects Delta tables, under Unity Catalog governance, on a **Temporal worker**. Pipeline runs are among the heaviest continuous-compute consumption surfaces. |
| [`with-databricks-genie-authoring`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-genie-authoring run test` `pnpm --dir examples/with-databricks-genie-authoring run run:lifecycle` | Teaches approval-gated Genie authoring, Agent Mode invocation of an existing resource, and a complete two-identity lifecycle with exact-resource cleanup in `finally`. |
| [`with-databricks-jobs-authoring`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-jobs-authoring run build` `pnpm --dir examples/with-databricks-jobs-authoring run run` | Creates, verifies, runs/repairs, explicitly updates, and deletes a multi-task Databricks Job through governed model tools. Every write/execute requires `data-platform` steward approval. Classic compute must use the configured Databricks cluster policy and stays within worker, concurrency, timeout, runtime, node-type, and tag bounds. |
| [`with-databricks-managed-mcp`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-managed-mcp run build` `pnpm --dir examples/with-databricks-managed-mcp run start` | This runnable example connects Harness's Databricks adapter to a local Streamable HTTP MCP server, discovers one explicitly allowed tool, invokes it under a rotating OBO credential, and records a governed lineage event. The local server stands in for Databricks managed Genie MCP, so no workspace or credentials are required. |
| [`with-databricks-ontology-export`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-ontology-export run build` `pnpm --dir examples/with-databricks-ontology-export run start` | Runs the Genie Ontology exporter end to end with **no Databricks credentials**: a recorded lending-shaped module manifest goes through `planModuleOntologyExport` (UC metric-view DDL, glossary terms, domain assignments, policy classifications, derived Genie Agent spec fingerprint, and the operational-truth feed table specs), recorded Platform history (AssetEvent / ActionInvocation / PolicyEvaluation records) goes through `planOntologyFeedHydration` with a demo redaction producer to produce merge-ready feed rows and deterministic `MERGE ... WHEN NOT MATCHED THEN INSERT` statements, and the idempotent re-run is proven against an in-memory stand-in for the Delta merge. |
| [`with-databricks-rag`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-rag run build` `pnpm --dir examples/with-databricks-rag run run` | Lighthouse use case #2 from the Databricks platform plan: a RAG support agent that answers from a Databricks AI Search knowledge base, with a Unity AI Gateway model service reasoning over the retrieved passages under Unity Catalog governance. |
| [`with-databricks-rag-admin`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-rag-admin run build` `pnpm --dir examples/with-databricks-rag-admin run run` | Demonstrates endpoint and delta-sync index lifecycle alongside the existing governed query tool. Set Databricks OAuth credentials plus `DATABRICKS_CATALOG`, `DATABRICKS_EXISTING_INDEX`, and `DATABRICKS_TEXT_COLUMN`; the principal needs Vector Search endpoint/index administration and `SELECT` on the source Delta table. `aiSearchAdmin` is bound to a resource policy, so also set `DATABRICKS_ADMIN_ENDPOINT`, `DATABRICKS_ADMIN_INDEX` (catalog-qualified), and `DATABRICKS_EMBEDDING_ENDPOINT` — those are the only endpoint, index, and embedding model the admin tools accept, pinned into the tool schemas and re-checked at call time. The embedding endpoint is bounded separately because the source column's contents are sent to it. Run `pnpm build && pnpm run`. |
| [`with-databricks-revenue-ops`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-revenue-ops run test` `pnpm --dir examples/with-databricks-revenue-ops run run:mock` | This is the opinionated customer reference for a governed, durable agent on Databricks. Databricks owns data, identity, Genie, the SQL Warehouse, the forecast Job, App hosting, Lakebase, and MLflow. Harness adds a bounded agent lifecycle, exact-operation approval, stable delivery identity, restart-safe submissions, and correlated terminal evidence. |
| [`with-databricks-simple`](/docs/reference/source-access) | `pnpm --dir examples/with-databricks-simple run test` `pnpm --dir examples/with-databricks-simple run run` | Minimal runnable TechFabric Harness example that demonstrates the Databricks happy path with mocked APIs — no real credentials required. |
## Provider evidence and release claims
Provider-backed examples have three distinct evidence levels:
1. **Local contract:** types, policy, serialization, and failure behavior pass without provider credentials.
2. **Live smoke:** the named account, workspace, region, identity, and resource pass one bounded run and cleanup.
3. **Protected certification:** retained evidence meets the release gate and scope stated by that provider's certification page.
Do not turn a build, mock run, optional probe, preview enrollment, or single-region smoke into a
general production claim. Databricks claims must match the current
[public certification record](/docs/databricks/live-certification).
---
# TechFabric Harness on Databricks
Canonical: https://harness.techfabric.com/docs/databricks
Build governed TypeScript agents with the Responses API, Unity AI Gateway, Unity Catalog, SQL Warehouses, Lakebase, MLflow and Databricks Apps.
## The 30 second answer
**Databricks owns the data, models, identity, and hosting plane. Fabric is an optional durable
control layer for agents that use that plane.**
| Question | Answer |
| --- | --- |
| Does Fabric replace Unity Catalog / Jobs / MLflow? | **No.** Those stay native and authoritative. |
| Does Fabric overlap AppKit? | **Yes**, at the agent-runtime layer (threads, tools, request-time approval). |
| When is Fabric still worth it? | When runs must **survive failure**, wait for **exact-operation approvals**, use **isolated shell/fs work**, keep **portable policy**, or leave **recovery evidence** across targets. |
| When should you skip Fabric? | Databricks-only interactive agents where AppKit’s request/thread lifecycle is enough. |
Read the full decision guide: [Why Fabric on Databricks](/docs/databricks/why-fabric).
---
TechFabric Harness provides a first-party Databricks package, Databricks deployment targets, and Unity
Catalog-aware connectors. It is designed for agents that need governed access to enterprise data,
durable state, approval controls, lineage, and cost attribution without replacing Databricks
authorization.
Databricks remains the native data, AI, compute, identity, governance, and hosting platform. Fabric
is an optional durable control layer around those services. The current Databricks developer stack
also includes a beta [AppKit agent host](https://developers.databricks.com/docs/appkit/v0/plugins/agents)
with TypeScript definitions, tools, threads, cancellation, approvals, subagents, and
Responses-compatible routes. That is direct overlap, not a gap Fabric should pretend still exists.
Fabric differentiates when execution must survive a failed process or long approval wait, policy
must follow the workload across infrastructure, an agent needs isolated shell/filesystem work, or
operators need portable recovery evidence.
## Why add Fabric to a Databricks workload
| Requirement | Fabric advantage |
| --- | --- |
| Recover after process or worker failure | Stable submissions, leases, bounded retry classification, checkpoints, replay, terminal states, and optional deterministic Temporal workflows |
| Wait safely for a person or external system | Exact-operation and principal-bound approvals can persist beyond one request stream and remain correlated with the resumed work |
| Govern effects outside a model-tool list | Definition policy constrains tools, commands, filesystems, networks, credentials, connectors, sandboxes, budgets, and timeouts |
| Perform isolated work | Sessions can run tasks and shell commands, mount sources, create attachments and artifacts, and checkpoint or fork capable sandboxes |
| Cross the workspace boundary | The same finite or persistent agent contract can run on Databricks, Node, Docker, Temporal, Cloudflare, Azure-oriented targets, Kubernetes patterns, and custom backends |
| Operate with evidence | Identity, events, lineage, cost, artifacts, cleanup records, certification, and terminal outcomes remain correlated to the submission |
The current npm release is `@fabric-harness/databricks@7.1.1`. The exact `7.1.1` package and App artifacts
passed all 21 release-blocking Tier R checks in protected Azure `eastus2`, including managed Genie
MCP under OBO, governed SQL, live RAG, actual-cost reconciliation, and Lakebase-backed App restart
recovery. The certifying run executed in single-user workspace mode, so it makes no claim about
cross-user or cross-tenant isolation; seven configured Tier O checks passed and two failed. A
byte-identical same-commit Tier A run also passed all ten required governed authoring
lifecycles with an empty cleanup ledger. See
the [retained compatibility evidence](/docs/databricks/compatibility#current-711-protected-evidence)
for the run, commit, digests, and explicit limitations. Stable Databricks services use exact-pinned
official modular TypeScript SDKs; preview and workspace-dependent features remain labeled rather
than inferred from mock tests.
## Start at your level
| Route | Start here | What you keep |
| --- | --- | --- |
| **Build your first agent** | [Scaffold, mock, connect, and deploy](/docs/databricks/quickstart) | One generated project, deterministic safe-tool test, reviewed environment template, and focused `NEXT_STEPS.md` |
| **Choose a workload** | [RAG, analytics, operations, persistent Apps, or native access](/docs/databricks/workloads) | The same agent structure with composable recipes and explicit resource ids |
| **Production and advanced controls** | [Identity, policy, durability, certification, and native SDK composition](/docs/databricks/enterprise-adoption) | Required policy-bearing model tools, request-scoped OBO, Lakebase/Temporal options, and certification evidence |
The beginner path is ordinary advanced TypeScript—not a separate wizard runtime. Add typed native
clients through `bundle.sdk`, custom governed tools, durability, or additional certification without
regenerating the project. Advanced control means making identity, resources, effects, and opt-outs
more explicit; it does not disable the safe defaults.
```mermaid
flowchart LR
U[User or system] --> I[HTTP, schedule, channel, or CLI]
I --> H[TechFabric Harness agent]
subgraph Control[Fabric control plane]
H --> P[Policy and approvals]
H --> S[Durable session runtime]
H --> T[Governed tools]
end
subgraph Databricks[Databricks data and AI services]
T --> M[Unity AI Gateway]
T --> Q[SQL Warehouse]
T --> V[AI Search]
T --> G[Genie Agents]
T --> F[Feature Serving]
T --> L[Lakeflow Jobs]
S --> B[Lakebase]
T --> C[Unity Catalog]
end
C --> D[(Tables and volumes)]
H --> O[MLflow and OpenTelemetry]
classDef fabric fill:#dbeafe,stroke:#2563eb,color:#172554
classDef data fill:#dcfce7,stroke:#16a34a,color:#052e16
classDef control fill:#fef3c7,stroke:#d97706,color:#422006
class H fabric
class M,Q,V,G,F,L,B,C,D data
class P,S,T control
```
## What is first party
| Area | TechFabric Harness surface | Databricks service |
| --- | --- | --- |
| Model runtime | `databricksFoundationModelProvider()` | Unity AI Gateway model services and custom Model Serving endpoints |
| Data access | `databricksSqlReadTool()`, `databricksSqlTool()`, `databricksSqlSandbox()`, `sandbox: 'databricks'` | SELECT-only analytics reads, approval-bound arbitrary SQL, and SQL Warehouses |
| Governance | `withGovernance()`, policy and approval helpers | Unity Catalog remains the authorization authority |
| Agent interoperability | `responses` endpoint, `databricks-app`, `databricks-serving` | Responses API and MLflow ResponsesAgent |
| Agent discovery and grants | `bundle.agentServices` | Unity Catalog Agent Services |
| Retrieval | `databricksAiSearch()`, `databricksEmbeddings()` | Databricks AI Search and Model Serving |
| Analytics | `databricksGenieTool()`, `DatabricksGenieAgentModeClient`, `databricksAiQueryTool()` | Ordinary Genie conversations, explicit Beta Agent Mode streaming, and AI Functions |
| Features | `databricksFeatureLookupTool()` | Feature Serving |
| Orchestration | Jobs, notebooks, and `databricksLakeflowTools()` | Jobs and Lakeflow pipelines |
| Resource management | Stable opt-in Jobs, Lakeflow, AI Search, custom-model serving, managed UC, workspace, and secret-reference tools; beta Genie Agent lifecycle | Governed create/verify/mutate/delete lifecycles with capability-specific evidence |
| Durable state | `lakebaseClient()`, `databricksPersistence()` | Lakebase Autoscaling |
| Files | Volume source/writer and `UcVolumesAttachmentStore` | Unity Catalog Volumes and Files API |
| Operations | MLflow tracing, usage capture, consumption, actual-cost budgets | MLflow and system tables |
| Hosting | `databricks-app`, `databricks-serving` | Databricks Apps and Model Serving proxy |
| Native access | `databricksSdk()`, `databricksWorkspaceApi()` | Generated service clients plus credential-safe access to the remaining workspace APIs |
## What Databricks teams can deliver
| Starting workload | Recommended Fabric path | First proof to capture |
| --- | --- | --- |
| Governed analytics copilot | Genie plus `databricksSqlReadTool()` and `analyticsCopilotGovernance()` | OBO identity, allowed `SELECT`, blocked mutation, inspectable SQL, actual-cost attribution |
| RAG or knowledge assistant | `databricksAiSearch()` plus `createDatabricksRagChain()` and AI Gateway | Known answer with citations, insufficient-context refusal, MLflow evaluation record |
| Durable App | `databricks-app` plus `databricksPersistence()` on Lakebase | Create a session, restart the App, recover conversation offsets and submissions |
| Data engineering operator | Jobs/Lakeflow tools with approval policy | Exact-input approval, idempotent run receipt, status/output collection, cleanup |
| Interoperable agent service | Responses API plus MLflow ResponsesAgent or Agent Services registration | Authenticated `/api/responses`, trace correlation, discovery and grant evidence |
For a multi-team rollout, use the [enterprise adoption guide](/docs/databricks/enterprise-adoption).
It maps workloads to platform prerequisites, identity choices, operating ownership, rollout phases,
and production exit evidence.
## Choose the right starting point
- Read [why Fabric on Databricks](/docs/databricks/why-fabric) to understand what Fabric adds above
native SDK clients, which responsibilities remain with Databricks, and when not to add a harness.
- Use the [enterprise adoption guide](/docs/databricks/enterprise-adoption) to align application,
platform, security, and data-governance teams on a workload and production path.
- Read [Databricks development with TechFabric Harness](/docs/databricks/development) to understand the
local-to-App workflow, supported workload patterns, AI Gateway integration, and where Fabric adds value.
- Use [native access and platform coverage](/docs/databricks/native-access) for the generated
clients, unrestricted workspace API escape hatch, coverage dimensions, and known gaps.
- Build an interoperable App or serving agent with [Responses API and ResponsesAgent](/docs/databricks/responses-agent).
- Register an external agent for discovery and governance with [Unity Catalog Agent Services](/docs/databricks/agent-services).
- Use the [Databricks workload map](/docs/databricks/workloads) to choose a typed API, managed recipe,
or Databricks-native escape hatch.
- Use [Fabric Desktop for Databricks projects](/docs/databricks/fabric-desktop) for a guided recipe,
workspace discovery, mock/live run, build, preview, and Databricks App deployment workflow.
- Use the [quickstart](/docs/databricks/quickstart) to run a mocked agent and then connect a workspace.
- Read [architecture](/docs/databricks/architecture) to understand identity, control, and data flow.
- Use [integrations](/docs/databricks/integrations) as the feature and API map.
- Use [resource management](/docs/databricks/authoring) for approval-gated creation, protected live
lifecycle tests, cleanup, and production-readiness status.
- Choose an execution surface with [compute patterns](/docs/databricks/compute).
- Read [connectors and sandboxes](/docs/databricks/sandboxes-connectors) before choosing SQL execution, Volumes,
or a general-purpose code sandbox.
- Apply the [enterprise controls](/docs/databricks/enterprise) for production identity, approvals, audit, durability,
and cost limits.
- Use the evidence-driven [workspace compatibility matrix](/docs/databricks/compatibility) for cloud, region,
auth, API, App, and Lakebase requirements.
- Review [authoring certification](/docs/databricks/authoring-certification) for automated test
coverage, destructive protected-workspace lifecycles, cleanup behavior, and retained release evidence.
## Workspace validation
Before deploying, run the Databricks certification command against the target workspace. It verifies
OAuth scopes, Unity Catalog grants, API access, App resources, Unity AI Gateway model services, SQL
Warehouses, and Lakebase connectivity while producing secret-redacted evidence for the deployment
record. Follow [authoring certification](/docs/databricks/authoring-certification) for the exact
configuration, cleanup contract, and retained release evidence.
---
# Why TechFabric Harness on Databricks
Canonical: https://harness.techfabric.com/docs/databricks/why-fabric
What Fabric adds above Databricks native SDKs, governance, data, AI, and compute—and when to use the native platform directly.
## The 30 second answer
**Databricks decides who can touch data, models, and compute. Fabric decides how an autonomous agent
runs.**
| Layer | Owner | Examples |
| --- | --- | --- |
| Platform of record | **Databricks** | Unity Catalog, AI Gateway, Apps, Genie, Jobs, Lakeflow, Lakebase, MLflow |
| Durable agent control | **TechFabric Harness** | Recoverable runs, exact-operation approvals, portable policy, sandboxes, evidence, multi-target deploy |
Fabric does **not** replace Databricks. It is optional when AppKit’s request/thread lifecycle is
enough, and valuable when work must outlive a request, wait for a steward, run isolated shell or
filesystem tasks, or leave portable recovery evidence—on Databricks and beyond.
### Use AppKit (or native) when
- the app and users live entirely in Databricks;
- the agent is primarily interactive chat, Genie, analytics, files, or Model Serving;
- request-scoped approval and thread persistence meet the recovery bar.
### Use Fabric on Databricks when
- a run must survive process failure, a multi-day approval, or worker replacement;
- policy must cover tools **and** commands, filesystems, networks, connectors, or sandboxes;
- the same agent must run locally, on a Databricks App, and optionally on Temporal or another cloud;
- operators need retry classification, principal lineage, replay, cost attribution, and cleanup evidence.
### Use both when
Fabric owns a broader durable workflow and calls Agent Bricks, Genie, Model Serving, Jobs, or AI
Search as governed native capabilities. Application code can use `bundle.sdk`; only selected
operations are exposed to the model through `bundle.tools`.
---
TechFabric Harness is a **Databricks partner-built application and agent accelerator**. It composes and
operationalizes Databricks-native capabilities; it does not replace or replicate them.
TechFabric Harness and the current Databricks developer platform are **complementary below the agent
runtime and competitive at the agent-runtime boundary**. The
[Databricks Developer Hub](https://developers.databricks.com/docs/start-here) brings Apps, Lakebase,
Agent Bricks, the Data Lakehouse, AppKit, agent skills, and coding-agent templates into one
developer path.
Databricks remains the native data, AI, compute, identity, governance, and hosting platform. Fabric
uses those capabilities as the native foundation. Databricks now also offers a beta
[AppKit agent host](https://developers.databricks.com/docs/appkit/v0/plugins/agents) with TypeScript
and Markdown definitions, scoped tools, subagents, threads, cancellation, human approval, and
Responses-compatible routes. Those features overlap directly with part of TechFabric Harness.
Use AppKit directly when a Databricks-only interactive agent and its request/thread lifecycle meet
the requirement. Add Fabric when execution must survive process failure or a long approval wait,
policy must travel across infrastructure, work needs an isolated shell or filesystem, or operators
need a durable record of recovery and effects. The result remains a Databricks App, Job, Lakeflow
pipeline, MLflow experiment, Unity Catalog securable, or other native resource that developers can
inspect and operate with Databricks tooling.
## The boundary
```mermaid
flowchart LR
APP[Your TypeScript agent]
subgraph Fabric[TechFabric Harness durable control layer]
RUN[Finite runs, persistent agents, recovery]
GOV[Portable policy and durable approvals]
ISO[Tasks, sources, shell, sandboxes]
OBS[Replay, evidence, cost, artifacts]
end
subgraph Native[Databricks application and agent platform]
APPKIT[Apps and AppKit agents]
BRICKS[Agent Bricks]
SDK[Official modular TypeScript SDKs]
IAM[OAuth, App identity, OBO]
UC[Unity Catalog and ACLs]
DATA[SQL, AI Search, Genie]
WORK[Jobs, Lakeflow, Model Serving]
STATE[Lakebase, Volumes, MLflow]
end
APP --> RUN
APP --> APPKIT
RUN --> GOV
RUN --> ISO
GOV --> SDK
APPKIT --> SDK
APPKIT --> BRICKS
SDK --> IAM
IAM --> UC
SDK --> DATA
SDK --> WORK
RUN --> STATE
RUN --> OBS
RUN -. governed endpoint .-> BRICKS
classDef fabric fill:#f8e8d9,stroke:#b8541f,color:#0e1116
classDef native fill:#f2ede4,stroke:#7a8290,color:#0e1116
class RUN,GOV,ISO,OBS fabric
class APPKIT,BRICKS,SDK,IAM,UC,DATA,WORK,STATE native
```
Stable Databricks operations use the official modular TypeScript SDKs. Every generated SDK package
is exact-pinned to one reviewed version and exercised through serialization, authentication, and
live-workspace tests. A small private protocol adapter is restricted to endpoints the SDK does not
yet expose, such as selected Beta protocols; it is not a public general-purpose REST client.
Fabric does not replace Unity Catalog or weaken its decisions. Fabric can reject an operation before
it reaches Databricks, but Databricks remains the final authorization boundary. Agent Bricks can be
a managed agent endpoint that Fabric invokes, while Apps and Lakebase can host and persist the
Fabric runtime.
## Where the products overlap
| Capability | Databricks AppKit or Agent Bricks | TechFabric Harness | Relationship |
| --- | --- | --- | --- |
| TypeScript agent authoring | AppKit `createAgent()` and `runAgent()` | `defineAgent()` for finite agents and `createAgent()` for persistent agents | Direct overlap, with different lifecycle semantics |
| Threads and streaming | AppKit thread stores, chat streaming, invocation and Responses routes | Sessions, conversation streams, submissions, typed client, React hooks, and Responses interoperability | Direct overlap |
| Tool scoping | Plugin toolkits, allowlists, read-only SQL, MCP host policy | Typed tools plus definition policy across tools, commands, filesystems, networks, sandboxes, credentials, and connectors | Overlap; Fabric policy spans more execution surfaces |
| Human approval | Request-stream approval for mutating AppKit tools | Exact-operation and principal-bound approvals that can be persisted by the selected runtime/store | Overlap; durable waits are a Fabric advantage |
| Identity | App service principals and OBO user execution | Principal propagation through model, tools, approvals, lineage, submissions, and request-scoped bundles | Complementary and overlapping |
| Managed agents | Knowledge Assistants, Genie, Supervisor Agents, and custom agents | Invoke managed endpoints as tools or compose custom finite and persistent agents | Complementary or substitutable by workload |
| Model and data governance | Unity AI Gateway, Unity Catalog, MLflow, system tables | Defense-in-depth policy, budgets, correlation, redaction, and governed tool contracts | Complementary |
| Durable workflow recovery | Request and thread lifecycle; AppKit documents cancellation and pluggable thread storage | Retry classification, leases, submissions, checkpoints, replay, cancellation propagation, recovery, and deterministic Temporal workflows | Fabric advantage |
| Isolated agent work | Databricks application and compute services | Portable sandboxes, tasks, shell, mounted sources, artifacts, snapshots, forks, and cross-process sandbox references | Fabric advantage |
| Runtime portability | Databricks workspace and App hosting | Node, Docker, Temporal, Cloudflare, Azure-oriented targets, Kubernetes patterns, Databricks, and custom backends | Fabric advantage |
## Advantages to lead with
### Durable execution beyond a request
Fabric distinguishes finite, bounded agents from persistent, addressable agents. Submissions carry
stable identity and terminal state; retries are bounded and classified; leases prevent competing
workers; cancellation propagates; checkpoints and replay make recovery inspectable. The optional
Temporal runtime preserves this contract across worker restarts and long waits without making
Temporal a requirement for local authoring.
### Governance that remains attached to the effect
Definition policy is a security floor: invocation policy may narrow it but cannot silently widen it.
Approval grants bind the exact tool or command input and executing principal. Policy covers more than
model tools—it can constrain commands, filesystem reads and writes, network destinations,
credentials, connectors, sandboxes, budgets, and timeouts. Events, lineage, cost, artifacts, and
terminal state remain correlated with the governed operation.
### Portable isolated execution
Fabric sessions can mount sources, run tasks and shell commands, create artifacts, checkpoint or fork
capable sandboxes, and hand portable sandbox references across processes. Backend capability
discovery keeps this contract explicit across local, Docker, Kubernetes, Cloudflare, Daytona, E2B,
Modal, Databricks SQL, and other adapters.
### One contract beyond Databricks
The same agent, session, tool, policy, source, store, and event contracts can run on Databricks or
outside it. This matters when a workflow starts in a Databricks App but also coordinates GitHub,
Slack, cloud infrastructure, external databases, an isolated code environment, or a durable worker.
## Full Fabric feature surface on Databricks
| Developer need | Databricks provides | Fabric adds above it |
| --- | --- | --- |
| Model inference | Unity AI Gateway and Model Serving | One model-provider contract, OAuth refresh, bounded retries, request correlation, usage, and redacted failures |
| Governed data access | SQL Warehouses and Unity Catalog | Model-safe typed tools, structured resource extraction, catalog allowlists, approval routing, and denial lineage |
| RAG | AI Search, embeddings, AI Gateway, MLflow evaluation | A composed retrieve/generate/validate path, typed citations, durable sessions, evaluation export, and release gates |
| Genie | Genie conversations, Agents, ACLs, and Agent Mode | Conversation tools, lifecycle management, managed-only deletion, approval binding, durable ownership, and bounded Beta streaming |
| Data operations | Jobs, Lakeflow, notebooks, and serving endpoints | Compute-policy validation, idempotency and fingerprints, durable receipts, repair/status collection, approvals, and cleanup |
| Resource management | Native create/update/delete APIs | Opt-in model write tools that fail closed, bind approval to exact input and principal, and delete only Harness-managed resources |
| Identity | OAuth M2M, Databricks App identity, OBO | One verified principal propagated through model, tools, approvals, lineage, and request-scoped bundles without putting tokens in model context |
| Durable state | Lakebase and UC Volumes | Sessions, submissions, conversation offsets, attachments, leases, restart recovery, and cascade deletion |
| Operations | MLflow and system tables | Agent/run/tenant correlation, estimated and actual-cost controls, cleanup ledgers, and capability-specific certification evidence |
| Delivery | Databricks Apps and Model Serving | Local mock mode, typed agent definitions, reproducible build targets, Responses API interoperability, and the same runtime contract on other clouds |
The complete Fabric surface also includes typed inputs and results, roles, Markdown-first skills,
session memory, tasks and subagents, attachments, artifacts, filesystem sources, MCP, channels,
schedules, webhooks, queues, the Fiber console, React hooks, OpenTelemetry, evaluation helpers,
portable builds, certification, and supply-chain evidence. Not every individual feature is unique;
the differentiator is the durable, policy-constrained, portable contract that holds them together.
## Typed clients and model tools are deliberately different
Fabric exposes two seams:
- `bundle.sdk` contains principal-bound native Databricks clients for deterministic application
code. Calling one is no more autonomous than calling the Databricks SDK yourself.
- `bundle.tools` contains bounded `ToolDef` objects that a model may select. Write and execute tools
carry effect metadata, governance descriptors, approval requirements, redaction, and lineage.
This separation prevents “the application can create a Job” from silently becoming “the model can
create any Job.” Model-facing write surfaces are opt-in and fail initialization when their approval
or compute policy is incomplete.
```ts
import { databricks } from '@fabric-harness/databricks';
import { init } from '@fabric-harness/sdk';
const dbx = databricks({
host: process.env.DATABRICKS_HOST!,
principal: {
kind: 'service-principal',
host: process.env.DATABRICKS_HOST!,
clientId: process.env.DATABRICKS_CLIENT_ID!,
clientSecret: process.env.DATABRICKS_CLIENT_SECRET!,
},
model: 'system.ai.gpt-oss-20b',
warehouseId: process.env.DATABRICKS_WAREHOUSE_ID,
aiSearch: {
index: 'main.knowledge.docs_index',
textColumn: 'chunk',
idColumn: 'id',
},
governance: {
catalogs: ['main'],
stewardAudience: 'data-platform',
},
});
const fabric = await init({
modelProvider: dbx.modelProvider,
tools: dbx.tools,
policy: dbx.policy,
store: dbx.store,
});
```
This bundle uses one identity for AI Gateway, generated SDK clients, SQL, retrieval, and optional
Lakebase credential exchange. Add resource-management flags only when the agent truly needs them.
## Choose the right runtime
Use Databricks AppKit directly when:
- the application and its users live entirely in Databricks;
- the agent is primarily interactive chat, Genie, analytics, files, or Model Serving;
- request-scoped approval and thread persistence meet the recovery requirement; and
- a Databricks-native application and deployment lifecycle is preferred over runtime portability.
Use Fabric on Databricks when one or more of these are requirements:
- a run must survive process failure, an approval delay, a worker replacement, or a multi-step
workflow;
- policy must remain enforceable across Databricks and non-Databricks tools, commands, filesystems,
networks, connectors, or sandboxes;
- the workload needs tasks, shell execution, mounted sources, checkpoints, forks, attachments, or
artifacts around Databricks data and AI;
- the same finite or persistent agent needs local tests, a Databricks App, another cloud deployment
target, or an optional Temporal worker;
- operators need retry and terminal-state classification, tenant/principal lineage, replay, cost
attribution, cleanup evidence, and reproducible certification; or
- a workflow coordinates Databricks with channels, external systems, or isolated compute.
Use both when Fabric owns a broader durable workflow and calls Agent Bricks, Genie, Model Serving,
Jobs, Lakeflow, AI Search, or another App as a governed native capability. Application code can use
`bundle.sdk`, while only selected operations are exposed to the model through `bundle.tools`.
Use the native Databricks SDK without either agent runtime when the flow is short, deterministic
application code.
## What Fabric does not claim
- It is not a replacement for Databricks SDKs, CLI, Declarative Automation Bundles, Terraform,
Apps or AppKit, Agent Bricks, Unity Catalog, MLflow, Jobs, Lakeflow, AI Search, Genie, Lakebase,
Model Serving, or AI Gateway.
- It does not claim that ordinary TypeScript agent definitions, threads, scoped tools, streaming, or
request-time HITL are unique to Fabric.
- It does not make a preview API stable; preview capabilities remain labeled and require their own
workspace evidence.
- It does not infer AWS or GCP compatibility from an Azure certification run.
- It does not put workspace tokens, raw secret values, or user-supplied identity labels into model
context.
- It cannot make an unbounded autonomous agent safe. Production deployments still need least
privilege, explicit tools, budgets, approvals, evaluation, and workspace-specific tests.
## Continue
- [Build a Databricks agent](/docs/databricks/development)
- [Run RAG on Databricks](/docs/databricks/rag)
- [Manage Databricks resources safely](/docs/databricks/authoring)
- [Understand identity and control flow](/docs/databricks/architecture)
- [Review native SDK and workspace compatibility](/docs/databricks/compatibility)
- [Inspect live authoring certification evidence](/docs/databricks/authoring-certification)
---
# Databricks development with TechFabric Harness
Canonical: https://harness.techfabric.com/docs/databricks/development
Turn Databricks AI and data services into durable, governed TypeScript applications with a local-first workflow.
TechFabric Harness is the application runtime around Databricks AI and data services. It does not
replace Unity Catalog, Spark, Lakeflow, MLflow, Jobs, SQL Warehouses, or Model Serving. It gives
developers one TypeScript workflow for composing those services into agents that can run locally,
deploy to Databricks Apps, preserve identity, enforce policy, persist state, and produce operational
evidence.
## Where Fabric fits
```mermaid
flowchart LR
subgraph Author[Developer workflow]
CODE[TypeScript job or agent]
MOCK[Local mock and tests]
BUILD[Fabric build]
end
subgraph Runtime[Fabric application runtime]
APP[Databricks App]
LOOP[Session and model loop]
POLICY[Policy and approvals]
AUDIT[Lineage, cost, and telemetry]
end
subgraph Platform[Databricks services]
GATEWAY[Unity AI Gateway]
DATA[SQL and Unity Catalog]
RAG[AI Search and Genie]
COMPUTE[Jobs and Lakeflow]
STATE[Lakebase and Volumes]
MLFLOW[MLflow and system tables]
end
CODE --> MOCK
MOCK --> BUILD
BUILD --> APP
APP --> LOOP
LOOP --> POLICY
POLICY --> GATEWAY
POLICY --> DATA
POLICY --> RAG
POLICY --> COMPUTE
LOOP --> STATE
LOOP --> AUDIT
AUDIT --> MLFLOW
classDef author fill:#f4f4f5,stroke:#71717a,color:#18181b
classDef fabric fill:#dbeafe,stroke:#2563eb,color:#172554
classDef control fill:#fef3c7,stroke:#d97706,color:#422006
classDef dbx fill:#dcfce7,stroke:#16a34a,color:#052e16
class CODE,MOCK,BUILD author
class APP,LOOP,AUDIT fabric
class POLICY control
class GATEWAY,DATA,RAG,COMPUTE,STATE,MLFLOW dbx
```
Databricks remains authoritative for data permissions, compute, model access, and platform
operations. Fabric owns the agent lifecycle around those services: admission, sessions, tools,
approvals, durable state, retries, typed results, audit correlation, and deployment artifacts.
## Start locally, connect later
Create a Databricks-oriented project without requiring workspace credentials:
```sh
npx @fabric-harness/cli init \
--template databricks \
--dir analytics-agent
cd analytics-agent
npm install
```
The template creates a finite analytics job, role, skill, governed SQL policy, Databricks App
configuration, environment sample, and certification manifest.
```sh
fh agents
fh describe databricks-analyst
fh run databricks-analyst \
--question 'Describe main.sales.orders' \
--mock
```
Mock mode exercises discovery, input validation, tool assembly, the model loop, and HTTP routing.
It deliberately does not simulate Unity Catalog grants or claim that a workspace API succeeded.
When the local behavior is ready, configure a PAT for single-user development or OAuth M2M for a
production-like service principal:
```dotenv title=".env.local"
DATABRICKS_HOST=https://adb-1234567890123456.7.azuredatabricks.net
DATABRICKS_CLIENT_ID=00000000-0000-0000-0000-000000000000
DATABRICKS_CLIENT_SECRET=resolve-from-your-secret-manager
DATABRICKS_WAREHOUSE_ID=0123456789abcdef
DATABRICKS_MODEL=databricks-gpt-oss-20b
DATABRICKS_INFERENCE_MODE=auto
```
Run the same definition without `--mock` to exercise the real workspace. See
[Local naming and authentication](/docs/databricks/local-naming-auth) for PAT, OAuth M2M, App
identity, OBO, tenant, and deployment-profile behavior.
### Model selection and precedence
The scaffolded default, `databricks-gpt-oss-20b`, is a bare foundation-model serving endpoint
(pay-per-token) that works on every workspace tier, including free tiers without Unity AI Gateway.
A `system.ai.*` model service routes through Unity AI Gateway instead and 404s on workspaces where
the gateway is not enabled.
When several sources name a model, the first one wins:
1. `fh run --model `
2. `FABRIC_MODEL`
3. `run.model` in `.fabricharness/config.ts`
4. the job file's `model:` field
5. `agent.model` in `.fabricharness/config.ts`
6. `DATABRICKS_MODEL` — the bundle model provider's default, used only when nothing above names a
model
`DATABRICKS_MODEL` therefore cannot rescue a bad value set higher in the chain; fix the job file or
config instead. A `databricks/`-prefixed reference (for example
`databricks/databricks-gpt-oss-20b`) is accepted anywhere a model is named — the Databricks
provider strips its routing prefix at the wire boundary, so serving endpoints always receive the
bare endpoint name and a prefixed `system.ai.*` reference still routes through AI Gateway.
If required environment variables are missing, the first live run fails once with the full list of
missing prerequisites (host, warehouse, Genie space, steward audience, cost enforcement) instead of
failing fast one variable at a time.
## Use AI Gateway without custom HTTP plumbing
Use a discovered `system.ai.*` service as the model:
```ts title=".fabricharness/jobs/databricks-analyst.ts"
import { defineDatabricksAgent } from '@fabric-harness/databricks';
import { schema } from '@fabric-harness/sdk';
import policy from '../policies/databricks.js';
export default defineDatabricksAgent({
name: 'databricks-analyst',
description: 'Answer governed questions using workspace data.',
input: schema.object({ question: schema.string() }),
output: schema.string(),
model: 'system.ai.gpt-oss-20b',
tools: ['sql-read', 'tables', 'table-info'],
triggers: { manual: true, webhook: true },
sandbox: 'empty',
policy,
});
```
Fabric automatically routes `system.ai.*` through the workspace Unity AI Gateway path. Custom
endpoint names route through Model Serving. The Databricks provider adds:
- OAuth token acquisition, caching, and early refresh;
- request tags for submission, attempt, tenant, and agent identifiers;
- retry-safe requests and redacted provider errors;
- model usage and cost correlation;
- a common provider contract for local tests and deployed runtimes.
Discover enabled services in the target workspace instead of assuming a model is available. The
[integration map](/docs/databricks/integrations#inference-names-and-urls) documents automatic routing
and explicit overrides.
## Compose the governed Databricks stack
Use `databricks()` when an application needs several Databricks services under one identity:
```ts
import { databricks } from '@fabric-harness/databricks';
import { init } from '@fabric-harness/sdk';
const dbx = databricks({
host: process.env.DATABRICKS_HOST!,
principal: {
kind: 'service-principal',
host: process.env.DATABRICKS_HOST!,
clientId: process.env.DATABRICKS_CLIENT_ID!,
clientSecret: process.env.DATABRICKS_CLIENT_SECRET!,
},
model: 'databricks-gpt-oss-20b',
warehouseId: process.env.DATABRICKS_WAREHOUSE_ID,
aiSearch: {
index: 'main.knowledge.docs_index',
textColumn: 'chunk',
idColumn: 'id',
strategy: 'hybrid',
},
genie: {
conversations: { agentId: process.env.DATABRICKS_GENIE_SPACE_ID! },
},
lakeflow: true, // list/status only; add { runPolicy } for start/stop
consumption: true,
governance: {
catalogs: ['main'],
stewardAudience: 'data-steward',
},
});
const runtime = await init({
modelProvider: dbx.modelProvider,
tools: dbx.tools,
policy: dbx.policy,
store: dbx.store,
});
```
One principal is threaded through model calls, generated Databricks SDK clients, SQL, and optional Lakebase credential
exchange. Fabric policy can narrow access or require approval, while Unity Catalog and Databricks
resource ACLs make the final authorization decision.
## Add only the workloads you need
Managed recipes add project-local wiring, compatible dependencies, environment stubs, and
verification commands:
```sh
fh add databricks core
fh add databricks sql
fh add databricks ai-search
fh add lakebase
fh add lakeflow
fh add jobs
```
Recipes are dependency-aware and refuse incompatible ranges rather than silently replacing them.
Use `fh add --dry-run` to inspect changes and `fh update` to update managed recipe files.
## Workload patterns
| Workload | Databricks services | What Fabric adds |
| --- | --- | --- |
| Governed lakehouse analyst | AI Gateway, SQL Warehouse, Unity Catalog | Typed input/output, SQL policy, approvals, tenant identity, audit lineage |
| RAG application | AI Search, AI Gateway, MLflow 3 | Retrieval orchestration, validated citations, evaluation export, release quality gates |
| Persistent copilot | Databricks Apps, Lakebase, UC Volumes | Addressable instances, conversation streams, durable submissions, attachments, deletion |
| Data operations agent | Jobs, notebooks, Lakeflow | Idempotent admission, status/output collection, approval gates, durable receipts |
| BI assistant | Genie, SQL Warehouse, Unity Catalog | Governed tool composition, session context, principal and tenant propagation |
| Feature-aware agent | Feature Serving, Model Serving | Low-latency feature lookup as a governed tool with model usage correlation |
| Model Serving integration | AI Gateway or custom endpoints | One provider API, OAuth refresh, request tags, retries, usage and cost attribution |
| Agent interoperability | Responses API, MLflow ResponsesAgent, Databricks Apps | Durable `/responses` endpoint plus a registered Python proxy in Model Serving |
| External agent governance | Unity Catalog Agent Services and HTTP connections | Discoverable agent registration, standard UC grants, lifecycle certification, and cleanup |
## Databricks-native RAG quality
Fabric's deterministic RAG chain follows a preprocess, retrieve, augment, generate, and validate
flow. AI Search performs retrieval, AI Gateway performs generation, and MLflow 3 performs
managed evaluation.
```mermaid
flowchart LR
QUESTION[Golden-set question] --> RETRIEVE[AI Search retrieval]
RETRIEVE --> GENERATE[AI Gateway generation]
GENERATE --> TRACE[MLflow trace]
TRACE --> JUDGES[Managed judges]
JUDGES --> REL[Answer relevance]
JUDGES --> RREL[Retrieval relevance]
JUDGES --> GROUND[Groundedness]
JUDGES --> SUFF[Sufficiency]
JUDGES --> CORRECT[Correctness]
REL --> GATE{All meet threshold?}
RREL --> GATE
GROUND --> GATE
SUFF --> GATE
CORRECT --> GATE
GATE -->|Yes| PASS[Release evidence]
GATE -->|No| BLOCK[Block release]
classDef source fill:#f4f4f5,stroke:#71717a,color:#18181b
classDef fabric fill:#dbeafe,stroke:#2563eb,color:#172554
classDef dbx fill:#dcfce7,stroke:#16a34a,color:#052e16
classDef decision fill:#fef3c7,stroke:#d97706,color:#422006
classDef deny fill:#fee2e2,stroke:#dc2626,color:#450a0a
class QUESTION source
class RETRIEVE,GENERATE,TRACE,JUDGES dbx
class REL,RREL,GROUND,SUFF,CORRECT,PASS fabric
class GATE decision
class BLOCK deny
```
The certification fixture uses managed relevance, retrieval relevance, groundedness, sufficiency,
and correctness judges with a configurable threshold. A project should replace the small
certification fixture with its own domain questions, expected facts, retrieval expectations,
insufficient-context cases, and adversarial inputs. See [RAG on Databricks](/docs/databricks/rag).
## Durable Apps instead of stateless demos
`databricks-app` bundles the Node server and agent definitions for Databricks Apps. Optional
Lakebase persistence stores sessions, submissions, and conversation streams. Unity Catalog Volumes
store governed attachments.
```sh
fh build --target databricks-app
fh deploy --preview --target databricks-app --profile analytics-dev
fh deploy --target databricks-app --profile analytics-dev
```
The generated artifact contains `app.yaml`, `databricks.yml`, the self-contained server bundle,
roles, skills, definitions, and build manifest. Runtime credentials come from App identity and
resource bindings, not the developer's deployment profile.
For configuration-preserving recovery, redeploy the generated bundle with the same variables. A
bare App start can omit values injected during deployment. The
[Databricks App tutorial](/docs/deployment/databricks-app) documents the certified deployment and
recovery procedure.
## Governance and operations are runtime behavior
Fabric controls are evaluated around every agent operation rather than described only in a prompt:
- capability and catalog policies constrain available actions;
- sensitive SQL, pipeline, and mutation tools can require durable approval;
- Fabric principals and tenants propagate into submissions and request tags;
- Unity Catalog remains authoritative for tables, schemas, volumes, rows, and columns;
- Lakebase telemetry joins actors, submissions, governed objects, outcomes, and estimated cost;
- System Tables reconcile delayed actual usage with tenant and agent budgets;
- cascade deletion removes sessions, submissions, streams, and attachments;
- MLflow and OpenTelemetry expose traces and operational evidence.
This is most useful when an agent moves from a notebook experiment to a shared application with
multiple users, governed data, long-running work, or production operating requirements.
## When to use TechFabric Harness
Use Fabric when the application needs several of these together:
- local TypeScript development and Databricks App deployment;
- model calls plus SQL, retrieval, Jobs, Lakeflow, Genie, or Feature Serving;
- persistent agents or durable workflow state;
- user, service-principal, tenant, and OBO identity propagation;
- approval, policy, audit, deletion, or cost enforcement;
- repeatable workspace certification and deployment evidence.
A direct Databricks SDK or notebook is usually simpler for a one-off query, a standalone Spark
transformation, or a single model request with no agent lifecycle. Fabric is valuable when those
calls need to become a governed application.
## Production validation
Fabric ships local contracts and a protected workspace certification runner. Before approving a
deployment, validate the actual cloud, region, workspace, identity, resources, data, and workload:
1. Verify OAuth M2M or App identity and every required resource permission.
2. Prove one allowed and one deliberately denied Unity Catalog operation.
3. Exercise AI Gateway, SQL, retrieval, Jobs, Lakeflow, and other enabled services.
4. Deploy the App, persist work, redeploy, and verify recovery and cascade deletion.
5. Run the project's MLflow evaluation dataset and enforce quality thresholds.
6. Verify lineage and System Tables cost reconciliation for real tenant tags.
7. Test concurrency, rate limits, long-running work, and expected failure modes.
8. Exercise OBO login, expiry, refresh, and user-specific grants when OBO is enabled.
9. Retain redacted certification, compatibility, recovery, and conformance evidence.
Certification records are environment-specific. Do not infer that a passing Azure workspace also
certifies an untested AWS/GCP workspace, region, preview feature, or production corpus. Use the
[workspace compatibility matrix](/docs/databricks/compatibility) and
[authoring certification guide](/docs/databricks/authoring-certification) for the complete evidence path.
## Next steps
- [Create the reference project](/docs/databricks/quickstart)
- [Understand local names and identities](/docs/databricks/local-naming-auth)
- [Choose Databricks integrations](/docs/databricks/integrations)
- [Build and evaluate RAG](/docs/databricks/rag)
- [Deploy a Databricks App](/docs/deployment/databricks-app)
- [Apply enterprise controls](/docs/databricks/enterprise)
---
# Responses API and ResponsesAgent
Canonical: https://harness.techfabric.com/docs/databricks/responses-agent
Expose a durable TechFabric Harness agent through the OpenAI Responses API and MLflow ResponsesAgent on Databricks Apps and Model Serving.
TechFabric Harness exposes persistent agents through `POST /responses`, the interface Databricks uses for
new agent Apps. The same contract is packaged as an MLflow `ResponsesAgent` for Model Serving,
AI Playground, evaluation, monitoring, and clients that use `DatabricksOpenAI.responses`.
```mermaid
flowchart LR
CLIENT[DatabricksOpenAI or HTTP client] -->|Direct POST /responses| APP[Databricks App]
CLIENT -->|Serving invocation| SERVING[Model Serving and AI Playground]
SERVING --> PROXY[MLflow ResponsesAgent proxy]
PROXY -->|POST /responses| APP
APP --> AUTH[OAuth or App OBO identity]
AUTH --> API[Fabric Responses adapter]
API --> SUB[Durable submission]
SUB --> AGENT[Persistent Fabric agent]
AGENT --> DBX[AI Gateway, SQL, AI Search, Genie, MCP]
AGENT --> EVENTS[Token and tool events]
EVENTS -->|SSE| CLIENT
SUB --> STORE[Lakebase session and submission stores]
SUB --> TRACE[MLflow trace]
```
## Build your first Responses agent
Create `.fabricharness/agents/analyst.ts`:
```ts
import { createAgent } from '@fabric-harness/sdk';
export default createAgent(({ id }) => ({
name: 'analyst',
model: 'databricks/system.ai.gpt-oss-20b',
sandbox: 'virtual',
instructions: `You are a governed data analyst for instance ${id}.`,
triggers: { webhook: true },
}));
```
Enable the endpoint in `.fabricharness/config.ts`:
```ts
import type { FabricHarnessConfig } from '@fabric-harness/node';
export default {
run: { idPrefix: 'analytics-agent' },
responses: {
agent: 'analyst',
defaultSession: 'default',
customOutputKeys: ['client_type'],
},
} satisfies FabricHarnessConfig;
```
`custom_inputs` are not added to model context. `customOutputKeys` is an explicit allowlist for
non-secret values that may be returned under `custom_outputs`; all other custom fields stay outside
the prompt and response.
By default, each authenticated principal gets a separate persistent agent instance. When you set
`responses.instanceId`, it acts as a namespace prefix and Harness still appends the principal id.
This prevents two App users who choose the same conversation id from sharing session history.
Start locally:
```bash
pnpm add @fabric-harness/sdk @fabric-harness/node @fabric-harness/databricks
pnpm exec fh dev
```
Call it without streaming:
```bash
curl http://localhost:3000/responses \
-H 'content-type: application/json' \
-H 'idempotency-key: analyst-demo-1' \
-d '{
"input": [{"role":"user","content":"Summarize yesterday orders"}],
"context": {"conversation_id":"orders-42"},
"custom_inputs": {"client_type":"local-demo"}
}'
```
Set `"stream": true` to receive `response.output_text.delta` events followed by one
`response.output_item.done` event. All events for an answer use the same item id. Harness normalizes
both ordinary string deltas and the typed reasoning/output blocks returned by Unity AI Gateway.
Reasoning blocks remain internal; typed `text` and `output_text` blocks become incremental response
deltas and the completed output item.
## Continue a conversation
Use either durable conversation mechanism:
- Send the same `context.conversation_id` for each turn. The authenticated principal selects the
persistent agent instance, so one caller cannot select another caller's instance.
- Send the prior response `id` as `previous_response_id`. Fabric resolves the original durable
submission and reuses its instance and session after checking tenant and principal ownership.
An `Idempotency-Key` is scoped to tenant, principal, agent, and key. Retrying the same request returns
the same response id. Reusing the key with a different payload is rejected by the submission store.
Treat aggregate and streaming calls as separate invocation modes: retry either mode with its original
key, but use a new key when changing `stream` because that changes the admitted request payload.
## Authentication on Databricks
Locally, use the server bearer token or your configured OIDC authenticator. In a Databricks App:
- App authorization runs as the App service principal.
- User authorization validates the forwarded token against the workspace current-user API before
accepting it. Validation is cached by token digest and bounded by token expiry.
- Each validated App user receives a stable, opaque Fabric tenant by default. Session listing,
direct reads, approvals, artifacts, abort, and deletion remain inside that tenant; legacy
unscoped sessions are not visible to a tenant-bound user.
- App users receive only invoke and own-session permissions. The generated authenticator does not
grant `admin:read`, `build:read`, `session:replay`, or wildcard access.
- Clients must call Apps with Databricks OAuth. Personal access tokens are not supported for App URLs.
The generated Declarative Automation Bundle requests the `sql`, `genie`, and `model-serving` user
API scopes. Databricks combines those scopes with each user's existing workspace and Unity Catalog
permissions. Users must consent after scopes change; restart an older App before adding scopes if
the workspace requires it.
The App build automatically exposes the first persistent agent. Set `responses.agent` when the
workspace contains multiple persistent agents.
```bash
fh build --target databricks-app
fh deploy --target databricks-app
TOKEN="$(databricks auth token --host "$DATABRICKS_HOST" | jq -r .access_token)"
curl "$DATABRICKS_APP_URL/responses" \
-H "Authorization: Bearer $TOKEN" \
-H 'content-type: application/json' \
-H 'x-mlflow-return-trace-id: true' \
-d '{"input":[{"role":"user","content":"What changed in revenue?"}],"stream":true}'
```
When `DATABRICKS_MLFLOW_EXPERIMENT_ID` configures the App's MLflow exporter and
`x-mlflow-return-trace-id: true` is present, a non-streaming response includes
`metadata.trace_id`; a stream emits a separate trace-id event before `[DONE]`. The identifier matches
the deterministic MLflow trace produced for the durable submission.
`GET /ready` reports `mlflowTracing: configured` without returning the experiment identifier. The
bundle attaches the experiment with `CAN_EDIT`, and the final `app.yaml` resolves that managed
resource through `valueFrom`. A Databricks App build is not ready when that value was not injected.
## Model Serving interoperability
Build the wrapper after the App is reachable:
```bash
pnpm add @fabric-harness/databricks
fh build --target databricks-serving
cd .fabricharness/build/databricks-serving
python serving/contract_test.py
python3 -m pip install -r serving/requirements.txt
fh deploy --target databricks-serving
```
The CLI loads the native Databricks SDK only for the Serving deployment operation. The integration
package is therefore an optional CLI peer and an explicit project dependency; a missing package
fails before model registration with an install instruction. Databricks init templates and managed
recipes add it automatically.
The artifact subclasses `mlflow.pyfunc.ResponsesAgent`, declares the MLflow task
`agent/v1/responses`, forwards non-streaming and streaming requests to the App, preserves
`custom_inputs`, and authenticates to a Databricks App with short-lived OAuth M2M credentials. Use
an App-dedicated service principal with `CAN USE`; do not put a personal access token or long-lived
workspace token in Model Serving environment variables.
The generated requirements accept MLflow `>=3.10`; the protected Azure release gate currently
validates model logging and deployment with MLflow `3.14.0`. Optional request fields are read with
attribute-safe fallbacks because MLflow's own model-validation input may omit
`databricks_options`. Run `serving/contract_test.py` before logging the model and retain the live
model-registration result as the compatibility record for your workspace.
After model registration, `fh deploy` uses the typed TypeScript serving-admin client to create or
update the endpoint and enable an AI Gateway inference table. Payload logging is written under the
configured serving catalog and schema with a table prefix derived from the endpoint name. The CLI
serializes these mutations: it waits for endpoint readiness before replacing an existing model,
waits again before the AI Gateway update, and waits once more before reporting success. This makes
release retries safe when an earlier endpoint or gateway configuration is still converging.
Databricks agent endpoints currently support inference tables but not Gateway rate limits; the
App's authenticated HTTP limiter remains the enforced request boundary.
## Request reference
| Field | Purpose | Harness behavior |
| --- | --- | --- |
| `input` | String or Responses message items | Converted to one bounded agent turn; required |
| `stream` | Enable SSE | Emits text deltas, a final output item, optional trace id, then `[DONE]` |
| `context.conversation_id` | Durable conversation key | Selects a named session inside the authenticated instance |
| `context.user_id` | Client correlation | Accepted as metadata; never trusted for authorization |
| `previous_response_id` | Continue a prior response | Resolves a tenant- and principal-owned durable submission |
| `custom_inputs` | Application-specific metadata | Kept out of model context unless application code explicitly uses it |
| `metadata` | Client metadata | Validated as JSON and kept outside the default prompt |
Production errors do not return upstream bodies or secrets. Aborted responses return `409` in
non-streaming mode; streams emit `response.failed` with a bounded Databricks error object.
## Release verification
Every change runs the Node request/parser and durable endpoint tests plus the generated Python
contract. The protected Databricks workflow deploys the App and MLflow wrapper, then verifies OAuth,
non-streaming output, streaming deltas, stable item ids, trace correlation, restart recovery, and
secret redaction. The public capability registry links the retained `4.4.1` Azure evidence and marks
this surface `protected-live`; its product status remains Beta because the upstream API is Beta and it
is still a Tier O, Serving-profile check rather than a core Tier R release requirement. See
[compatibility and certification](/docs/databricks/compatibility) for the exact artifact identity.
References: [Databricks agent authoring](https://docs.databricks.com/aws/en/agents/agent-framework/author-agent),
[query deployed agents](https://docs.databricks.com/aws/en/agents/agent-framework/query-agent), and
[MLflow ResponsesAgent](https://mlflow.org/docs/latest/genai/serving/responses-agent/).
---
# Unity Catalog Agent Services
Canonical: https://harness.techfabric.com/docs/databricks/agent-services
Register an external TechFabric Harness agent in Unity Catalog, make it discoverable, manage grants, certify the lifecycle, and clean up safely.
Unity Catalog Agent Services gives an externally hosted TechFabric Harness agent a governed identity in
Databricks. The registration appears beside tables, models, and functions in Catalog Explorer. Teams
can discover it with `READ_METADATA`, and administrators can control access with Unity Catalog grants.
Agent Services is a Databricks Beta. During the current Beta it supports registration, discovery,
metadata updates, permissions, and deletion. Databricks does not yet route runtime requests through
the registered service. Use the Harness `/responses` endpoint to invoke the agent directly; use the
Agent Service as its Unity Catalog catalog entry and permission boundary.
This page also covers the separate **Supervisor Agents** and **managed memory** Betas. Supervisor
Agents are Databricks-owned agent runtimes with queryable serving endpoints; Unity Catalog Agent
Services registers an externally hosted agent. Harness composes either surface without presenting
the native Databricks loop as a Harness loop.
## Compose a native Supervisor Agent
Supervisor Agent discovery and invocation use the official generated Databricks SDK. The lifecycle
is deliberately hidden until the caller acknowledges the upstream Beta:
```ts
import {
databricks,
databricksManagedAgentTool,
} from '@fabric-harness/databricks';
const workspace = databricks({
host: process.env.DATABRICKS_HOST!,
principal: {
kind: 'service-principal',
host: process.env.DATABRICKS_HOST!,
clientId: process.env.DATABRICKS_CLIENT_ID!,
clientSecret: process.env.DATABRICKS_CLIENT_SECRET!,
},
supervisorAgents: { acknowledgeBeta: true },
});
const agents = await workspace.supervisorAgents!.list({ maxPages: 10 });
const tools = await workspace.supervisorAgents!.listTools(agents[0]!.supervisorAgentId!);
const answer = await workspace.supervisorAgents!.invoke(agents[0]!.supervisorAgentId!, {
input: 'Summarize the renewal risk.',
});
const assistant = databricksManagedAgentTool(workspace.agentEndpoints.client, {
endpointName: agents[0]!.endpointName!,
kind: 'supervisor-agent',
resourceId: agents[0]!.name,
maxOutputBytes: 512_000,
});
```
`list()` and `listTools()` bound pagination and reject repeated page tokens. `invoke()` resolves the
current endpoint with the official SDK, propagates cancellation, disables ambiguous invocation
retries, and caps the native response admitted into Harness model context. Knowledge Assistants use
the same fixed-resource tool projection with `kind: 'knowledge-assistant'`. The official Beta SDK is
available as `workspace.supervisorAgents.sdk` for explicit create, update, tool, example, and delete
operations. Those mutations have contract coverage only; they are not yet protected-live certified.
## Use managed memory explicitly
Managed memory remains distinct from Harness session and submission persistence. An application
must supply the tenant/user scope at a trusted boundary for every entry operation; the scope is not
accepted from the model:
```ts
const workspace = databricks({
host,
principal,
managedMemory: { acknowledgeBeta: true },
});
const memory = workspace.managedMemory!;
await memory.createEntry(
'main.agents.revenue_memory',
`tenant/${authenticatedTenant}/user/${authenticatedUser}`,
{
path: '/memories/preferences/reporting',
contents: 'Prefer quarter-over-quarter comparisons.',
},
);
const matches = await memory.search('main.agents.revenue_memory', {
scope: `tenant/${authenticatedTenant}/user/${authenticatedUser}`,
query: 'reporting preference',
topK: 5,
});
```
Store and entry create/read/update/delete plus bounded search are supported behind explicit Beta
acknowledgement. Paths are restricted to `/memories/`, parent traversal is rejected, search limits
are bounded, and deletion semantics remain an application-owned retention decision. This adapter
uses a reviewed narrow protocol allowlist because Databricks does not currently ship the managed
memory API in its generated TypeScript SDK. It has deterministic contract tests, not retained live
certification.
```mermaid
flowchart LR
DEV[Developer or CI] --> SDK[databricks bundle.agentServices]
SDK --> UC[Unity Catalog Agent Service]
UC --> META[Catalog Explorer discovery]
UC --> GRANTS[READ_METADATA and EXECUTE grants]
UC --> CONN[Unity Catalog HTTP connection]
CONN -. records host and credentials .-> APP[TechFabric Harness App]
USER[Application caller] -->|POST /responses| APP
APP --> SESSION[Durable Harness session]
SESSION --> DBX[Databricks data and AI services]
classDef fabric fill:#dbeafe,stroke:#2563eb,color:#172554
classDef governed fill:#dcfce7,stroke:#16a34a,color:#052e16
classDef identity fill:#fef3c7,stroke:#d97706,color:#422006
class SDK,APP,SESSION fabric
class UC,META,GRANTS,DBX governed
class CONN identity
```
## The four objects involved
| Object | Purpose | Credential owner |
| --- | --- | --- |
| Harness App or external server | Runs the agent and exposes `POST /responses` | TechFabric Harness deployment |
| Unity Catalog HTTP connection | Stores the agent host and external-service credential | Unity Catalog |
| Unity Catalog Agent Service | Makes the agent discoverable and permissioned | Unity Catalog |
| Workspace OAuth/PAT identity | Creates and manages the registration | Developer, CI service principal, or App service principal |
The workspace credential used by `@fabric-harness/databricks` is not sent to the external agent. The
HTTP connection stores the credential used to reach the agent, such as `FABRIC_HARNESS_API_TOKEN`.
Keeping these identities separate prevents a workspace administrator token from becoming an agent
runtime secret.
## Before you begin
1. Ask an account administrator to enable the **Agent Services** preview for the account.
2. Deploy a persistent Harness agent to a reachable HTTPS endpoint. A Databricks App built with
`fh build --target databricks-app` exposes it at `/responses`.
3. Create or choose a Unity Catalog schema for agent registrations.
4. Create a Unity Catalog HTTP connection whose host points at the deployed agent.
5. Give the automation principal the required grants.
The registration principal needs:
- `USE CATALOG` on the parent catalog;
- `USE SCHEMA` and `CREATE SERVICE` on the parent schema;
- `USE CONNECTION` on the HTTP connection;
- `MANAGE_ACCESS_CONTROL` on the Agent Service before it manages grants.
Agent consumers typically receive `READ_METADATA` to discover the service and `EXECUTE` to express
permission to use it. `EXECUTE` is a governance grant in this Beta; it does not create an invocation
route by itself.
## Install and scaffold
Install the Databricks package directly:
```bash
pnpm add @fabric-harness/databricks @fabric-harness/sdk
```
Or let the CLI install the package, environment template, implementation, and contract test:
```bash
fh add databricks agent-services --dry-run
fh add databricks agent-services
pnpm exec vitest run test/databricks/agent-services.test.ts
```
The managed recipe writes `.fabricharness/databricks/agent-service.ts`. Existing files are preserved
unless `--force` is supplied, and an incompatible installed package range stops the operation before
files are written.
## Deploy the Harness agent
A persistent agent provides the durable conversation behind `/responses`:
```ts
// .fabricharness/agents/support.ts
import { createAgent } from '@fabric-harness/sdk';
export default createAgent(({ id }) => ({
name: 'support',
description: 'Answer governed support questions.',
model: `databricks/${process.env.DATABRICKS_MODEL ?? 'system.ai.gpt-oss-20b'}`,
instructions: `You are the support agent for conversation ${id}. Answer using approved support sources only.`,
triggers: { webhook: true, manual: true },
}));
```
Build and deploy it as a Databricks App:
```bash
fh doctor --target databricks-app
fh build --target databricks-app
fh deploy --target databricks-app
```
Confirm the runtime boundary before registering it:
```bash
curl --fail --request POST "$FABRIC_AGENT_URL/responses" \
--header "Authorization: Bearer $FABRIC_HARNESS_API_TOKEN" \
--header 'Content-Type: application/json' \
--data '{
"model": "support",
"input": "Where is my order?",
"stream": false
}'
```
See [Responses API and ResponsesAgent](/docs/databricks/responses-agent) for streaming, continuation,
custom inputs, trace IDs, tenancy, and durable deployment behavior.
## Create the HTTP connection
The beginner path is **Catalog Explorer → Add → Create a connection**:
1. Select connection type **HTTP**.
2. Enter the deployed agent HTTPS host.
3. Keep the connection base path at `/`; the Agent Service supplies `/responses`.
4. Select bearer-token authentication and reference a secret containing
`FABRIC_HARNESS_API_TOKEN`.
5. Record the connection resource name returned by Databricks.
For SQL-driven infrastructure, store the token in Databricks Secrets and reference it instead of
placing a credential in source code or command history:
```sql
CREATE CONNECTION fabric_support_connection TYPE HTTP
OPTIONS (
host 'https://fabric-support-app.example.com',
port '443',
base_path '/',
bearer_token secret ('fabric', 'harness_api_token')
)
COMMENT 'Credentialed connection to the Fabric support agent';
```
Use the resource name returned by the connection API. Current metastores can return a single-part
name such as `connections/fabric_support_connection`; the Agent Services API also documents a scoped
form such as `connections/main.agents.fabric_support_connection`. The Harness client accepts either
shape, with or without the `connections/` prefix.
For production private networking, use a Databricks-supported Private Link path where available. If
the service uses IP allowlisting, allow the serverless outbound addresses used by Unity Catalog HTTP
connections. The connection secures credentials but does not replace network policy.
## Register the agent
Use OAuth M2M for release automation and a short-lived user OAuth profile for local administration.
PATs work for local testing but should not be embedded in application configuration.
```ts
import {
databricks,
databricksPrincipalFromEnv,
} from '@fabric-harness/databricks';
const workspace = databricks({
host: process.env.DATABRICKS_HOST!,
principal: databricksPrincipalFromEnv(process.env),
// Agent Services is a Databricks Beta; acknowledgement is intentionally explicit.
agentServices: { acknowledgeBeta: true },
});
const services = workspace.agentServices!;
const registered = await services.create({
catalog: 'main',
schema: 'agents',
id: 'support_agent',
connection: 'fabric_support_connection',
basePath: '/responses',
comment: 'Governed support agent for the customer operations team',
systemPrompt: 'Answer customer support questions using approved sources.',
});
console.log(registered.name);
```
`create()` does not silently retry. The preview API does not provide an idempotency token, and a
network retry after an ambiguous response could conceal whether a registry entry was created. For
reconcilers, call `get()` first, create on a confirmed 404, and update known fields when the entry
already exists.
## Discover and update registrations
```ts
const service = await services.get('main.agents.support_agent');
const page = await services.list({
catalog: 'main',
schema: 'agents',
});
await services.update('main.agents.support_agent', {
comment: 'Support agent owned by Customer Operations',
systemPrompt: 'Answer concisely from approved support sources.',
basePath: '/responses',
});
```
Agent Services is a Beta endpoint that is not yet present in the modular Databricks SDK. The
explicit preview client is intentionally limited to such gaps; stable services use `databricksSdk()`.
Updates use an explicit Databricks `update_mask`; omitted fields remain unchanged. The current API
allows updates to `comment`, `config.system_prompt`, and `config.base_path`. The connection is fixed
at creation. To move a service to another connection, create a replacement registration and migrate
grants before deleting the old one.
List the entire metastore only for administrative inventory jobs:
```ts
let pageToken: string | undefined;
do {
const page = await services.list({ ...(pageToken ? { pageToken } : {}) });
for (const service of page.agent_services ?? []) console.log(service.name);
pageToken = page.next_page_token;
} while (pageToken);
```
Application discovery should remain schema-scoped so it does not depend on broad metastore metadata
access.
## Grant and revoke access
```ts
await services.grant(
'main.agents.support_agent',
'customer-support',
['EXECUTE', 'READ_METADATA'],
);
const grants = await services.permissions('main.agents.support_agent');
console.log(grants.privilege_assignments);
await services.revoke(
'main.agents.support_agent',
'customer-support',
['EXECUTE', 'READ_METADATA'],
);
```
Assignable privileges are `EXECUTE`, `READ_METADATA`, `MANAGE`, `MANAGE_ACCESS_CONTROL`, and
`ALL_PRIVILEGES`. Prefer group grants, reserve `MANAGE_ACCESS_CONTROL` for the platform automation
principal, and avoid `ALL_PRIVILEGES` for application callers.
Multiple changes can be applied in one permission request:
```ts
await services.updatePermissions('main.agents.support_agent', [
{ principal: 'support-users', add: ['EXECUTE', 'READ_METADATA'] },
{ principal: 'former-support-users', remove: ['EXECUTE'] },
]);
```
Harness validates principal names and non-empty changes before issuing a request. Unity Catalog is
still the authority that decides whether the acting principal can manage the securable.
## Delete safely
Deleting the Agent Service removes only its Unity Catalog registration. It does not delete or stop
the external Harness App, erase its Lakebase sessions, or remove its HTTP connection.
```ts
await services.delete('main.agents.support_agent');
```
Use this order when retiring an agent:
1. Remove consumer grants or redirect clients.
2. Preserve required MLflow traces, audit records, and deployment receipts.
3. Delete the Agent Service registration.
4. Delete the HTTP connection only when no other governed object references it.
5. Retire the App and apply the session-retention policy.
For ephemeral CI registrations, always delete in `finally`:
```ts
let releaseProbeRegistered = false;
try {
await services.create({
catalog: 'main',
schema: 'agents',
id: 'fabric_release_probe',
connection: 'fabric_support_connection',
});
releaseProbeRegistered = true;
// discovery, update and permission assertions
} finally {
if (releaseProbeRegistered) await services.delete('main.agents.fabric_release_probe');
}
```
## Certify in every release
TechFabric Harness includes a protected live check that creates a unique temporary registration, reads
it, updates it, discovers it in the schema, reads its permissions, optionally exercises grant/revoke,
and deletes it. Configure the `databricks-live` GitHub Environment:
| Name | Kind | Purpose |
| --- | --- | --- |
| `DATABRICKS_HOST` | Secret | Workspace URL |
| `DATABRICKS_CLIENT_ID` | Secret | OAuth M2M service principal |
| `DATABRICKS_CLIENT_SECRET` | Secret | OAuth M2M secret |
| `DATABRICKS_CATALOG` | Variable | Parent catalog |
| `DATABRICKS_SCHEMA` | Variable | Parent schema |
| `DATABRICKS_AGENT_SERVICES_TEST` | Variable | Set to `1` after enabling the Beta |
| `DATABRICKS_AGENT_SERVICE_CONNECTION` | Variable | Existing UC HTTP connection resource |
| `DATABRICKS_AGENT_SERVICE_BASE_PATH` | Variable | Optional; defaults to `/responses` |
| `DATABRICKS_AGENT_SERVICE_TEST_PRINCIPAL` | Variable | Optional principal for grant/revoke evidence |
The certification runner automatically makes `agent-services` required when
`DATABRICKS_AGENT_SERVICES_TEST=1`:
```bash
pnpm --filter @fabric-harness/databricks build
node packages/databricks/dist/certify.js
```
Evidence is written to `artifacts/databricks-certification.json` with credentials redacted. A green
unit suite proves request and validation contracts; only the protected workspace run proves that the
preview is enabled and the actual principal has the necessary Unity Catalog grants.
## Naming and tenancy
Use one schema per ownership or data-governance boundary, then give services stable names:
```text
main.customer_operations.support_agent
main.data_platform.data_quality_agent
main.finance.close_assistant
```
Do not place tenant IDs or secrets in `comment`, `systemPrompt`, service names, or connection names.
For a multi-tenant Harness deployment, keep tenant enforcement in the Harness principal/RBAC layer
and use Unity Catalog schemas or grants for organizational boundaries. An Agent Service registration
does not weaken the agent definition's Harness capability policy.
## Failure guide
| Failure | Likely cause | Action |
| --- | --- | --- |
| `400 AgentServices feature is not available` or `404` | Account preview is disabled or unavailable in the region | Enable Agent Services in account Previews and confirm workspace availability; a successful empty list alone is not proof that create is enabled |
| `PERMISSION_DENIED` on create | Missing parent or connection privileges | Grant `USE CATALOG`, `USE SCHEMA`, `CREATE SERVICE`, and `USE CONNECTION` |
| `PERMISSION_DENIED` on grants | Missing control privilege | Grant `MANAGE_ACCESS_CONTROL` to the automation principal |
| Registration exists but calls fail | Registration is not an invocation proxy | Test the external App `/responses` route and its bearer token directly |
| Agent missing from Catalog Explorer | Consumer lacks metadata access | Grant `READ_METADATA` and verify the selected schema |
| Connection reaches the wrong path | Base paths are composed incorrectly | Keep the connection base path `/` and the service base path `/responses` |
| CI leaves an entry after interruption | Process was terminated before `finally` | Use a scheduled inventory cleanup by a name prefix and maximum age |
For the Databricks product contract and current limitations, see the official
[Agent Services documentation](https://docs.databricks.com/aws/en/ai-gateway/agent-services) and
[Unity Catalog HTTP connections](https://docs.databricks.com/aws/en/query-federation/http).
---
# Databricks workload map
Canonical: https://harness.techfabric.com/docs/databricks/workloads
Choose the TechFabric Harness API, managed recipe, or Databricks-native escape hatch for agent, data, ML, deployment, and governance workloads.
TechFabric Harness uses three integration levels. First-party workloads have typed APIs and an explicit
certification level in the package capability registry; Beta surfaces can remain contract-tested
until their protected live gate passes. Managed recipes orchestrate the official Databricks CLI,
Python SDK, SQL, or Jobs for
workloads whose authoring experience is Python-native. The escape hatch lets an agent invoke any
current Databricks REST, SQL, notebook, Job, or Declarative Automation Bundle resource without waiting
for a dedicated TypeScript wrapper. One bundle operation is no longer escape-hatch territory: the
validate/deploy/run/destroy lifecycle of a checked-in bundle is a first-party governed surface through
the [`assetBundles` option](/docs/databricks/authoring#checked-in-asset-bundles), with steward
approval and source-fingerprint drift detection.
| Workload | Use now | Integration level |
| --- | --- | --- |
| Conversational agent App | [`responses` configuration](/docs/databricks/responses-agent) and `databricks-app` | First-party |
| Model Serving agent | `databricks-serving` MLflow `ResponsesAgent` artifact | First-party |
| Unity AI Gateway models | `databricksFoundationModelProvider()` and model-service discovery | First-party, Databricks Beta |
| SQL and Unity Catalog | SQL tools/sandbox plus governance preflight | First-party |
| Stateful App | `databricksPersistence()` on Lakebase and UC Volume attachments | First-party |
| AI Search RAG | `databricksAiSearch()` query adapter and `createDatabricksRagChain()` | First-party query lifecycle |
| MLflow GenAI | trace exporter, evaluation rows, managed evaluation Job | First-party core lifecycle |
| Genie, Feature Serving, Jobs and Lakeflow | Package tools and Jobs client | First-party control APIs |
| Managed or custom MCP | `databricksWithManagedMcp()` with rotating OBO/M2M auth, allowlists, effects, and governed context | First-party Databricks adapter on shared MCP transport |
| Unity Catalog Agent Services | `bundle.agentServices` registration, discovery, grants and cleanup | First-party, Databricks Beta |
| Training, registry promotion and batch inference | Official Python SDK/notebook executed as a Harness-managed Job | Managed recipe |
| Dashboards, Sharing, Clean Rooms and Marketplace | Declarative Automation Bundle, Job, SQL or REST call | Databricks-native escape hatch |
## Start from the business workload
### Analytics and BI copilots
Use ordinary Genie conversations for governed natural-language analytics and
`databricksSqlReadTool()` when the model needs inspectable SQL. The
`analyticsCopilotGovernance()` pack leaves Genie questions and single-statement `SELECT` reads
interactive while routing arbitrary SQL and Genie lifecycle changes to a steward. The runnable
[`with-analytics-copilot`](/docs/reference/source-access)
example proves the allow, approval, and local-denial behavior without credentials.
For an employee-facing Databricks App, build the bundle per request with a verified OBO principal.
For a shared automation service, use OAuth M2M and a least-privilege service principal. In both
cases, retain the generated SQL, tool decision, acting principal label, Unity Catalog outcome, and
cost correlation with the submission.
### Knowledge assistants and RAG
Keep indexing, embedding pipelines, and governed source tables native to Databricks. Fabric queries
the AI Search index, builds bounded context, invokes AI Gateway or Model Serving, validates citations,
and exports evaluation rows to MLflow. This makes the application layer testable without pretending
to replace Databricks retrieval or evaluation.
### Data engineering and operations agents
Use Jobs and Lakeflow for compute. Fabric should admit the request, bind approval to the exact
definition or run input, submit idempotently, persist the receipt, collect terminal status and
outputs, and make cleanup visible. Keep pipeline source, cluster policies, service-principal grants,
and production promotion in the platform team's existing Databricks delivery process.
### Stateful customer and employee Apps
Deploy the Node runtime as a Databricks App, persist sessions/submissions/conversation streams in
Lakebase, and store governed attachments in UC Volumes. The release proof is behavioral: create a
session, restart or redeploy the App, recover it, continue from the prior offset, and verify tenant
deletion. A successful build alone is not durability evidence.
```mermaid
flowchart TB
NEED[Databricks workload] --> NATIVE{Typed Harness API?}
NATIVE -->|Yes| FIRST[First-party package surface]
NATIVE -->|No| PYTHON{Python or SQL native?}
PYTHON -->|Yes| RECIPE[Managed Job or notebook recipe]
PYTHON -->|No| ESCAPE[REST, SQL, Job, or Bundle escape hatch]
FIRST --> POLICY[Harness policy, identity, approvals, telemetry]
RECIPE --> POLICY
ESCAPE --> POLICY
POLICY --> EVIDENCE[Receipts, MLflow traces, lineage, cost, cleanup]
classDef decision fill:#fef3c7,stroke:#d97706,color:#422006
classDef route fill:#dbeafe,stroke:#2563eb,color:#172554
classDef evidence fill:#dcfce7,stroke:#16a34a,color:#052e16
class NATIVE,PYTHON decision
class FIRST,RECIPE,ESCAPE,POLICY route
class EVIDENCE evidence
```
## Agent and governance workloads
Use the Responses endpoint as the common boundary for Apps and subagents. Use
`databricksWithManagedMcp()` for Databricks managed MCP endpoints, Unity Catalog MCP Services, Genie
tools, or another App's MCP endpoint. The adapter rotates OBO/M2M credentials, enforces workspace
origin, applies `allowTools`, and requires an explicit effect for surfaces where read versus mutation
cannot be inferred. Remote discovery therefore cannot silently widen the local capability policy.
[Unity Catalog Agent Services](/docs/databricks/agent-services) is a typed discoverability and
permission surface. The package requires explicit Beta acknowledgement, defaults external Harness
registrations to `/responses`, and supports create, get, list, update, permission, grant, revoke, and
delete operations. Runtime invocation is not available through the Databricks service during the
current Beta; call the Harness endpoint directly.
For long-lived conversations, use Lakebase-backed Harness sessions today. Databricks managed agent
memory can be introduced as another session-memory adapter without changing agent code because
Harness sessions already depend on the common store interface.
## Retrieval and data workloads
Databricks AI Search is the current product name. Harness uses `databricksAiSearch()` and `aiSearch`
for new code while retaining the former names as deprecated source-compatible aliases. The REST API
continues to use its existing `vector-search` path. The online RAG path is documented in
[Databricks RAG](/docs/databricks/rag). Create and update indexes using a Databricks Job or Bundle,
then give the App service principal `SELECT` and endpoint access before invoking the retriever.
Lakeflow authoring belongs in declarative SQL or Python source. Keep pipeline definitions in the
deployed project, run them through a Job or Bundle, and use Harness Lakeflow tools for start, stop,
status and agent approvals. When the bundle itself is checked in, hand its validate/deploy/run/destroy
lifecycle to the governed [`assetBundles` surface](/docs/databricks/authoring#checked-in-asset-bundles)
instead of an ungated shell call. Treat expectations and event-log quality metrics as release evidence.
## ML and evaluation workloads
Use MLflow for trace storage, evaluation datasets, judges, labeling sessions, reusable scorers and
production monitoring. Harness supplies execution identity and correlation; MLflow remains the
evaluation engine. Use [RAG evaluation](/docs/databricks/rag#managed-mlflow-evaluation) as the working
pattern for a generated Python evaluation Job in [evaluation and quality](/docs/databricks/rag#evaluation-and-quality-native-databricks-first).
Training and fine-tuning should run on Databricks AI Runtime through a Job or notebook. Register the
result in Unity Catalog, promote with model aliases, and deploy with Model Serving traffic controls.
Do not build new workflows on the deprecated Foundation Model Training API.
## Adding a new workload
Before exposing a new Databricks API, define its auth modes, UC permissions, cloud availability,
preview status, cleanup behavior, retry/idempotency rules, mock contract, live gate, runnable example,
and documentation route. The package capability registry is the source of truth for these claims:
```ts
import { listDatabricksCapabilities } from '@fabric-harness/databricks';
const productionSurfaces = listDatabricksCapabilities({ status: 'stable' });
```
This keeps experimental Databricks surfaces usable without presenting them as generally available or
silently enabling them in production.
---
# Databricks recipes (`fh add`)
Canonical: https://harness.techfabric.com/docs/databricks/recipes
Scaffold Genie analytics copilots, Lakebase, SQL, AI Search, Lakeflow, Jobs, cost controls, and Apps wiring with managed TechFabric Harness recipes.
Use managed **Databricks recipes** to wire `@fabric-harness/databricks` into an existing project without copying examples by hand. Recipes write versioned files under `.fabricharness/databricks/` (or a job for the analyst composite), update dependencies, and leave secrets in environment variables.
Greenfield projects can still start with:
```sh
fh init --template databricks
```
Recipes are for **adding one product surface** (or a small workload composite) afterward.
Generated Databricks recipes default to `DATABRICKS_MODEL=databricks-gpt-oss-20b` (a bare
foundation-model serving endpoint, available on free-tier workspaces) and
`DATABRICKS_INFERENCE_MODE=auto`. Use a discovered `system.ai.*` model service instead when the
workspace has Unity AI Gateway enabled. Set `DATABRICKS_HOST` to the workspace origin, for example
`https://`; do not include `/ai-gateway/mlflow/v1` in the host value.
## List and install
```sh
fh add
fh add --json
fh add databricks core
fh add databricks sql
fh add lakebase # alias
fh add ai-search
fh add lakeflow
fh add jobs
fh add bundle # checked-in Asset Bundle lifecycle
fh add system-tables-cost
fh add apps
fh add agent-services
fh add genie # governed Genie + SELECT-only SQL copilot
fh add mcp--databricks # governed managed MCP client
fh add data--databricks # compatibility alias for genie
fh add kb--databricks # compatibility alias for rag-chain
fh add lakehouse # analyst composite (sql + tables)
fh add databricks lakebase --dry-run
fh update databricks sql
```
## Catalog
| Recipe | Alias examples | Managed files | Dependencies | Maps to |
| --- | --- | --- | --- | --- |
| `core` | `databricks-core` | `databricks/identity.ts`, `policies/databricks.ts` | `@fabric-harness/databricks` | Workspace identity + UC egress policy |
| `sql` | `databricks-sql` | `databricks/sql.ts` | `@fabric-harness/databricks` | SELECT-only SQL Warehouse + table discovery |
| `lakebase` | `databricks-lakebase` | `databricks/lakebase.ts` | `@fabric-harness/databricks`, `pg` | Lakebase persistence |
| `ai-search` | `databricks-ai-search` | `databricks/ai-search.ts` | `@fabric-harness/databricks` | Agentic `search` tool bundle |
| `rag-chain` | `databricks-rag-chain` | `databricks/rag-chain.ts`, `jobs/rag-answer.ts` | `@fabric-harness/databricks` | Cookbook online chain + MLflow 3 eval export ([RAG docs](/docs/databricks/rag)) |
| `lakeflow` | `databricks-dataeng` | `databricks/lakeflow.ts` | `@fabric-harness/databricks` | [with-databricks-dataeng](/docs/reference/source-access) |
| `jobs` | `databricks-compute` | `databricks/jobs.ts` | `@fabric-harness/databricks` | [with-databricks-compute](/docs/reference/source-access) |
| `bundle` | `databricks-bundle`, `databricks-asset-bundle` | `databricks/bundle.ts`, `policies/databricks-bundle.ts` | `@fabric-harness/databricks` | Governed checked-in Asset Bundle lifecycle; deploy/run/destroy added to `requireApproval` ([resource management](/docs/databricks/authoring#checked-in-asset-bundles)) |
| `system-tables-cost` | `databricks-cost` | `databricks/cost.ts` | `@fabric-harness/databricks` | [with-databricks-cost-attribution](/docs/reference/source-access) |
| `apps` | `databricks-app` | `databricks/apps.ts` | `@fabric-harness/databricks`, `@fabric-harness/node` | Apps runtime preset + deploy targets |
| `agent-services` | `databricks-agent-services` | `databricks/agent-service.ts` | `@fabric-harness/databricks` | [Unity Catalog registration and grants](/docs/databricks/agent-services) |
| `genie` | `analytics-copilot`, `data--databricks` | `databricks/genie.ts` | `@fabric-harness/databricks` | Genie questions + statement-level SELECT-only SQL + copilot governance |
| `managed-mcp` | `mcp--databricks` | `databricks/managed-mcp.ts` | `@fabric-harness/databricks` | Managed MCP / AI Gateway MCP Service discovery with explicit allowlist and effects |
| `analyst` | `lakehouse` | `jobs/databricks-analyst.ts` | `@fabric-harness/databricks` | init template / simple analyst job |
“Lakehouse” is an **alias for the analyst composite** (governed SQL + UC tools), not a separate Databricks SDK product.
`kb--databricks` maps to `rag-chain`. `mcp--databricks` now maps to the tested managed-MCP adapter;
the generated factory is asynchronous because remote tool discovery happens at startup.
## Typical flows
### SQL analyst on Apps
```sh
fh init --template minimal
fh add databricks core
fh add databricks sql
fh add databricks analyst
# set DATABRICKS_* in .env
fh run databricks-analyst --question "What tables are in main?" --mock
fh build --target databricks-app
```
### Genie analytics copilot
```sh
fh init --template minimal
fh add databricks genie
# set host, identity, Warehouse, Genie Space, catalog, and steward audience
fh test
```
The generated bundle adds `databricks_genie_ask` for governed natural-language questions and a
separate `sql_read` tool. `sql_read` rejects mutations and multiple statements before calling
Statement Execution. Arbitrary SQL is not generated; add `databricksSqlTool()` only with its
required statement policy and separate approval routing. Genie lifecycle tools remain
approval-bound. Missing
Warehouse, Genie Space, identity, or steward configuration fails during bundle construction.
The `DATABRICKS_GENIE_SPACE_ID` environment value is passed to the current conversation contract as
`agentId`; generated code does not emit the retired legacy configuration alias.
### RAG support agent
```sh
# Deterministic cookbook chain (retrieve → augment → generate):
fh add databricks rag-chain
fh run rag-answer --question "How do I reset my password?"
# Or agentic multi-tool search (SQL + search + …):
fh add ai-search
# Import createVectorSearchBundle() and pass modelProvider + tools into init()
```
See [RAG on Databricks](/docs/databricks/rag) for citation validation, MLflow 3 export, and the quality workflow.
### Managed MCP or AI Gateway MCP Service
```sh
fh add mcp--databricks
# set host, identity, a same-workspace MCP URL, and one safe tool name
fh test
```
The generated `createDatabricksManagedMcp()` returns a principal-bound bundle. Close
`bundle.managedMcp` when the request or worker stops. Its default recipe exposes one named tool as
`read`; expand `allowTools` and `effects` deliberately after reviewing the server contract. Missing
host, credentials, URL, tool name, remote permission, or effect classification fails before the tool
is usable. See the [managed MCP integration guide](/docs/databricks/integrations#managed-mcp-and-mcp-services).
### Lakebase durable state
```sh
fh add lakebase
# In .fabricharness/config.ts:
# import { createLakebasePersistence } from './databricks/lakebase.js';
# persistence: createLakebasePersistence(),
```
### Data engineering
```sh
fh add lakeflow
fh add jobs
# Use createLakeflowBundle() / createDatabricksJobsTools() in agent init
```
### Checked-in Asset Bundle lifecycle
```sh
fh add databricks bundle
# set DATABRICKS_HOST, identity, DATABRICKS_BUNDLE_DIR, DATABRICKS_BUNDLE_TARGET
# Use createDatabricksAssetBundleTools() in agent init; deploy/run/destroy require approval
```
The generated `createDatabricksAssetBundleTools()` wires `databricksAssetBundleLifecycle` and its
tools for the checked-in bundle in `DATABRICKS_BUNDLE_DIR`; the policy file adds
`databricks_bundle_deploy`, `databricks_bundle_run`, and `databricks_bundle_destroy` to
`toolPolicy.requireApproval` while validation stays read-only. Run submission is always
`--no-wait`; poll status through the bounded jobs/lakeflow tools. The Databricks CLI must be on
`PATH`. See
[Checked-in Asset Bundles](/docs/databricks/authoring#checked-in-asset-bundles) for the fingerprint
and managed-only-destroy semantics.
## Conventions
- **Package owns behavior** — recipes only wire `@fabric-harness/databricks` into your workspace.
- **Managed markers** — files start with `// fabric-harness-recipe: databricks/@1` for safe `fh update`.
- **No secrets in source** — only env names are written to `.env.example`.
- **Runnable verification** — generated tests import from `../../.fabricharness/` and are compiled by the recipe contract suite.
- **Merge config yourself** — recipes do not silently rewrite `config.ts`; they print paths and env requirements.
- **Deploy** — Apps/Serving artifacts still come from `fh build --target databricks-app|databricks-serving`.
## See also
- [Databricks quickstart](/docs/databricks/quickstart)
- [Integrations map](/docs/databricks/integrations)
- [Connector recipes reference](/docs/reference/recipes)
- [Examples](/docs/examples)
---
# RAG on Databricks
Canonical: https://harness.techfabric.com/docs/databricks/rag
Use Databricks-native AI Search and Unity AI Gateway for bounded, citation-validated online RAG with MLflow 3 evaluation records.
TechFabric Harness wires Databricks-native products for retrieval, generation, tracing, and evaluation. It does **not** reimplement AI Search, Unity AI Gateway, the offline index pipeline, or MLflow judges.
Follow the same mental model as the [Databricks AI Cookbook RAG inference chain](https://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/fundamentals-inference-chain-rag):
1. (Optional) preprocess the user query
2. **Retrieve** with Databricks AI Search
3. **Augment** the prompt with retrieved context
4. **Generate** through Unity AI Gateway or a custom Model Serving endpoint
5. Validate inline citations and apply answer limits
Offline chunk → embed → index remains a **Databricks Job / notebook / Lakeflow** concern. Quality measurement uses **[MLflow 3 evaluation](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/)** and managed Databricks judges, not a parallel Fabric-only judge product.
## Quick start (managed recipe)
```sh
fh add databricks rag-chain
# set DATABRICKS_HOST, OAuth credentials, DATABRICKS_AI_SEARCH_INDEX, DATABRICKS_MODEL
fh run rag-answer --question "How do I reset my password?"
```
For a credential-free local run, add `--mock`:
```sh
fh run rag-answer --question "How do I reset my password?" --mock
```
The generated recipe then selects its deterministic retriever and model fixture. This exercises
discovery, input/output validation, the RAG chain, and citation handling without calling AI Search
or Model Serving. Remove `--mock` to use the configured Databricks index and model.
Scaffolded files:
| Path | Role |
| --- | --- |
| `.fabricharness/databricks/rag-chain.ts` | `createRagChain()` + `evaluationArtifacts()` |
| `.fabricharness/jobs/rag-answer.ts` | Finite job calling the chain |
Related recipes:
| Recipe | Use when |
| --- | --- |
| `fh add databricks ai-search` | Bundle-only / agentic `search` tool (multi-tool agents) |
| `fh add databricks rag-chain` | Fixed online inference chain (cookbook path) |
| `fh add lakeflow` / Jobs | Offline pipeline ops (index refresh), not the chain itself |
## Code API
### Deterministic chain (cookbook online path)
```ts
import { databricksRagChain, toMlflow3EvaluationRecord, exportMlflow3EvaluationJsonl } from '@fabric-harness/databricks';
const chain = databricksRagChain({
databricks: {
host: process.env.DATABRICKS_HOST!,
principal: { kind: 'pat', token: process.env.DATABRICKS_TOKEN! },
model: 'databricks-gpt-oss-20b',
aiSearch: {
index: 'main.support.kb_index',
textColumn: 'chunk',
idColumn: 'id',
inputMode: 'text',
strategy: 'hybrid',
},
},
retrieval: { k: 5, scoreThreshold: 0.25 },
postProcess: {
requireCitations: true,
citationRepairAttempts: 2,
},
});
const turn = await chain.invoke({ question: 'How do I reset my password?' });
// turn.answer, turn.sources, turn.citations, turn.retrieval, turn.usage
```
With `requireCitations`, Fabric asks the model to revise an uncited answer or an answer that cites an
unknown ID using only the exact source IDs returned by retrieval. Repair is bounded and fails closed
if the model still omits a valid marker or retains a fabricated ID; Fabric never adds a citation to
the answer on the model's behalf.
### Complete AI Search query controls
Text input uses native `HYBRID` retrieval by default. Query representation and retrieval strategy
are separate:
```ts
const result = await bundle.retriever.query('quarterly retention', {
inputMode: 'text',
strategy: 'hybrid', // ann | hybrid | full-text
k: 20,
filter: { language: 'en' },
scoreThreshold: 0.35,
queryColumns: ['chunk', 'title'],
sortColumns: ['published_at DESC'],
facets: ['language', 'product'],
reranker: { model: 'databricks_reranker' },
columnsToRerank: ['title', 'chunk'],
signal: abortController.signal,
});
result.chunks;
result.facetResult;
result.nextPageToken;
result.response; // complete generated-SDK response
```
Use `inputMode: 'vector'` with `strategy: 'ann'` for a self-managed embedding index. Supply an
`embeddingEndpoint` when configuring the bundle or provide a precomputed `queryVector` for a
specific request. `text-and-vector` sends both inputs for a supported native strategy.
The same controls can be defaults on `databricksRagChain({ retrieval: ... })` or per-turn:
```ts
const turn = await chain.invoke({
question: 'What changed?',
retrieval: {
strategy: 'hybrid',
k: 12,
facets: ['release'],
reranker: { model: 'databricks_reranker' },
},
});
```
Pagination and facet metadata are retained on `RagTurn.retrieval`. For direct search calls,
`retriever.nextPage(token, { signal })` retrieves the following page.
### Streaming
`chain.stream()` yields incremental `delta` events while the model generates, then one terminal
`turn` event carrying the same validated `RagTurn` that `invoke` returns:
```ts
for await (const event of chain.stream({ question: 'How do I reset my password?' })) {
if (event.type === 'delta') process.stdout.write(event.textDelta);
if (event.type === 'turn') persist(event.turn); // validated answer, citations, usage
}
```
Deltas are raw first-pass model output for live UI — they are emitted **before** citation
validation, bounded repair, truncation, and the sources footer run, so the concatenated deltas can
differ from the final answer. The terminal `turn` is authoritative: use it for persistence,
evaluation export, and cost telemetry. A failed citation validation raises from the iterator after
the deltas were observed. When the model provider does not implement `stream()`, the chain falls
back to `generate()` and emits the whole answer as one delta.
The protected `rag` certification check uses this streaming path against the configured live AI
Search index and Databricks model endpoint. It requires text deltas, a terminal turn, retrieved
context, and at least one valid inline citation.
Release run `29622641870` certified the exact `2.0.0` package candidate in the protected Azure
`eastus2` workspace: the streaming chain emitted 12 text deltas, retrieved three source chunks,
returned a valid `fabric-databricks` citation, and produced its authoritative terminal turn. The
same run completed Databricks Job `613907318899842` for the managed 50-case RAG evaluation with a
`SUCCESS` result. Agent Mode was not configured in that workspace and is not implied by this RAG
evidence.
That record predates the 3.0 native-SDK transport and is historical for 3.0 promotion. The 3.0
release gate reruns the same retrieval, streaming, citation, and managed-evaluation checks against
the exact package artifact.
What runs under the hood:
| Step | Databricks product | Fabric helper |
| --- | --- | --- |
| Retrieve | Databricks AI Search | `databricksAiSearch` / UC principal |
| Generate | Unity AI Gateway / Model Serving | `databricksFoundationModelProvider` |
| Orchestrate | — | `databricksRagChain` (thin glue only) |
### Agentic multi-tool RAG
When the agent must also call SQL, Genie, or Jobs, keep tool-calling:
```ts
const chain = databricksRagChain({ databricks: { /* + aiSearch */ } });
const { modelProvider, tools, policy } = chain.asAgentTools();
// pass into init({ modelProvider, tools, policy })
```
Or continue using `databricks({ aiSearch })` + `bundle.tools` as in `examples/with-databricks-rag`.
## Evaluation and quality (native Databricks first)
### Local CI smoke (optional)
Lightweight checks on a `RagTurn` validate actual `[source-id]` markers, required facts, and retrieval:
```ts
import { scoreRagTurn, toMlflow3EvaluationRecord, exportMlflow3EvaluationJsonl } from '@fabric-harness/databricks';
const scores = scoreRagTurn(turn, {
mustContain: ['Settings'],
requireCitations: true,
mustRetrieveIds: ['doc-1'],
});
```
These are **smoke scorers**, not a substitute for managed evaluation.
### What developers inspect
Keep the answer and its source references visible beside quality evidence. The representative state
below shows the relationship between retrieved context, citations, managed judges, latency, and
cost; replace the fixture corpus and expected facts with the application's governed domain data.
### MLflow 3 managed evaluation
Export MLflow 3 rows with structured inputs, outputs, expectations, retrieved context, and trace metadata:
```ts
const record = toMlflow3EvaluationRecord(turn, {
expectedAnswer: 'Use Settings, then Security.',
traceId: 'tr-...',
submissionId: 'sub-...',
});
const jsonl = exportMlflow3EvaluationJsonl([record]);
// Merge the record into an MLflow Evaluation Dataset from a Databricks notebook or job.
```
The release certification fixture runs this loop as a serverless Databricks Job. The notebook at
`scripts/databricks/rag_evaluation.py` queries five candidates from the real AI Search index,
uses the configured Databricks model to retain only sources that supply relevant or complementary
evidence, records the filtered documents in the retriever span, and generates a citation-backed
answer. It merges the governed 50-case golden set into a content-hashed Unity Catalog evaluation
dataset, so changed fixtures cannot inherit stale rows from an earlier release, and runs the
following managed judges:
| Case category | Count | Purpose |
| --- | ---: | --- |
| Factual | 30 | Paraphrased questions over individual governed documents |
| Multi-document | 10 | Answers that must combine identity, governance, runtime, deployment, or RAG evidence |
| Insufficient context | 5 | Unsupported questions where the correct behavior is to abstain |
| Adversarial | 5 | Retrieved prompt-injection text that must be treated as untrusted data |
The source documents and cases live together in `scripts/databricks/rag_fixture.json`. The
provisioner merges those documents into the Delta source table, triggers the AI Search index,
and embeds the same cases into the evaluation notebook. This prevents the index fixture and golden
set from drifting apart.
| Judge | What it catches |
| --- | --- |
| Relevance to query | The answer does not address the request |
| Retrieval relevance | Retrieved chunks add irrelevant context |
| Retrieval groundedness | Retrieved claims are not supported by the source chunks |
| Retrieval sufficiency | Retrieval omitted context needed to answer |
| Correctness | The answer misses the expected facts |
Each aggregate must meet `DATABRICKS_RAG_EVAL_THRESHOLD` (default `0.8`). Evidence includes the
MLflow run, versioned dataset and fixture hash, generation and judge models, category counts,
selected aggregate metrics, and threshold result. A failed judge fails the Databricks Job and
therefore the release certification gate.
The protected release workflow for commit `5fcf927bcf0a3a15e3aec86e2629516cbb76ad26`
evaluated fixture `18bd4dffc32c` as MLflow run `18131356bcac4460a26de20d40b9573a` in the
reference Azure workspace. It covered all 50 cases with `databricks-gpt-oss-120b` for generation
and managed judging, and passed the unchanged `0.8` floor:
| Metric | Score |
| --- | ---: |
| Relevance to query | 0.86 |
| Retrieval relevance | 0.895 |
| Retrieval groundedness | 0.84 |
| Retrieval sufficiency | 0.86 |
| Correctness | 0.88 |
This is retained workspace evidence for the reference fixture, not a claim that every application
or dataset will achieve the same scores. Replace the documents, question-level expected facts, and
adversarial cases with your domain corpus before treating the gate as production evidence.
```mermaid
flowchart LR
G[UC golden dataset content-hashed version] --> J[Serverless evaluation Job]
J --> V[AI Search top five candidates]
V --> R[Model-assisted reranker relevant sources only]
R --> T[MLflow retriever span]
T --> M[AI Gateway generation bounded context and citations]
T --> E[Five MLflow managed judges]
M --> E
E -->|all scores at least 0.8| P[Release evidence]
E -->|score below threshold| F[Block release]
F --> D[Diagnose retrieval, prompt, or index]
D --> J
classDef source fill:#dcfce7,stroke:#16a34a,color:#052e16
classDef gate fill:#fef3c7,stroke:#d97706,color:#422006
classDef fail fill:#fee2e2,stroke:#dc2626,color:#450a0a
class G,V,R,T,M source
class J,E,P gate
class F,D fail
```
Provision or repair the disposable fixture, then run the same gate used by protected CI:
```sh
FABRIC_DATABRICKS_PROVISION=1 pnpm databricks:cert:provision
pnpm databricks:certify
```
The provisioner is idempotent: it reuses the UC dataset, Jobs, AI Search index, Feature Serving
endpoint, Genie Agent, and Lakeflow pipeline. Improve quality by diagnosing retrieval versus
generation, changing `topK`, prompts, chunks, or embeddings, and rerunning the managed judges before
redeploying Apps or Model Serving.
The default job limits MLflow to one prediction worker and one scorer worker, with prediction and
scorer rate limits, so pay-per-token endpoints do not exceed workspace output-token quotas. Raise
`MLFLOW_GENAI_EVAL_MAX_WORKERS`, `MLFLOW_GENAI_EVAL_MAX_SCORER_WORKERS`,
`MLFLOW_GENAI_EVAL_PREDICT_RATE_LIMIT`, or `MLFLOW_GENAI_EVAL_SCORER_RATE_LIMIT` only when the
workspace uses capacity that supports the additional concurrency.
Do **not** build a second full judge product in Fabric when MLflow managed evaluation already exists in the workspace.
## Offline index pipeline (not the chain)
Building/updating the AI Search index (chunking, embedding, write) stays on Databricks:
- Notebooks / Jobs
- Lakeflow pipelines (`fh add lakeflow`)
- UC Volumes as document sources
Fabric agents **consume** the index via AI Search at inference time.
## Guardrails
| Concern | Prefer |
| --- | --- |
| Data access | Unity Catalog grants on the index + service principal |
| Tool / SQL risk | `databricks()` governance policy / approvals |
| Content policy / gateway | Mosaic AI Gateway / serving policies (configure on the endpoint) |
| Audit | MLflow traces (`bundle.mlflowTraceExporter()`) + lineage hooks |
The default chain treats retrieved chunks as untrusted data, serializes them as bounded JSONL,
limits per-chunk and total context size, and rejects citation markers that do not match a retrieved
source. `turn.sources` means "retrieved"; only source IDs referenced by the final answer appear in
`turn.citations`.
```ts
const chain = databricksRagChain({
databricks: { /* model + aiSearch */ },
topK: 5,
maxContextChars: 32_000,
maxChunkChars: 8_000,
postProcess: {
citationPolicy: 'validate',
maxAnswerChars: 8_000,
},
});
```
## See also
- [Databricks recipes](/docs/databricks/recipes)
- [Integrations map](/docs/databricks/integrations)
- [example: with-databricks-rag](/docs/reference/source-access)
- [Cookbook: RAG chain for inference](https://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/fundamentals-inference-chain-rag)
---
# Databricks architecture
Canonical: https://harness.techfabric.com/docs/databricks/architecture
How TechFabric Harness routes identity, policy, model calls, data tools, durable state, telemetry, and deployments through Databricks.
TechFabric Harness separates the agent runtime from the services the agent is allowed to call. A single
Databricks principal can be threaded through Model Serving, REST tools, SQL, and Lakebase credential
exchange. Fabric policy adds approvals, egress restrictions, budgets, and audit correlation; Unity
Catalog still makes the final data authorization decision.
## Runtime layers
```mermaid
flowchart TB
subgraph Entry[Invocation]
CLI[fh run]
API[Jobs and agents API]
EVT[Schedule or channel]
end
subgraph Runtime[TechFabric Harness runtime]
DEF[Job or persistent agent definition]
SES[Session and model loop]
POL[Capability policy]
APR[Human approval]
AUD[Audit, lineage, and cost]
end
subgraph Identity[Databricks identity]
APP[App service principal]
OBO[On-behalf-of user]
M2M[OAuth M2M]
end
subgraph Services[Databricks services]
SERVE[Model Serving]
SQL[SQL Warehouse]
UC[Unity Catalog]
RAG[AI Search]
STATE[Lakebase]
OPS[MLflow and system tables]
end
CLI --> DEF
API --> DEF
EVT --> DEF
DEF --> SES
SES --> POL
POL -->|sensitive action| APR
POL -->|allowed action| Services
SES --> AUD
APP --> SES
OBO --> SES
M2M --> SES
SERVE --> UC
SQL --> UC
RAG --> UC
STATE --> AUD
OPS --> AUD
classDef entry fill:#f4f4f5,stroke:#71717a,color:#18181b
classDef fabric fill:#dbeafe,stroke:#2563eb,color:#172554
classDef identity fill:#fef3c7,stroke:#d97706,color:#422006
classDef dbx fill:#dcfce7,stroke:#16a34a,color:#052e16
class CLI,API,EVT entry
class DEF,SES,POL,APR,AUD fabric
class APP,OBO,M2M identity
class SERVE,SQL,UC,RAG,STATE,OPS dbx
```
## One request, one governed identity
```mermaid
sequenceDiagram
autonumber
actor User
participant App as Databricks App
participant Fabric as TechFabric Harness
participant Policy as Policy and approvals
participant Model as Model Serving
participant Tool as SQL or Databricks API
participant UC as Unity Catalog
participant State as Lakebase
User->>App: Prompt or job input
App->>Fabric: Request plus app or user identity
Fabric->>State: Load session and submission
Fabric->>Model: Prompt with short-lived OAuth token
Model-->>Fabric: Tool request
Fabric->>Policy: Evaluate capability and approval rules
alt approval required
Policy-->>User: Approval request
User-->>Policy: Approve or deny
end
Policy->>Tool: Execute as the governed principal
Tool->>UC: Enforce catalog, schema, table, row, and column grants
UC-->>Tool: Authorized result or denial
Tool-->>Fabric: Redacted result plus lineage metadata
Fabric->>State: Persist events and final result
Fabric-->>User: Typed response or durable receipt
```
## Finite jobs and persistent agents
Finite jobs live in `.fabricharness/jobs/`, return one typed result, and are invoked at
`POST /jobs/:name`. Persistent agents live in `.fabricharness/agents/`, retain an addressable
conversation, and return a submission receipt from `POST /agents/:name/:id`. Both use the same model,
tool, policy, and Databricks integration surfaces.
Lakebase can back sessions, submissions, and conversation streams. The runtime exchanges a workspace
OAuth token for a short-lived database credential, supplies that credential through the Postgres
pool, refreshes early, and deduplicates concurrent refreshes.
## Deployment boundaries
`databricks-app` runs the bundled Node server inside Databricks Apps. `databricks-serving` builds an
MLflow `ResponsesAgent` proxy that calls an agent hosted elsewhere; it does not execute the TypeScript
runtime inside Model Serving. The App exposes the same `/responses` contract directly. The proxy
preserves `response.output_text.delta` and `response.output_item.done` events and returns a normal
`ResponsesAgentResponse` for non-streaming clients.
See [Databricks deployment](/docs/deployment/databricks) for target details.
---
# Databricks quickstart
Canonical: https://harness.techfabric.com/docs/databricks/quickstart
Install, scaffold, mock, connect, test, build, and deploy a governed Databricks agent with TechFabric Harness.
This walkthrough starts with no credentials, proves the local agent contract, then connects the same
project to Databricks with OAuth. Use the **workspace origin** everywhere, for example
`https://adb-1234567890123456.7.azuredatabricks.net`. Do not use an AI Gateway path as the host.
For an end-to-end customer architecture—OBO identity, Genie, SELECT-only verification,
exact-operation approval for one allowlisted Job, Lakebase recovery, MLflow correlation, and an
optional Temporal worker—use the runnable
[`with-databricks-revenue-ops`](/docs/reference/source-access) reference after this quickstart.
```mermaid
flowchart LR
Install[Install project CLI] --> Scaffold[Scaffold Databricks agent]
Scaffold --> Mock[Run with deterministic mock]
Mock --> Login[Databricks CLI OAuth login]
Login --> Live[Live Genie + governed SQL + cost run]
Live --> Extend[Add RAG, Lakeflow, or Lakebase recipes]
Extend --> Build[Build portable Databricks App]
classDef local fill:#e8f1ff,stroke:#4f7fd9,color:#172033
classDef workspace fill:#e5f7ec,stroke:#2f9e62,color:#172033
class Install,Scaffold,Mock local
class Login,Live,Extend,Build workspace
```
## 1. Check the prerequisites
The core TechFabric Harness packages support Node.js 20.18 or later, but
`@fabric-harness/databricks` requires **Node.js 22 or later**. Use Node 22 LTS for this walkthrough.
The Databricks CLI is optional for mock mode and required for the recommended local OAuth and
deployment flow.
```sh
node --version
npm --version
databricks version
```
If `databricks` is missing, install the current CLI using the
[official Databricks instructions](https://docs.databricks.com/aws/en/dev-tools/cli/install):
```sh
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
databricks version
```
## 2. Create the agent project
Run the published CLI directly. The generated project pins compatible Harness packages together,
so a global CLI is not required.
```sh
npx --yes @fabric-harness/cli@latest init \
--template databricks \
--dir analytics-agent
cd analytics-agent
npm install
```
The template creates:
```txt
.fabricharness/
jobs/databricks-analyst.ts
policies/databricks.ts
roles/data-analyst.md
skills/analyze-table/SKILL.md
fixtures/analytics.json
config.ts
databricks-certification.json
test/databricks-safe-tools.test.ts
NEXT_STEPS.md
.env.example
```
Use the project-local executable for every following command:
```sh
npx fh --version
npx fh doctor --getting-started --tools --model mock/test-model
npm test
npx fh agents
npx fh describe databricks-analyst
```
## 3. Run without credentials
```sh
npx fh run databricks-analyst \
--question "What tables are in main?" \
--mock
```
`--mock` replaces the model with a deterministic in-process provider. It needs no Databricks or AI
credentials and verifies that Harness can discover and load the TypeScript definition, local policy,
role, skill, schema, model loop, and result contract.
`npm test` separately proves that the generated SQL tool admits a SELECT and rejects a mutation
before the mock Statement Execution client is called. Neither check requires credentials.
The answer is synthetic. Mock mode does **not** list real tables, execute SQL, check Unity Catalog
grants, call AI Gateway, or validate OAuth. A response such as `Mock response: What tables are in
main?` proves the application path, not the factual quality of a data answer.
## 4. Create another simple agent
Generate a finite job in the same project:
```sh
npx fh new job hello
```
Replace `.fabricharness/jobs/hello.ts` with:
```ts title=".fabricharness/jobs/hello.ts"
import { defineAgent, schema } from '@fabric-harness/sdk';
export default defineAgent({
name: 'hello',
description: 'Give a short personalized greeting.',
input: schema.object({ name: schema.string() }),
output: schema.string(),
triggers: { manual: true, webhook: true },
async run({ input, prompt }) {
return await prompt(`Greet ${input.name} in one sentence.`);
},
});
```
Discover and run it without provider credentials:
```sh
npx fh agents
npx fh describe hello
npx fh run hello --name Ada --mock
```
To test the deployment protocol locally, start the HTTP server in one terminal:
```sh
npx fh dev --mock --port 4000
```
Invoke the job from another terminal:
```sh
curl --fail --request POST http://127.0.0.1:4000/jobs/hello \
--header 'content-type: application/json' \
--data '{"name":"Ada"}'
```
## 5. Authenticate to Databricks
For a developer workstation, use a named Databricks CLI OAuth profile. The browser login stores and
refreshes the user OAuth credential through the Databricks CLI; TechFabric Harness asks the CLI for a
short-lived access token when it needs one.
```sh
export DATABRICKS_HOST=https://adb-1234567890123456.7.azuredatabricks.net
databricks auth login \
--host "$DATABRICKS_HOST" \
--profile fabric-harness
databricks auth profiles
databricks current-user me --profile fabric-harness
npx fh doctor --target databricks-app
```
In a headless shell (CI container, SSH without a desktop keyring) the Databricks CLI cannot reach
its secure token cache; add `DATABRICKS_AUTH_STORAGE=plaintext` so `databricks auth token` works.
Create `.env.local`:
```dotenv title=".env.local"
DATABRICKS_HOST=https://adb-1234567890123456.7.azuredatabricks.net
DATABRICKS_CONFIG_PROFILE=fabric-harness
DATABRICKS_AUTH_MODE=cli
DATABRICKS_MODEL=databricks-gpt-oss-20b
DATABRICKS_WAREHOUSE_ID=0123456789abcdef
DATABRICKS_GENIE_SPACE_ID=0123456789abcdef0123456789abcdef
DATABRICKS_CATALOG=main
DATABRICKS_ANALYTICS_STEWARD_AUDIENCE=analytics-stewards
DATABRICKS_COST_TENANT_ID=acme
DATABRICKS_COST_PER_DAY_USD=50
DATABRICKS_INFERENCE_MODE=auto
```
The workspace OAuth credential authenticates both management/data APIs and the selected Databricks
model surface. You do not configure a separate OpenAI key. The principal still needs permission to
use the model or serving endpoint, `CAN USE`/`CAN QUERY` for the SQL Warehouse as appropriate, and
the required Unity Catalog grants.
For unattended CI or production-like local tests, replace CLI auth with OAuth M2M:
```dotenv
DATABRICKS_HOST=https://adb-1234567890123456.7.azuredatabricks.net
DATABRICKS_CLIENT_ID=00000000-0000-0000-0000-000000000000
DATABRICKS_CLIENT_SECRET=resolve-from-a-secret-store
DATABRICKS_MODEL=databricks-gpt-oss-20b
DATABRICKS_WAREHOUSE_ID=0123456789abcdef
DATABRICKS_GENIE_SPACE_ID=0123456789abcdef0123456789abcdef
DATABRICKS_CATALOG=main
DATABRICKS_ANALYTICS_STEWARD_AUDIENCE=analytics-stewards
DATABRICKS_COST_TENANT_ID=acme
DATABRICKS_COST_PER_DAY_USD=50
```
Do not set CLI-profile, PAT, and M2M credentials together when testing authentication behavior. See
[Local naming and authentication](/docs/databricks/local-naming-auth) for PAT, M2M, Databricks App
service principal, and OBO patterns.
## 6. Run the analyst against the workspace
First run the target-aware preflight, then remove `--mock`:
```sh
npx fh doctor --target databricks-app
npx fh run databricks-analyst \
--question "List the tables in main.default"
npx fh run databricks-analyst \
--question "Describe main.default.my_table and suggest three quality checks"
```
The generated analyst is the governed analytics-copilot spine. Ordinary Genie questions and one
statement `SELECT` through `sql_read` stay interactive; arbitrary SQL remains available only behind
the analytics-steward approval route. Unity Catalog discovery and System Tables consumption are
included, and an actual-cost-backed daily tenant limit blocks the next model ask after the configured
budget is exhausted. Test an allowed query, inspect the SQL, verify a denied Unity Catalog object,
and exercise the approval path.
Live startup fails once, before the first model ask, listing every missing prerequisite (workspace
host, Warehouse, Genie Agent, steward audience, tenant id, positive daily budget) in a single error
instead of failing fast one variable at a time. Mock mode deliberately bypasses those workspace
bindings while retaining definition and policy assembly.
A run whose model turns end with no content — typically because every tool call was denied by the
generated policy — is not reported as success: `fh run` prints a warning and exits nonzero. The
session always offers the built-in filesystem tools alongside the declared Databricks tools; with
`sandbox: 'empty'` and the generated tool allowlist they are denied, so a model that wanders to
`grep`/`read` produces exactly this signal. Narrow the role/instructions or widen the policy
deliberately rather than ignoring it.
Inspect the resulting session and audit trail:
```sh
npx fh sessions
npx fh inspect
npx fh logs
npx fh approvals
```
After deployment, discover and resolve approvals through the App without granting the caller
operator-console access:
```sh
TOKEN="$(databricks auth token --host "$DATABRICKS_HOST" | jq -r .access_token)"
export DATABRICKS_OAUTH_TOKEN="$TOKEN"
npx fh approvals \
--url "$DATABRICKS_APP_URL/api" \
--token-env DATABRICKS_OAUTH_TOKEN
```
The remote list is tenant-scoped. Use the returned session and approval ids with `fh approve` or
`fh reject` and the same URL and token environment variable.
## 7. Add and test Databricks features
Preview recipes before changing the workspace, then install only the surfaces the agent needs:
```sh
npx fh add databricks sql --dry-run
npx fh add databricks sql
npx fh add databricks ai-search
npx fh add databricks rag-chain
npx fh add databricks lakeflow
npx fh add databricks jobs
npx fh add databricks lakebase
npx fh add databricks system-tables-cost
npx fh add databricks apps
npx fh add databricks agent-services
npm install
npm run build
```
| Capability | Credential-free check | Live proof |
| --- | --- | --- |
| AI Gateway/model | `fh run ... --mock` | A run without `--mock` returns a model response. |
| SQL + Unity Catalog | Definition and policy load | Allowed metadata/query succeeds; denied catalog access remains denied. |
| AI Search/RAG | Recipe build and deterministic mock tests | Known document is retrieved with citations; insufficient context is declined. |
| Lakeflow + Jobs | Recipe build and request-contract tests | Create/read/cancel against a test job or pipeline. |
| Lakebase | Persistence bundle build | Session survives App process replacement and credential refresh. |
| Cost + lineage | Event schema tests | Tenant/principal tags appear in MLflow/system-table evidence. |
| Human approval | Local approval contract | Pause, App redeploy, approve, and resume without losing state. |
| OBO | Identity contract tests | Signed-in user permissions differ from the App service principal as expected. |
| Agent Services | Lifecycle request contracts | Temporary registration is created, discovered, updated, permissioned, and deleted in Unity Catalog. |
Use [Databricks recipes](/docs/databricks/recipes) for required environment variables and [RAG on
Databricks](/docs/databricks/rag) for the MLflow evaluation and adversarial/insufficient-context
workflow. Mocked unit tests and live certification answer different questions; keep both in release
CI.
## 8. Build and deploy a Databricks App
Build and inspect the deployment plan first:
```sh
npx fh build --target databricks-app
npx fh deploy \
--target databricks-app \
--profile fabric-harness \
--preview
```
Then deploy with the same profile:
```sh
npx fh deploy \
--target databricks-app \
--profile fabric-harness
```
The build output is under `.fabricharness/build/databricks-app/`. TechFabric Harness produces the App
server, manifest, `app.yaml`, and Databricks Declarative Automation Bundle configuration. Databricks App resources and
secret references should supply production configuration; do not commit tokens or client secrets.
The output is a portable package: workspace-owned contracts are bundled and the directory can be
handed to Runway or another deployer without the source repository. See
[Portable agent packages](/docs/deployment/portable-packages).
### What developers see in the App
The deployed App should make the selected agent, runtime health, acting identity, region, and bound
Databricks resources inspectable without rendering secret values. This is a representative,
sanitized deployment state; names and resource health come from the active environment.
Continue with the [Databricks App tutorial](/docs/deployment/databricks-app) for App resources,
Lakebase durability, OBO, approvals, restart/redeploy recovery, and protected live certification.
---
# Databricks integrations
Canonical: https://harness.techfabric.com/docs/databricks/integrations
Complete map of TechFabric Harness integrations for Databricks data, AI, orchestration, state, governance, telemetry, and cost.
`@fabric-harness/databricks` exposes composable factories and a `databricks()` bundle. Use the bundle
for the common governed stack; use individual exports when you need a narrower integration.
Resource lifecycle APIs are covered in [Databricks resource authoring](/docs/databricks/authoring).
To scaffold project wiring (managed files, env stubs, dependencies), use **[Databricks recipes](/docs/databricks/recipes)**:
```sh
fh add databricks sql
fh add lakebase
fh add ai-search
fh add lakeflow
fh add agent-services
```
## Data and AI
| Service | TechFabric Harness API | Agent use |
| --- | --- | --- |
| Unity AI Gateway | `bundle.aiGateway`, `databricksFoundationModelProvider()` | Discover and invoke `system.ai.*` model services with submission/tenant request tags |
| Model Serving | `databricksFoundationModelProvider({ mode: 'serving-endpoints' })` | Custom serving-endpoint inference |
| SQL Warehouse | `databricksSqlReadTool()`, `databricksSqlTool()` | SELECT-only analytics reads plus construction-time policy-bound SQL execution |
| Unity Catalog | `unityCatalogTablesTool()`, `databricksTableInfoTool()`, `DatabricksUnityCatalogAdmin` | Discover metadata; opt-in grants/catalog/schema/volume lifecycle |
| Unity Catalog Agent Services | `bundle.agentServices` | Register external agents, discover them, update metadata, manage grants, and delete registrations |
| AI Search | `databricksAiSearch()`, `databricksRagChain()`, `DatabricksAiSearchAdmin` | Retrieval plus opt-in endpoint/index lifecycle |
| RAG quality | `scoreRagTurn()`, MLflow 3 export, managed evaluation Job | Local smoke checks plus a UC evaluation dataset and Databricks managed relevance, groundedness, sufficiency, and correctness judges |
| Embeddings | `databricksEmbeddings()` | Query embeddings through a serving endpoint |
| AI Functions | `databricksAiQueryTool()` | Invoke `ai_query()` through a SQL Warehouse against explicitly allowed endpoints |
| Genie Agents (formerly AI/BI Genie spaces) | `DatabricksGenieClient`, `DatabricksGenieAdmin`, `DatabricksGenieAgentModeClient` | Stable governed conversations, beta normalized lifecycle/ACL management with managed-only model deletion, and explicit Beta Agent Mode SSE streaming |
| Feature Serving | `databricksFeatureLookupTool()` | Low-latency governed feature lookup |
| Workspace files | `bundle.workspaceSource()`, `workspaceWrite` | Read-only context plus opt-in import/mkdir/delete tools |
| Asset Bundles | `bundle.assetBundle`, `assetBundles`, `databricksAssetBundleLifecycle()` | Governed checked-in bundle validate/deploy/run/destroy with source-fingerprint drift detection, no-wait run submission by resource key, and managed-only teardown |
## Managed MCP and MCP Services
Use `databricksWithManagedMcp()` when a bundle consumes Databricks managed MCP servers or a Unity
Catalog MCP Service through Unity AI Gateway. Discovery is asynchronous, so the synchronous
`databricks()` factory rejects a configuration containing `mcp` rather than silently omitting its tools.
```ts
import {
databricksPrincipalFromEnv,
databricksWithManagedMcp,
} from '@fabric-harness/databricks';
const dbx = await databricksWithManagedMcp({
host: process.env.DATABRICKS_HOST!,
principal: databricksPrincipalFromEnv(process.env),
governance: {
catalogs: ['main'],
},
mcp: [
{
name: 'support_genie',
endpoint: { kind: 'genie', spaceId: process.env.DATABRICKS_GENIE_SPACE_ID! },
},
{
name: 'delivery',
endpoint: { kind: 'mcp-service', name: 'main.agents.fabric_runway' },
effects: {
'runway_catalog': 'read',
'runway_request_*': 'write',
},
},
],
});
// Remote names are stable and collision-safe.
console.log(dbx.tools.map((tool) => tool.name));
// mcp__support_genie__ask_genie, mcp__delivery__runway_catalog, ...
try {
// Use dbx.modelProvider, dbx.tools, and dbx.policy with init().
} finally {
await dbx.managedMcp.close();
}
```
The adapter builds the documented workspace URLs for Genie, AI Search, SQL, Unity Catalog functions,
and `/ai-gateway/mcp-services/`. It obtains a fresh bearer token from the bundle's
PAT, OAuth M2M, CLI-profile, or OBO provider on every transport request and refuses to send that token to
another origin. Tokens are absent from tool metadata and redacted from connection and tool errors.
Genie and AI Search are classified read-only. SQL, Unity Catalog functions, registered MCP Services, and
custom workspace URLs must supply `effects` globs or `defaultEffect`; otherwise discovery fails closed.
Write/execute tools receive a static governed MCP resource, lineage, catalog policy, and approval binding
through the same `withGovernanceTools()` path as native tools. Request-scoped Apps must call
`forPrincipalWithManagedMcp()` and close the returned bundle after the request; the synchronous
`forPrincipal()` method rejects principal reuse for remote connections.
Authentication requires a workspace bearer credential accepted by the selected managed server. A
registered MCP Service also needs `EXECUTE` on the service and `USE CATALOG` / `USE SCHEMA` on its
parents. Missing Preview enrollment, OAuth authorization, UC privilege, effect classification, or
approval is returned as an error before the underlying mutation can execute.
Dynamic agents can mount the same governed connection conditionally. The connector is lazy: token
resolution and tool discovery occur only when the current render declares it, and the runtime closes
the connection after the interaction.
```ts
import { connectDatabricksManagedMcpServer } from '@fabric-harness/databricks';
import {
createAgent,
defineMcpConnection,
useMcpConnection,
usePersistentState,
} from '@fabric-harness/sdk';
const server = {
name: 'catalog',
endpoint: { kind: 'functions', catalog: 'main', schema: 'agent_tools' },
defaultEffect: 'execute',
} as const;
const catalogTools = defineMcpConnection({
name: server.name,
connect: () => connectDatabricksManagedMcpServer({
host: process.env.DATABRICKS_HOST!,
tokenProvider: async () => process.env.DATABRICKS_TOKEN,
server,
}),
});
export default createAgent(() => {
const [approved] = usePersistentState('catalogApproved', false);
if (approved) useMcpConnection(catalogTools);
return approved
? 'Use the governed catalog functions when needed.'
: 'Complete approval before using catalog functions.';
});
```
For an employee-facing Databricks App, construct the definition with a request-scoped OBO token
provider. For shared automation, use a least-privilege OAuth M2M provider. Raw workspace credentials
must never enter persistent state or initial data.
## Inference names and URLs
Use the workspace **origin** for `DATABRICKS_HOST`, not an API path. Fabric selects the inference
base from the model name:
| Model value | Mode | OpenAI-compatible base URL |
| --- | --- | --- |
| `system.ai.gpt-oss-20b` (or another discovered `system.ai.*` service) | Unity AI Gateway | `${DATABRICKS_HOST}/ai-gateway/mlflow/v1` |
| A custom endpoint name such as `support-agent-prod` | Custom Model Serving | `${DATABRICKS_HOST}/serving-endpoints` |
`DATABRICKS_INFERENCE_MODE=auto` is the default. Set `ai-gateway` or `serving-endpoints` only when
overriding automatic routing. Use `DATABRICKS_AI_GATEWAY_BASE_URL` for a proxy or explicitly
configured Gateway base. Discover services enabled in the current workspace with
`await bundle.aiGateway.listModelServices()` instead of assuming every `system.ai.*` service is available.
## Engineering and operations
| Service | TechFabric Harness API | Agent use |
| --- | --- | --- |
| Jobs | `databricksRunJobTool()`, `DatabricksJobsAuthoring` | Run existing Jobs within a required `runPolicy` allowlist, or opt into policy-bounded Jobs 2.2 lifecycle |
| Notebooks | `databricksNotebookTool()` | Submit a one-time notebook run within a required `notebookPolicy` allowlist |
| Lakeflow | `databricksLakeflowTools()`, `DatabricksLakeflowAuthoring` | Operate existing pipelines; separately opt into create/update/delete/events |
| MLflow runs | metric and parameter tools | Write run metadata |
| MLflow tracing | `bundle.mlflowTraceExporter()` | Export settled agent spans to MLflow Tracing |
| Serving usage | `servingUsageCapture()` | Associate inference-table usage with submissions |
| System tables | `databricksConsumption()` | Aggregate billable usage for tenants and agents |
| Actual-cost budgets | `databricksActualCostSource()`, `databricksTenantCostLimit()` | Reconcile policy budgets against usage records |
| Lakebase | `lakebaseClient()`, `databricksPersistence()` | Sessions, submissions, conversation streams, and telemetry |
| UC Volumes | `UcVolumesAttachmentStore`, Volume source/writer | Governed attachments and non-tabular context |
## Reuse from vertical applications
Application repositories must not create their own Databricks client package. Compose a principal
with `databricksPrincipalFromEnv()`, use `databricksSdk()` for generated service clients, and use the
bundle only for reviewed SDK gaps such as AI Gateway and SQL warehouse discovery.
For libraries that require `fetch`, `createDatabricksAuthenticatedFetch()` supplies a rotating
workspace credential and bounds authorization retry to one 401/403 response. For SQL,
`runStatement()` uses the generated Statement Execution client with named parameters and bounded
polling. Provider-service and warehouse listings are available from `bundle.aiGateway` and
`bundle.sqlWarehouses`; raw protocol transport remains private to Harness.
## Add a standalone tool
```ts
import {
databricksSdk,
databricksFeatureLookupTool,
withGovernance,
} from '@fabric-harness/databricks';
const principal = {
kind: 'service-principal',
host: process.env.DATABRICKS_HOST!,
clientId: process.env.DATABRICKS_CLIENT_ID!,
clientSecret: process.env.DATABRICKS_CLIENT_SECRET!,
} as const;
const sdk = databricksSdk({
host: process.env.DATABRICKS_HOST!,
principal,
});
const customerFeatures = withGovernance(
databricksFeatureLookupTool(sdk.modelServingQuery, {
endpoint: 'customer-features',
name: 'lookup_customer_features',
}),
{
principal: `sp:${process.env.DATABRICKS_CLIENT_ID}`,
onLineage: (record) => auditSink.write(record),
},
);
```
## Integration boundaries
- TechFabric Harness does not replace Unity Catalog permissions or Databricks resource ACLs.
- Managed MCP and Unity AI Gateway are Databricks Preview/Beta surfaces; run the protected live
certification in every target workspace before release.
- The SQL tool and SQL sandbox execute statements; they do not provide a Linux shell.
- Workspace files and Unity Catalog Volumes are separate APIs and path spaces.
- `databricks-serving` is a proxy target, while `databricks-app` hosts the Node runtime.
- Availability and API behavior can vary by cloud, region, workspace feature enablement, and preview
status. Validate each enabled integration in the target workspace.
---
# Databricks resource management
Canonical: https://harness.techfabric.com/docs/databricks/authoring
Author governed Genie agents, Jobs, Lakeflow pipelines, Model Serving and Unity Catalog resources from an agent, with approvals and managed-only deletion.
TechFabric Harness lets an agent (or a user driving one) **manage** the Databricks resources it
operates — create a Job, define a pipeline, build an AI Search index, size a serving endpoint,
create a Genie Agent, grant access — through the same governed, typed seams the consumption tools
already use.
Resource management is **opt-in per surface** and **fails closed**: a consumption-only `databricks()` bundle
keeps its existing tools, and any model-exposed write surface refuses to initialize unless approval
routing is configured. Stable services run through Databricks' official modular TypeScript SDK.
Fabric owns the agent layer: approval binding, policy, lineage, durable retries, managed-resource
fingerprints, and cleanup. Protocols absent from the SDK stay in a small, explicit raw protocol
adapter. This does not replace Databricks Asset Bundles, Terraform, or checked-in pipeline source:
Harness never *generates* that IaC. But the lifecycle of a checked-in bundle — validate, deploy,
run, destroy — is itself a governed agent surface; see [Checked-in Asset Bundles](#checked-in-asset-bundles).
The docs use **resource management** as the developer-facing term. “Authoring” remains in a few
flag and certification names where it distinguishes create/update/delete from consumption; it does
not mean a second infrastructure-as-code system.
## Quickstart: author your first Job
Enable one surface, bound its compute, and route writes to an approval audience:
```ts
import { init } from '@fabric-harness/sdk';
import { databricks } from '@fabric-harness/databricks';
const dbx = databricks({
host: process.env.DATABRICKS_HOST!,
principal: {
kind: 'service-principal',
host: process.env.DATABRICKS_HOST!,
clientId: process.env.DATABRICKS_CLIENT_ID!,
clientSecret: process.env.DATABRICKS_CLIENT_SECRET!,
},
jobAuthoring: true,
computePolicy: {
maxConcurrentRuns: 1,
maxRunTimeoutSeconds: 1800,
requiredTags: { 'cost-center': 'agents' },
// schedules stay disabled unless you opt in with `schedules: true`
},
governance: {
stewardAudience: 'data-platform',
catalogs: ['main'],
onLineage: (record) => audit(record),
},
});
const fabric = await init({ modelProvider: dbx.modelProvider, tools: dbx.tools, policy: dbx.policy });
```
What happens when the model calls `databricks_create_job`:
1. The call routes to the `data-platform` approval audience. Execution does not start without a
grant bound to this exact call (see [approvals](#how-approval-and-governance-work)).
2. The job spec is validated against `computePolicy` before any API call — schedules off, timeout
and concurrency capped, required tags present.
3. The created job is stamped with the `fabric-harness:managed` tag and a canonical spec
fingerprint, so later updates and deletes can prove ownership.
4. The operation is recorded in lineage with the executing principal and every approver.
For local development only, `allowUnapprovedAuthoring: true` bypasses the fail-closed approval
requirement. It is the single escape hatch and should never be set in a deployed app.
Runnable end-to-end versions of this flow live in the
[authoring examples](#examples-and-failure-behavior).
## How approval and governance work
An approval **grant** binds the logical tool-call id, a canonical digest of the input, the
executing principal, the approver identities, and an expiry. A durable retry may replay the same
operation, but a grant can never be reused for different input or a different identity. Grant TTL
starts when the approval is decided, not when it was requested.
The `governance` block controls routing and audit:
| Field | Effect |
| --- | --- |
| `stewardAudience` | Required for any model-exposed authoring surface. Write/execute tools route approval requests to this audience. |
| `approvalServices` | Optional narrowing: only write/execute tools whose `metadata.service` appears here require approval. Excluded services remain usable without a grant, which supports separate dev and production tools. Omit it to gate every write/execute tool. |
| `approvalTtlSeconds` | Grant time-to-live, measured from the approval decision. |
| `catalogs` | Belt-and-suspenders allowlist checked against every catalog-qualified resource in structured tool inputs. Unity Catalog remains the authoritative enforcement boundary. |
| `principalLabel` | Label recorded in lineage for the acting principal (never a token). Defaults to a label derived from the configured principal. |
| `onLineage` | Sink for audit/lineage records; wire to OTel or your event pipeline. |
Read tools remain approval-free when an admin surface is enabled. The typed clients
(`DatabricksJobsAuthoring`, `DatabricksGenieAdmin`, …) can always be used without model approval
policy — "the SDK may author" is deliberately separate from "a model may autonomously author".
Native Databricks/Unity Catalog authorization is authoritative in either path.
Narrow by service only when that service boundary is explicit in the tool inventory. For example,
use separate `deploy_dev` (`service: "dev"`) and `deploy_prod` (`service: "prod"`) tools, then set
`approvalServices: ["prod"]`. The governance wrapper and approval policy use the same selection,
so the dev tool runs without a grant while the production tool fails closed until approved.
Genie conversations, lifecycle management, and Agent Mode are configured together under `genie`.
Supplying `genie.manage` is the explicit write opt-in; no separate authoring boolean is required.
## Request-scoped user identity
Use a principal returned by forwarded-token validation/current-user inspection, never a label
supplied by the request:
```ts
const inspection = await inspectDatabricksAppUserAuthorization({ headers, host });
const user = dbx.forPrincipal({ tokenProvider: forwardedTokenProvider, principal: inspection.principal });
```
The scoped bundle has isolated generated SDK/model/embedding/credential clients and does not mutate the
parent. Persistence wiring and policy/lineage sinks remain app-scoped. Approval records and lineage
distinguish the executing principal from all approvers.
## Native SDK boundary
`databricks()` exposes the official principal-bound clients at `bundle.sdk`. Application code can
also create them directly:
```ts
import { databricksSdk, databricksJobs } from '@fabric-harness/databricks';
const sdk = databricksSdk({ host, principal });
const jobs = databricksJobs(sdk.jobs);
```
Generated clients own stable Jobs, Statement Execution, Lakeflow, Vector Search, Model Serving,
Files, Secrets, Genie, Access Management, Unity Catalog, MLflow Experiments, and Lakebase
request/response models. Fabric's private raw protocol transport has an exhaustive method-and-path
allowlist for endpoints that the modular SDK does not yet expose. Anything else is denied before
credentials are resolved or network I/O starts. Source-boundary and runtime-denial tests enforce that quarantine. Every generated module
is exact-pinned to SDK `0.21.0`, and CI verifies the package pins, runtime constant, compatibility
page, README, generated serialization, and packaged import together.
Genie Agent permissions use the generated `@databricks/sdk-accessmanagement` client; they are not a
raw-protocol exception. The only stable-API fallbacks are the exact Genie query-result and AI Search floating-score decoder mismatches
documented on the compatibility page.
## Resource-management surfaces
Each surface is an independent opt-in. Enable only what the agent needs:
| Flag | Typed client | Model tools and constraints |
| --- | --- | --- |
| `jobAuthoring` | `DatabricksJobsAuthoring` | CRUD/list/repair; Jobs API 2.2; classic compute requires an allowed cluster-policy id |
| `oneOffCompute` | `submitRun()` | **Separate flag** — `jobAuthoring` alone does not expose `databricks_submit_run`; policy-bounded multi-task one-off runs |
| `lakeflowAuthoring` | `DatabricksLakeflowAuthoring` | Pipeline create/update/delete; `lakeflow: true` is read-only and `lakeflow.runPolicy` adds bounded start/stop |
| `lakeflowEvents` | `pipelineEvents()` | Requires `lakeflowAuthoring` and `warehouseId`; queries `event_log('')` |
| `genie.manage` | `DatabricksGenieAdmin` | Normalized version-2 create/get/list/update/export/import plus ACLs; the object itself is the explicit write opt-in |
| `genie.manage.delete` | `databricks_delete_genie_agent` | Must be `managed-only`; exact Harness-managed id and expected fingerprint only |
| `genie.agentMode` | `DatabricksGenieAgentModeClient` | Explicit Beta SSE stream, deadlines, lineage callbacks, pagination, and optional bounded model tool |
| `aiSearchAdmin` | `DatabricksAiSearchAdmin` | Endpoint/index create, describe, sync, wait, and delete; delta-sync and direct-vector unions. Takes a **required policy**, not a boolean: `allowedOperations` decides which tools register, and endpoint/index/embedding/source-table bounds are pinned into the schemas and re-checked at call time. `{ allowAnyAiSearchAdmin: true }` is the explicit unbounded opt-out |
| `servingAdmin` | `DatabricksServingAdmin` | Custom/foundation/external/agent endpoint variants, config updates, and separate AI Gateway updates |
| `ucAdmin` | `DatabricksUnityCatalogAdmin` | Grants plus catalog/schema/volume creation |
| `ucDestructiveAdmin` | `deleteManaged()` tools | Requires `ucAdmin`; exact identifiers only; managed resources only; no force/cascade |
| `workspaceWrite` | Governed bundle tools | Notebook import, mkdirs, and exact-path deletion |
| `secretsWrite` | `DatabricksSecretsAuthoring` | Requires `secretProvider`; scope lifecycle and `SecretRef` writes; built-in tools never accept raw values |
| `assetBundles` | `DatabricksAssetBundleLifecycle` (or one per name) | Checked-in Asset Bundle validate/deploy/run/destroy through the Databricks CLI; source-fingerprint drift detection; managed-only destroy; a name-keyed map adds a required `bundle` selector |
A bundle with **every** surface enabled looks like this — treat it as a reference for field names,
not a starting point:
```ts
const dbx = databricks({
host,
principal,
jobAuthoring: true,
oneOffCompute: true,
computePolicy: {
allowedPolicyIds: ['policy-id'],
allowedExistingClusterIds: ['explicit-shared-cluster-id'],
maxWorkers: 2,
maxConcurrentRuns: 1,
maxRunTimeoutSeconds: 1800,
allowedNodeTypes: ['Standard_D4ds_v5'],
allowedRuntimeVersions: ['15.4.x-scala2.12'],
serverlessPerformance: 'STANDARD',
schedules: false,
requiredTags: { 'cost-center': 'agents' },
},
lakeflowAuthoring: true,
lakeflowEvents: true,
assetBundles: { bundleDir: '.', target: 'dev' },
genie: {
manage: {
resourceStore: managedResourceStore,
delete: 'managed-only',
},
},
aiSearchAdmin: {
allowedOperations: ['createIndex', 'syncIndex', 'describeIndex'],
allowedIndexes: ['main.kb.docs_idx'],
allowedEndpoints: ['kb-search'],
allowedEmbeddingModelEndpoints: ['databricks-bge-large-en'],
allowedSourceTables: ['main.kb.docs'],
},
servingAdmin: true,
ucAdmin: true,
ucDestructiveAdmin: true,
workspaceWrite: true,
secretsWrite: true,
secretProvider,
warehouseId,
governance: {
stewardAudience: 'data-platform',
approvalTtlSeconds: 900,
catalogs: ['main'],
onLineage: audit,
},
});
```
## Compute policy reference
`computePolicy` bounds every Jobs-authoring and one-off-run spec **before** any API call:
| Field | Enforcement |
| --- | --- |
| `allowedPolicyIds` | Classic VM compute requires a non-empty list and a matching `policyId` on every new cluster. Serverless jobs take the serverless branch and bypass cluster policy. |
| `allowedExistingClusterIds` | Existing all-purpose clusters are denied unless their exact id is listed. |
| `maxWorkers` | Caps `numWorkers` on every new cluster. |
| `maxConcurrentRuns` | Caps the job's `maxConcurrentRuns` (default 1). |
| `maxRunTimeoutSeconds` | Caps `timeoutSeconds`. |
| `allowedNodeTypes` | When set, every new cluster's `nodeTypeId` must be listed. |
| `allowedRuntimeVersions` | When set, every new cluster's `sparkVersion` must be listed. |
| `requiredTags` | Every new cluster must carry exactly these `custom_tags`; empty keys or values are rejected at configuration time. |
| `schedules` | Schedules are rejected unless explicitly `true`. |
| `serverlessPerformance` | When set (`'STANDARD'` or `'PERFORMANCE_OPTIMIZED'`), a spec that names a different serverless performance target is rejected. |
Allowlist fields must be non-empty when configured; numeric caps must be non-negative. Violations
fail at bundle initialization or before the write executes — never after.
`computePolicy` does **not** bound execution of an *existing* job. `databricksRunJobTool` and
`databricksNotebookTool` carry their own required `runPolicy` / `notebookPolicy`, enforced on the run
path itself. See [Bound what a model can run](/docs/databricks/compute).
## Managed resources and safe deletion
Destructive tools only touch resources TechFabric Harness can **prove** it created. Because Databricks
services expose different metadata capabilities, three marker schemes exist:
- **Jobs** — created jobs carry the tags `fabric-harness:managed=true` and
`fabric-harness:fingerprint=`. `ifExists: 'reuse'` adopts only a single job
matching name, managed tag, and fingerprint; anything ambiguous is refused. Updates refuse to
remove the ownership tags and re-stamp the fingerprint.
- **Unity Catalog** — created catalogs and schemas carry the property
`fabric-harness.managed="true"`; volumes (which have no properties) carry a comment prefixed
`[fabric-harness:managed]`. `deleteManaged()` reads the object first and refuses to delete
anything not carrying its marker. No force or cascade options exist.
- **Genie Agents** — the Genie management API supports neither tags nor properties, so ownership
lives in a durable `DatabricksManagedResourceStore` manifest recording fingerprint, creator, and
tool-call provenance. Deletion requires the exact managed id and expected fingerprint; an
already-trashed managed id is treated as an idempotent success.
`MemoryDatabricksManagedResourceStore` is only for tests and local development; use
`LakebaseDatabricksManagedResourceStore` or another durable implementation in production.
- **Asset Bundles** — bundle ownership lives in the same `DatabricksManagedResourceStore` manifest
(`resourceType: "databricks-bundle"`), recording a sha256 fingerprint of the sorted bundle source
tree (excluding `.databricks/` CLI state, `node_modules/`, and `dist/`). Deploy records the
fingerprint after a successful CLI run; destroy requires the record and refuses to tear down a
drifted tree unless `force` is set. See [Checked-in Asset Bundles](#checked-in-asset-bundles).
A create whose API response is ambiguous is never retried automatically, so a transient failure
cannot orphan an unmarked resource. Schedules default off. Serverless jobs take the serverless
branch; classic VM job compute requires a non-empty `allowedPolicyIds` and a matching `policyId`
on every new cluster. Existing all-purpose clusters are denied unless their exact id appears in
`allowedExistingClusterIds`.
## Checked-in Asset Bundles
A job/pipeline bundle authored in YAML — by hand, by `databricks bundle init`, or by exporting a
Lakeflow Designer canvas as a `.designer.ipynb` referenced from a `notebook_task` — stays
checked-in IaC. What Harness governs is its **lifecycle**. The `assetBundles` option points the
bundle at a directory containing `databricks.yml` and adds four model tools plus a typed client at
`bundle.assetBundle`:
```ts
const dbx = databricks({
host,
principal,
assetBundles: { bundleDir: './bundle', target: 'dev' },
governance: { stewardAudience: 'data-platform' },
});
```
| Tool | Effect | Behavior |
| --- | --- | --- |
| `databricks_bundle_validate` | read | Runs `databricks bundle validate` in `bundleDir`; no workspace mutation. |
| `databricks_bundle_deploy` | write | Validates, deploys, then records the source fingerprint in the managed-resource store. Optional `expectedFingerprint` input fails with `DatabricksManagedResourceConflictError` when the recorded fingerprint differs — optimistic concurrency so one agent cannot stomp a bundle recorded from another tree. |
| `databricks_bundle_run` | execute | Submits a run of a bundle-defined job/pipeline by its `databricks.yml` resource key. Always `--no-wait` — waiting inside a tool call would violate finite-agent boundedness. The result includes `resourceKey` plus a job `runId`, or the Lakeflow `pipelineId`/`updateId`, and `runUrl` when supplied by the CLI, so the bounded jobs/lakeflow status tools compose without parsing text. No managed record is required: bundles deployed outside Harness (e.g. by CI) stay runnable, and the CLI fails cleanly against an undeployed bundle. |
| `databricks_bundle_destroy` | write | Requires the managed record (`DatabricksUnmanagedResourceError` otherwise), refuses a drifted tree unless `force: true`, runs `databricks bundle destroy --auto-approve`, and deletes the record. |
The fingerprint is a deterministic sha256 over the sorted bundle source tree (relative path plus
content), excluding `.databricks/`, `node_modules/`, and `dist/`. The lifecycle shells out to the
Databricks CLI (`executable` defaults to `databricks`, so it must be on `PATH`) and authenticates
through the CLI's normal environment or `--profile` resolution; `target`, `profile`, and `env` pass
through unchanged. CLI failures surface with a bounded stderr tail and are never retried
automatically.
### Several bundles in one agent
Passing a name-keyed map instead of one entry configures several bundles — a separate lifecycle per
name, exposed as `bundle.assetBundles`:
```ts
const dbx = databricks({
host,
principal,
assetBundles: {
jobs: { bundleDir: './bundles/jobs', target: 'dev' },
pipelines: { bundleDir: './bundles/pipelines', target: 'dev' },
},
governance: { stewardAudience: 'data-platform' },
});
await dbx.assetBundles?.jobs?.validate();
```
The tool names stay the same four. Every one of their input schemas gains a required `bundle`
selector enumerating the configured names (`jobs`, `pipelines`), so a call must always say which
bundle it means — including when only one bundle is configured, which keeps the schema and its
approval digests stable if a second bundle is added later. Names must match
`^[A-Za-z0-9][A-Za-z0-9_-]*$`, and an empty map is rejected. Governed lineage names only the bundle
the call selected, never the whole configured set, because the descriptor is a `/bundle` input path
rather than a static value.
An approval grant binds the tool-call id, the executing principal, and the canonical tool **input**.
`bundle` is part of that input, so a grant approved for `{ bundle: 'jobs' }` cannot be replayed
against `{ bundle: 'pipelines' }` — distinct names cannot cross-approve. A name's `target`, however,
is configuration rather than input: retargeting `jobs` from `dev` to `prod` does **not** invalidate
outstanding grants for `jobs`, so a durable approved call replayed after that change would hit the
new target. When dev and prod need separate approvals, model them as separate named entries
(`jobs_dev`, `jobs_prod`) instead of editing one name's target.
Managed-resource identity is scoped per name and target —
`#[#]`, where the `#` suffix is
present only when that entry configures a `target` — so several lifecycles can share one durable
`DatabricksManagedResourceStore` without overwriting each other's fingerprint records. The
single-bundle shape keeps the bare manifest name, so existing records stay addressable.
One limitation: approval routing is per **tool name**, so every named bundle routes to the same
`governance.stewardAudience`. Per-bundle audiences are not expressible — if `jobs` and `pipelines`
need different approvers, build two agents.
Like every authoring surface, `assetBundles` fails closed: without `governance.stewardAudience`
(or the local-only `allowUnapprovedAuthoring: true`), bundle initialization throws instead of
exposing ungated deploy/run/destroy tools. Application code can also drive the same lifecycle directly
through `databricksAssetBundleLifecycle()` / `databricksAssetBundleTools()` without a model. See
`examples/with-databricks-bundle-deploy` for a runnable steward-gated flow and
[`fh add databricks bundle`](/docs/databricks/recipes) for the scaffold.
## Genie Agents
Databricks renamed Genie spaces to **Genie Agents**, while the stable management and Conversation
API paths retain `/genie/spaces`. Fabric consistently calls the resource identifier `agentId`.
The stable consumption path uses the typed `DatabricksGenieClient`. It starts or continues a
conversation, requires a terminal `COMPLETED` state, retrieves every query result from its
attachment-specific endpoint, and returns typed text, query, result, suggested-question, and
visualization metadata. Missing ids, terminal failure, polling exhaustion, cancellation, oversized
responses, and malformed attachments fail explicitly. Application code can also list conversation
history and message comments or delete an exact conversation id; only ask is model-facing by
default.
Genie lifecycle management is **stable for the certified create/query/update/trash scope**:
```ts
const dbx = databricks({
host,
principal,
genie: {
manage: {
resourceStore: new LakebaseDatabricksManagedResourceStore({ client: lakebase }),
delete: 'managed-only',
},
},
governance: { stewardAudience: 'data-platform', catalogs: ['main'] },
});
const salesAnalyst = await dbx.genieAdmin?.create({
version: 2,
title: 'Sales analyst',
parentPath: '/Shared/agents',
warehouseId: 'warehouse-id',
dataSources: [{ table: 'main.sales.orders' }],
});
```
`DatabricksGenieAdmin` provides list/get/create/update/delete, export/import, and permissions APIs.
The serializer emits Databricks `serialized_space` version 2, deterministic 32-character ids, and
the required sorted collections. Updates compare the caller's expected canonical fingerprint and
send the last observed Databricks ETag. The management API has no general field mask: `description`
is the only field that supports explicit clearing via `remove`; other fields can only be set to new
values. Ambiguous creates, updates, and deletes are not retried. Raw `serialized_space` is accepted
only by typed-client import/export; no built-in ToolDef accepts it.
Model-facing create, inspect, update, and optional delete tools use a normalized schema containing
tables, columns, sample questions, and one text instruction. SQL examples, functions, joins,
filters, expressions, measures, and benchmarks become available only when `genie.manage.sqlPolicy` is
configured. The policy runs server-side for every SQL fragment, must return the complete parsed
`catalog.schema.object` set, and denies any resource outside `dataSources`. An allowlisted prepared
statement is also valid; returning an empty or partial parse for dynamic SQL is not. Mutation
requires steward approval and a durable ownership record.
```ts
const certifiedSql = new Map([
['SELECT COUNT(*) FROM main.sales.orders', ['main.sales.orders']],
]);
const dbx = databricks({
// ...identity, ownership store, governance...
genie: {
manage: {
resourceStore: managedResourceStore,
sqlPolicy: ({ sql }) => {
const resources = certifiedSql.get(sql);
if (!resources) throw new Error('SQL is not in the certified statement set');
return { resources };
},
},
},
});
```
Fabric's ordinary Genie client uses the Conversation API. The separately configured Agent Mode
client supports Databricks' Beta SSE endpoint:
```ts
const dbx = databricks({
host,
principal,
genie: {
agentMode: {
agentId: '0123456789abcdef0123456789abcdef',
acknowledgeBeta: true,
timeoutMs: 5 * 60_000,
idleTimeoutMs: 2 * 60_000,
modelTool: { maxOutputBytes: 512 * 1024 },
onEvent: (event) => auditAgentModeProgress(event),
},
},
});
for await (const event of dbx.genieAgentMode!.respond('Summarize revenue by region')) {
if (event.type === 'response.completed') console.log(event.response.output);
}
```
It validates the preview's 32-hex agent id, never retries the response-creating POST, propagates
cancellation, enforces idle and overall deadlines, parses arbitrarily chunked SSE, preserves unknown
Beta events, enforces monotonic sequence numbers and event/stream limits, and requires
`response.completed` or `response.failed`. `modelTool` adds the governed
`databricks_genie_agent_mode` tool; progress goes only to `onEvent`, while the model receives one
bounded terminal response.
`listConversationItems()` exposes the cursor-based history endpoint. A failed terminal event is
yielded and then raises `DatabricksGenieAgentModeResponseError` on the next iterator step.
The workspace must be enrolled by the Databricks account team and a workspace admin must enable
**Genie Agents Agent Mode API** in Previews. Agent Mode allows one in-flight response per
conversation and can keep an SSE connection open for up to 90 minutes. Fabric does not silently
substitute it for ordinary Genie conversations. See the
[Genie Agents API](https://docs.databricks.com/aws/en/genie-agents/conversation-api) and
[Beta Agent Mode API](https://docs.databricks.com/aws/en/genie-agents/api) for the native contracts.
### Bind a Genie Agent to a Databricks App
```ts title=".fabricharness/config.ts"
export default {
target: 'databricks-app',
databricks: {
app: {
genie: {
agentId: '0123456789abcdef0123456789abcdef',
// permission defaults to CAN_RUN
},
},
},
};
```
The generated bundle attaches a `genie_space` resource and injects
`DATABRICKS_GENIE_AGENT_ID` through `valueFrom`; it does not embed the id in model context.
`CAN_EDIT` and `CAN_MANAGE` are rejected unless the deployment block also sets `authoring: true`.
### Genie Ontology export
`planModuleOntologyExport` is the first piece of the Genie Ontology exporter: it consumes a
governed Fabric module manifest — the JSON-serializable `ModuleManifest` /
`ModuleManifestBundle` contract from `@fabricorg/platform/manifest`, consumed type-only — and
produces a deterministic, fingerprinted `DatabricksOntologyExportPlan` describing how the
module's declared ontology maps onto Unity Catalog Semantics (metric views, glossary terms,
domain assignments) and a Genie Agent spec:
```ts
import { planModuleOntologyExport } from '@fabric-harness/databricks';
const plan = await planModuleOntologyExport({
manifest, // ModuleManifest or ModuleManifestBundle produced by the vertical's Platform process
catalog: 'main',
schema: 'lending_semantics',
// domain defaults to the manifest namespace; genie/feed options are optional
});
```
The plan step is pure — no network, no clock — so its output is safe to render for review and
diffs byte-identically across runs for an unchanged manifest; `plan.fingerprint` changes when the
manifest or plan inputs change. The manifest is an export source only: UC Metric Views remain the
runtime semantic source of truth, export is one-directional (Platform → UC/Genie), and Genie stays
read-only over Fabric-governed domains.
`plan.policyClassifications` records the class (`threshold-ratio-metric`, `gate-compliance-rule`,
or `code-evaluator`) plus the matched evidence for every manifest policy, and `plan.metricViewDdl`
carries one `CREATE OR REPLACE VIEW ... WITH METRICS LANGUAGE YAML` statement per module
(`_metrics`), tagged with the `fabric-harness.managed` marker and a manifest-fingerprint
comment so drift is detectable by inspection and re-export is byte-identical. Threshold/ratio data
policies become metric-view measures; compliance gates (FCRA/TCPA-style) become glossary business
rules and never `MEASURE(...)` — a code policy becomes a metric seed only through the explicit
`metricSeedPolicyIds` opt-in. Measures from countable action volumes arrive with the
operational-truth feed tables in a later release.
When the `feed` option is present, `plan.feedTables` carries the operational-truth feed: three
Delta/UC tables per module — `._asset_events` (domain events with
`schema_version`, tenant, correlation, and causation), `_action_invocations` (action id, version,
terminal status, failure classification, duration, and redacted parameters only), and
`_policy_evaluations` (decision, reasons, evaluator identity/version, and failure mode). The
generated DDL is `CREATE TABLE IF NOT EXISTS ... USING DELTA` with `'delta.appendOnly' = 'true'`,
the managed tag, and a feed-fingerprint comment. Each table is replay-safe by construction: the
hydration pipeline merges on the Platform id column (`spec.replayKey` — `event_id`,
`invocation_id`, `evaluation_id`) with `MERGE ... WHEN NOT MATCHED THEN INSERT`, so re-hydrating
recorded history is idempotent. Tenant scoping is a `tenant_id` column enforced by UC grants per
consumer, never filtered at export, and the feed reads durable history records only — it is
orthogonal to the vertical's host execution runtime. Events land at their declared
`schema_version`; Harness performs no upcasting at append time (the conservative resolution of
design question Q3 — a vertical that wants a current-shape stream upcasts in its own producer).
`applyModuleOntologyExport` executes the feed DDL through the same approval-gated statement seam
as the metric views, ahead of them (the metric view sources the asset-events feed), with one
provenance record per table.
Every feed table spec carries an explicit `publishableFields` allowlist, and row materialization
projects each record onto exactly that list before Delta append — anything not allowlisted is
dropped, so PII/PHI never reaches Genie's training surface. **Redaction ownership stays with the
vertical**: Harness defines no redaction policy of its own. Feed rows are materialized only
through the caller's redacted-record producer (the fabric-autorefi `redactActionParameters`
pattern); materialization fails closed without it, so raw Platform records can never be appended
unredacted. Harness's role is limited to the schema contracts, the publish-time allowlist check,
and absence tests proving raw parameters and PII-bearing fields never land. Retention for the
append-only feed is defined in `docs/databricks-genie-ontology-feed-retention.md`: a 90-day hot
retention window by default (per-table configurable, keyed to each table's recorded domain time),
no Harness-side archive by default, and a scheduled expiry job that lands with the first dogfood
deployment — re-hydration through `planOntologyFeedHydration` can always restore expired rows from
Platform history.
`planOntologyFeedHydration` is the replay-safe append pipeline (design §3.7). It takes the plan's
feed table specs, recorded Platform history grouped by kind (`assetEvents`, `actionInvocations`,
`policyEvaluations`), and the required redacted-record producer, and returns merge-ready rows plus
one deterministic `MERGE INTO ... USING (VALUES ...) WHEN NOT MATCHED THEN INSERT` statement per
table, keyed by the spec's replay key. Rows are redacted, allowlist-projected, deduplicated on the
replay key within the batch (first occurrence wins, drops counted), and sorted by replay key, so
identical history yields a byte-identical plan and re-applying a statement inserts no duplicate
rows — the idempotent re-run contract. The pipeline is pure (no network, no credentials): the
scheduled feed pipeline executes the statements through `runStatement` with a
`DatabricksStatementClient` constructed for the per-tenant workspace service principal (§5), while
tenant scoping stays a `tenant_id` column enforced by UC grants. For asset events,
`assetEventRecordFromEnvelope` maps the shipped `AssetEventEnvelope` shape from
`@fabricorg/platform/events` (consumed type-only, exactly like the manifest) onto the feed record
fields; Platform ships no durable ActionInvocation/PolicyEvaluation record contract, so the
minimal `DatabricksOntologyActionInvocationRecord` / `DatabricksOntologyPolicyEvaluationRecord`
types define what the vertical's history reader hands over. The runnable
`examples/with-databricks-ontology-export` workspace demonstrates the full story credential-free —
plan, hydration with a demo redaction producer, idempotent re-run, and a PII-absence check — with
exact expected output in its README.
`plan.glossaryTerms` and `plan.domainAssignments` carry the Unity Catalog glossary upsert payloads:
one term per object type (`displayName` names the term; `idPrefix` and the module `idPrefixes`
registry are preserved as metadata for identifier parsing), subject-vocabulary terms, one term per
event type with `schemaVersion` metadata, lifecycle terms for actions with `mutatesDomain: true`
(read-side phrasing only), business-rule terms referencing `policyId@policyVersion` for
gate/compliance and code policies (a `dataDefinition` summary for data policies; code-evaluator
logic is Platform-internal and never exported), and one term per state-machine state label. Every
term carries `manifestVersion` metadata. Domain assignments bind one UC domain per module
namespace — the `domain` option overrides the default — with the module description as the domain
comment. Adapter steps, code-policy bodies, and `requiredRoles` / `requiredPermissions` /
`eventPhase` are deliberately excluded. These payloads are plan data until apply; the governed
publish path goes through `applyModuleOntologyExport` (see below), and the glossary/domain UC REST
surface maturity is still being tracked while Genie Ontology is in public preview — apply accepts
an injected `DatabricksOntologyGlossaryPublisher` so the publish client can evolve without
changing the plan contract.
When the `genie` option is present, `plan.genieSpec` carries a fully derived
`DatabricksGenieAgentSpecV2` — the same spec shape the Genie authoring tools accept, normalized and
fingerprinted through the identical canonicalization `DatabricksGenieAdmin` applies. Each module in
the bundle contributes a `kind: "metric-view"` data source (`.._metrics`),
plus its declared feed tables when `feed` options are present; threshold/ratio policies become
`sqlSnippets.measures` reusing the exact metric-view measure names and expressions; object-type
`displayName`s and state labels land as column synonyms; bundle `dependencies` become joins on the
shared `object_type` key (only when both manifests are in the bundle); sample questions come from
action and state vocabulary with read-side phrasing only; and the single permitted instruction
carries the namespace description, the event lifecycle vocabulary, and the read-only constraint —
Genie never mutates Fabric-governed domains. Action parameter names are treated as untrusted text:
they appear only as read-side slot hints in sample questions, and only after passing a denylist
filter (`password`, `secret`, `token`, `ssn`, …) plus the vertical's own
`genie.sensitiveFieldPatterns`. `plan.genieSpecFingerprint` is the canonical spec fingerprint the
governed apply path will use as `expectedFingerprint`, so a plan-time diff keys off the same
conflict contract as the shipped authoring path:
```ts
const plan = await planModuleOntologyExport({
manifest: bundle, // ModuleManifestBundle — joins derive from cross-module dependencies
catalog: 'main',
schema: 'vertical_semantics',
genie: {
agentId: existingAgentId, // omit to create; used for the conflict-check fingerprint
admin: genieAdmin, // existing fingerprinted authoring client (apply only; wiring, not an input)
parentPath: '/Workspace/Genie',
warehouseId: 'wh-1',
// sensitiveFieldPatterns: ['drivers_license'], // vertical-declared additions to the denylist
},
});
// plan.genieSpec — reviewable, deterministic DatabricksGenieAgentSpecV2
// plan.genieSpecFingerprint — desired post-apply fingerprint for no-op/conflict detection
```
`applyModuleOntologyExport` is the governed write side of the plan/apply split. It takes the
plan plus already-constructed clients (it never resolves credentials — interactive export should
build them with OBO identity so UC enforces the curator's own grants) and applies the artifacts in
dependency order: feed tables → metric views → glossary/domains → Genie Agent:
```ts
import { applyModuleOntologyExport } from '@fabric-harness/databricks';
const result = await applyModuleOntologyExport(plan, {
resourceStore, // shared managed-resource provenance store
principal: 'user:curator@example.com', // exporting principal label, never a token
toolCallId, // stable correlation id for this apply invocation
statements, // statement-execution client (metric-view DDL seam)
warehouseId: 'wh-1',
approval: async (request) => {
// Required. Called once per write, before it happens: feed-table DDL,
// metric-view DDL, domain assignment, glossary publish, Genie Agent
// create/update — each
// with effect: 'write', the governed resource, and the content fingerprint.
// Route to the steward approval flow; throw to deny.
},
sqlPolicy, // required when the derived spec carries SQL fragments (fail-closed)
genie: { admin: genieAdmin, agentId: existingAgentId }, // admin must share resourceStore
glossary, // optional DatabricksOntologyGlossaryPublisher
});
```
Every authoring mutation is approval-gated: the `approval` callback fires per write with
`effect: "write"` and is the caller's hook into the same steward approval flow that gates the
authoring tools — apply never approves itself. Genie spec changes go through
`DatabricksGenieAdmin.update` compare-and-swapped against the fingerprint recorded at last apply
in the managed-resource store: a live fingerprint that drifted (another author's edit) surfaces
the typed `DatabricksManagedResourceConflictError` — no blind overwrite, no silent retry. Updates
also require the agent to be recorded as Harness-managed, exactly like the model-facing authoring
tool. SQL fragments in the derived spec (measure snippets, join SQL) are admitted only through
`sqlPolicy`, and apply throws before any write when the spec carries fragments but no policy was
supplied. Exporter-generated metric-view and feed-table DDL is deterministic plan output; it
executes through `runStatement` with approval rather than a raw fetch.
On success the managed-resource store records provenance for every artifact — one record per
metric view (`ontology-metric-view/`), per glossary domain
(`ontology-glossary-domain/`), the Genie Agent ownership record written by
`DatabricksGenieAdmin` itself, and an append-only export record keyed by the plan fingerprint
(`ontology-export/`) — each carrying the exporting principal, `toolCallId`,
timestamps, and the `fabric-harness.managed` tag. A changed manifest yields a new plan fingerprint
and therefore a new export record; historical records are never rewritten.
Re-export against an unchanged manifest produces an empty diff: when every recorded fingerprint
matches the plan and the managed Genie Agent still matches live, apply performs no writes and
returns `result.noChanges: true` with the per-artifact `applied: false` markers. The result also
carries the applied artifacts (view names, agent id, glossary term counts, domain assignments),
provenance record ids, and post-apply fingerprints. Apply is not transactional across artifacts —
a conflict on a later artifact leaves earlier ones applied and recorded — but every artifact is
independently idempotent, so re-running apply converges.
#### Drift detection and steward attestation
Genie Ontology's knowledge store *infers* expressions, synonyms, concepts, and experts; the
manifest *declares* semantics. They will diverge, and the divergence is surfaced as evidence —
never applied back. `compareOntologyDrift` is the pure declared-vs-inferred comparison: it takes
the export plan's declared side (glossary terms and measure-seed policy classifications) plus the
knowledge store's inferred concepts as plain data, and classifies every finding:
| Finding kind | Severity | Meaning |
| --- | --- | --- |
| `inferred-synonym-vs-declared-term` | `analytics-only` (or `conflict`) | A learned synonym mapped onto a declared glossary term. |
| `inferred-expression-vs-declared-measure` | `analytics-only` (or `conflict`) | A learned expression mapped onto a declared measure. |
| `inferred-concept-without-declared-counterpart` | `informational` | An inference (including every learned expert) with no declared counterpart. |
| `declared-definition-without-inferred-counterpart` | `informational` | A declared term or measure no inference has observed. |
Severity encodes the conflict policy: the manifest is authoritative for mutation semantics, so an
inference whose label collides with a *different* declared definition than its target is a
`conflict` routed to steward review; a tolerated inference mapped onto its declared target is
`analytics-only`; an exact restatement of the declared name is agreement and produces no finding.
The comparison is deterministic — no network, no clock — and runs credential-free against plain
fixtures. The knowledge-store read goes through the injected
`DatabricksGenieKnowledgeStoreReader` interface (one read method returning plain data), so the
preview/unsettled knowledge-store APIs (design question Q1) stay behind the caller's adapter and a
preview-API change never reshapes the module.
`attestOntologyDrift` wraps the comparison into the steward-surfacing path: it reads inferred
concepts through the injected reader, compares, and appends exactly one attestation record per run
to the caller-supplied `DatabricksOntologyDriftEvidenceSink`:
```ts
import { attestOntologyDrift } from '@fabric-harness/databricks';
const record = await attestOntologyDrift(plan, {
stewardAudience: 'lending-stewards', // required — routes the record to steward review
knowledgeStore, // injected read-only client (design Q1 preview caveat)
evidence, // append-only audit sink
agentId: existingAgentId, // optional Genie Agent scope for the read
});
// record.attestation — ExecutionAttestation via databricksExecutionAttestation
// record.comparison — the classified findings
```
The record's attestation is built through the same `databricksExecutionAttestation` helper the
Platform bridge uses, with `runId` (a content fingerprint of plan + findings + audience) as the
external operation id and the steward audience in its metadata. Records are append-only audit
evidence: each run appends, historical records are never rewritten. **Nothing auto-applies back** —
the drift module exposes no apply path, the knowledge-store interface is read-only by construction,
and the only side effect is the single evidence append. A fingerprint mismatch on the Genie side
still blocks silent re-export through the existing `DatabricksManagedResourceConflictError`
contract on the next `applyModuleOntologyExport`; nothing flows back into the module without a
human changing governed code.
Current status: the `genie-ontology-export` capability is registered at **beta** in the
capability registry while Genie Ontology is in public preview. Plan, apply, and drift carry
credential-free contract coverage; retained protected-workspace apply evidence is still pending
and is the remaining P0 exit item. Promotion to stable follows the same rule as
`genie-authoring`: retained protected-workspace evidence on non-preview APIs.
## Serving and AI Search constraints
`aiSearchAdmin` takes a resource policy, not a boolean. `allowedOperations` decides which of the six
tools are registered at all — an agent that only needs to sync an index never sees
`databricks_delete_search_endpoint`:
| Policy field | Enforcement |
| --- | --- |
| `allowedOperations` | Non-empty list of `createEndpoint`, `deleteEndpoint`, `createIndex`, `deleteIndex`, `syncIndex`, `describeIndex`. Only the listed tools are constructed. |
| `allowedIndexes` | Exact index-name membership. Required when any registered operation names an index. |
| `allowedEndpoints` | Exact endpoint-name membership. Required when any registered operation names an endpoint — `createIndex` names both dimensions. |
| `allowedEmbeddingModelEndpoints` | Exact serving-endpoint membership for `embedding.modelEndpoint` on delta-sync index creation. The source column's contents are sent to this endpoint, so it is a separate dimension from the Vector Search endpoint and the two never substitute for each other. **Omitting both this and `allowAnyEmbeddingModelEndpoint` removes delta-sync creation entirely** — the branch is dropped from the tool schema and a delta-sync spec is rejected at runtime. A direct-vector-only agent therefore needs no opt-out arm; reaching for `allowAnyEmbeddingModelEndpoint` to satisfy the type would instead grant delta-sync against any embedding endpoint. |
| `allowedSourceTables` | Optional exact `sourceTable` membership for delta-sync index creation. When present, names are pinned and re-checked like the other resources. When omitted, the governance catalog allowlist and the executing principal's Unity Catalog grants remain the bound. It is invalid when the embedding arm is omitted because that policy withholds the delta-sync branch entirely. |
| `allowAnyIndex` / `allowAnyEndpoint` / `allowAnyEmbeddingModelEndpoint` | Per-dimension explicit opt-outs. Mutually exclusive with the matching allowlist. |
| `allowAnyAiSearchAdmin` | Whole-surface explicit opt-out: all six tools, unpinned. |
Names are pinned into the tool `inputSchema` (`const` for one, `enum` for several) and re-checked at
call time; an out-of-policy name throws `DatabricksResourceNotAllowedError` before the Vector Search
API is reached.
AI Search direct-access creation requires a full schema and vector column dimension. The
model-facing serving tools reject `environmentVars`; applications that deliberately need raw
values must use the typed client so those values never enter model context. See the native
[Serving Endpoints API](https://docs.databricks.com/api/workspace/servingendpoints),
[AI Search Indexes API](https://docs.databricks.com/api/workspace/vectorsearchindexes/createindex),
and [Unity Catalog Grants API](https://docs.databricks.com/api/workspace/grants/update).
## Secrets and lineage
The `databricks_put_secret` input is `{ scope, key, secretRef: { kind: 'secret', name } }`. The
configured `SecretProvider` resolves material during server-side execution and passes it directly to
Databricks. The value never appears in model context, tool input, or lineage. The raw-value method
exists only on the typed client and is an application-level trust decision.
Structured governance metadata uses extended JSON Pointer paths, including `*` for arrays. Bundle
initialization validates every authoring descriptor against its input schema. At execution, a
required zero-match fails closed, every catalog-qualified resource is allowlist-checked, and every
resource is recorded in lineage.
## Examples and failure behavior
- `examples/with-databricks-jobs-authoring`: policy-bound multi-task Job lifecycle.
- `examples/with-databricks-bundle-deploy`: governed checked-in Asset Bundle validate/deploy/run/destroy with fingerprint drift detection.
- `examples/with-databricks-dataeng`: Lakeflow operation, authoring, and event expectations.
- `examples/with-databricks-rag-admin`: AI Search endpoint/index lifecycle and query verification.
- `examples/with-databricks-authoring-admin`: serving, non-destructive UC, workspace, and secret-reference writes.
- `examples/with-databricks-genie-authoring`: normalized Genie Agent lifecycle with a durable Lakebase ownership manifest.
- `examples/with-databricks-ontology-export`: credential-free Genie Ontology export plan, feed hydration, idempotent re-run, and PII-absence check.
All examples document exact credentials and expected output in their README. Missing approval
routing fails at bundle initialization. Missing warehouse/secret provider dependencies, governance
descriptor errors, grant/principal mismatches, catalog denial, native Databricks authorization,
timeouts, ambiguous creates, and cleanup leaks fail explicitly.
## Stability
Jobs, Lakeflow, AI Search administration, non-preview custom-model serving, managed-only UC
administration, workspace writes, secret-reference writes, and Genie Agent management (for its
create/query/update/trash scope) are **stable** for their documented scope, backed by retained
protected-workspace lifecycle evidence. Jobs evidence covers serverless STANDARD and policy-bound
classic definitions; AI Search evidence covers direct-vector and Delta Sync lifecycles.
Provisioned Throughput, AI Gateway administration, Agent
Mode streaming, and Databricks App OBO authentication retain their separate Databricks preview
constraints. The full release gate, destructive certification lifecycle, and retained evidence are
documented in
[Authoring certification and release evidence](/docs/databricks/authoring-certification).
## Upgrade note
Approvals persisted before grant provenance existed cannot authorize a post-upgrade tool call. An
idempotent session detects the missing grant and requests approval again under a new provenance id.
This is intentionally fail-closed; operators may see one new approval after upgrading a durable
session.
Pair compute limits with `databricksTenantCostLimit()` and System Tables reconciliation when
authoring Jobs or serving resources. Compute lifecycle outside governed Jobs remains deferred until
budget guardrails can be enforced server-side.
---
# Databricks compute patterns
Canonical: https://harness.techfabric.com/docs/databricks/compute
Choose SQL Warehouses, Lakeflow Jobs and notebooks, Databricks Apps, or an isolated sandbox without conflating their execution models.
Databricks offers several execution surfaces. TechFabric Harness keeps them explicit so policy, identity,
idempotency, and output handling match the workload.
```mermaid
flowchart TD
Q{What must execute?}
Q -->|Governed SQL| SQL[SQL Warehouse]
Q -->|Existing workflow| JOB[Lakeflow Job]
Q -->|One-off notebook| NB[Notebook task]
Q -->|Fabric HTTP runtime| APP[Databricks App]
Q -->|General shell or untrusted code| SB[External isolated sandbox]
SQL --> UC[Unity Catalog permissions and lineage]
JOB --> VOL[Logs and output envelope in UC Volumes]
NB --> VOL
APP --> LB[Lakebase durable state]
SB --> NET[Container, cluster, or provider egress boundary]
classDef decision fill:#fff4d6,stroke:#d97706,color:#451a03
classDef compute fill:#e8f0fe,stroke:#2563eb,color:#172554
classDef governed fill:#dcfce7,stroke:#16a34a,color:#052e16
class Q decision
class SQL,JOB,NB,APP,SB compute
class UC,VOL,LB,NET governed
```
| Workload | Fabric API | Durable result | Policy effect |
| --- | --- | --- | --- |
| SQL statement | `databricksSqlReadTool()`, policy-bound `databricksSqlTool()`, or `databricksSqlSandbox()` | Statement id/result | Read-only by default; bind and gate broader execution |
| Existing Job | `databricksJobs().runJob()` | Typed run receipt and state | `execute`; bounded by `runPolicy.allowedJobIds`; use delivery id as idempotency token |
| Notebook | `databricksJobs().submitNotebook()` | Typed run receipt and task output | `execute`; bounded by `notebookPolicy`; gate data mutations |
| Agent hosting | `fh build --target databricks-app` | Lakebase session/submission stream | HTTP runtime, not a shell |
| General code | Docker, Kubernetes, E2B, Daytona, Modal, or another sandbox | Provider-specific snapshot/ref | Enforce filesystem, process, and network boundaries |
## Run and monitor a Job
```ts
import { databricksJobs, databricksSdk } from '@fabric-harness/databricks';
const sdk = databricksSdk({
host: process.env.DATABRICKS_HOST!,
principal: { kind: 'pat', token: process.env.DATABRICKS_TOKEN! },
});
const jobId = Number(process.env.DATABRICKS_JOB_ID);
const jobs = databricksJobs(sdk.jobs, {
runPolicy: { allowedJobIds: [jobId] },
notebookPolicy: { allowedNotebookPathPrefixes: ['/Workspace/Shared/fabric'] },
});
const receipt = await jobs.runJob({
jobId,
idempotencyToken: submission.id,
notebookParams: { customer: tenant.id },
});
const state = await jobs.wait(receipt.runId, {
timeoutMs: 15 * 60_000,
signal: abortController.signal,
});
if (state.resultState !== 'SUCCESS') {
throw new Error(`Job ${receipt.runId} ended in ${state.resultState}`);
}
```
The caller-supplied idempotency token survives HTTP retries and upstream message redelivery. Calling
`cancel(runId)` is safe to retry. `wait()` returns typed lifecycle/result state and throws
`DatabricksRunTimeoutError` rather than returning an ambiguous running result after its deadline.
## Bound what a model can run
Every model-callable execute/write factory takes a **required** resource policy. There is no
unbounded default: omitting it is a compile error, runtime checks run before the Databricks API, and
opting out uses a deliberate, greppable `allowAny...: true` arm.
```ts
import { databricksNotebookTool, databricksRunJobTool } from '@fabric-harness/databricks';
const tools = [
databricksRunJobTool(sdk.jobs, { allowedJobIds: [Number(process.env.DATABRICKS_JOB_ID)] }),
databricksNotebookTool(sdk.jobs, {
allowedNotebookPathPrefixes: ['/Workspace/Shared/fabric'],
}),
];
```
```ts
const sql = databricksSqlTool(
sdk.statements,
{ allowedStatements: ['CALL main.ops.refresh_daily()'] },
{ warehouseId },
);
const ai = databricksAiQueryTool(
sdk.statements,
{ allowedEndpoints: ['support-classifier'] },
{ warehouseId },
);
const pipeline = databricksPipelineStartTool(sdk.pipelines, {
allowedPipelineIds: [pipelineId],
});
const metric = databricksMlflowLogMetricTool(sdk.experiments, {
allowedRunIds: [runId],
});
```
Pinned input schemas guide the model, but runtime enforcement is the security boundary. An
out-of-policy SQL statement, endpoint, pipeline id, or MLflow run id never reaches the generated
client.
The Jobs and notebook model tools use the Harness tool-call id as their default Databricks
idempotency token. Harness guarantees that id is unique per logical call even when a model provider
omits an id, while preserving it across the call's approval, lineage, execution, and result records.
Typed `databricksJobs()` callers should continue to supply a stable delivery or submission id when
they need retry coalescing across process boundaries.
| Policy field | Enforcement |
| --- | --- |
| `runPolicy.allowedJobIds` | Exact numeric membership. A non-empty array of non-negative integers, checked at construction. |
| `runPolicy.allowAnyJobId` | Explicit opt-out. Mutually exclusive with `allowedJobIds`. |
| `notebookPolicy.allowedNotebookPaths` | Exact workspace path membership. |
| `notebookPolicy.allowedNotebookPathPrefixes` | Subtree match on segment boundaries only, so `/Workspace/prod` admits `/Workspace/prod/ingest` and rejects `/Workspace/production-evil`. |
| `notebookPolicy.allowAnyNotebookPath` | Explicit opt-out. Mutually exclusive with both allowlists. |
Enforcement lives in `databricksJobs()`, **before** `run-now` or `runs/submit` is called: an
out-of-policy target throws `DatabricksRunNotAllowedError` and never reaches the workspace. Any
`notebookPath` containing a `..` segment is rejected while a policy is configured. The tool's
`inputSchema` is also pinned (`const` for one id, `enum` for several) — that is a model-facing hint,
not the boundary; the runtime check is.
The boundary is **job ids, not job names**. The run client is deliberately narrow (`runNow`,
`submitRun`, `getRun`, `cancelRun`, `getRunOutput`) and cannot resolve names, and resolving a name at
call time would be a time-of-check/time-of-use hole. Resolve names to ids yourself at configuration
time; a helper built on the authoring client may return `number[]` in a future release.
Note that `computePolicy` does **not** bound this path. It applies to Jobs authoring and one-off
compute specs, not to triggering an existing job.
## Submit a notebook
```ts
const receipt = await jobs.submitNotebook({
notebookPath: '/Workspace/Shared/fabric/daily-report',
existingClusterId: process.env.DATABRICKS_CLUSTER_ID!,
idempotencyToken: submission.id,
baseParameters: { report_date: '2026-07-09' },
});
```
Submission is bounded the same way as `runJob`: the `jobs` client above already carries a
`notebookPolicy`, so an out-of-policy `notebookPath` throws `DatabricksRunNotAllowedError` before
`runs/submit` is called.
Notebook submission is asynchronous Jobs compute. It does not turn the SQL sandbox into a shell and
does not execute TypeScript inside Model Serving.
## Persist logs and outputs in a UC Volume
```ts
import { UcVolumesAttachmentStore } from '@fabric-harness/databricks';
const store = new UcVolumesAttachmentStore({
client: sdk.files,
catalog: 'main',
schema: 'agents',
volume: 'fabric_attachments',
rootPrefix: 'job-output',
});
const ref = await jobs.exportOutput(receipt.runId, store, `submission:${submission.id}`);
```
For multi-task Jobs, `getOutputs()` retrieves each task run output. `exportOutput()` stores one
content-addressed JSON envelope with notebook result, driver logs, truncation state, errors, and raw
metadata. Unity Catalog controls access to the Volume.
The [runnable compute example](/docs/reference/source-access)
uses a deterministic mock by default and switches to a real workspace when credentials are present.
---
# Databricks connectors and sandboxes
Canonical: https://harness.techfabric.com/docs/databricks/sandboxes-connectors
Choose between the Databricks SQL sandbox, Unity Catalog Volume connectors, workspace sources, attachment storage, and general-purpose compute sandboxes.
Databricks appears in both the sandbox and connector layers, but those layers solve different
problems. Select them by capability rather than by provider name.
```mermaid
flowchart TD
NEED{What must the agent do?}
NEED -->|Run governed SQL| SQL[Databricks SQL sandbox or SQL tool]
NEED -->|Read or write governed files| VOL[Unity Catalog Volume connector]
NEED -->|Read workspace source files| WS[Workspace Files source]
NEED -->|Persist user attachments| ATT[UC Volumes attachment store]
NEED -->|Run bash, packages, or arbitrary code| GEN[Docker or remote code sandbox]
SQL --> WH[SQL Warehouse]
VOL --> FILES[Files API]
WS --> WAPI[Workspace API]
ATT --> FILES
GEN --> DATA[Call Databricks tools over governed APIs]
classDef question fill:#fef3c7,stroke:#d97706,color:#422006
classDef fabric fill:#dbeafe,stroke:#2563eb,color:#172554
classDef dbx fill:#dcfce7,stroke:#16a34a,color:#052e16
class NEED question
class SQL,VOL,WS,ATT,GEN fabric
class WH,FILES,WAPI,DATA dbx
```
## SQL sandbox
`databricksSqlSandbox()` adapts `session.shell(sql)` to SQL Statement Execution on a SQL Warehouse.
It returns JSONL or CSV in `stdout`. Its small filesystem is in-memory session storage; it is not
DBFS, a Volume, a cluster driver, or an operating-system shell.
```ts
import { databricksSqlSandbox } from '@fabric-harness/databricks/sql-sandbox';
import { init } from '@fabric-harness/sdk';
const runtime = await init({
sandbox: databricksSqlSandbox({
host: process.env.DATABRICKS_HOST!,
principal: { kind: 'on-behalf-of', userToken: () => workspaceIdentity() },
warehouseId: process.env.DATABRICKS_WAREHOUSE_ID!,
catalog: 'main',
schema: 'analytics',
resultFormat: 'jsonl',
}),
});
const session = await runtime.session();
const result = await session.shell(`
SELECT region, sum(net_revenue) AS revenue
FROM orders
GROUP BY region
ORDER BY revenue DESC
`);
```
The SDK backend name is also usable after registering the Databricks provider adapter:
```ts
import { registerDatabricksSqlSandboxBackend } from '@fabric-harness/databricks';
import { init } from '@fabric-harness/sdk';
registerDatabricksSqlSandboxBackend();
const runtime = await init({ sandbox: 'databricks' });
```
The factory reads host, Warehouse, and identity from the sandbox creation environment. A portable
`databricks-sql` ref stores only host, Warehouse, catalog, schema, result format, and timeout; PAT,
OAuth secret, OBO token, generated SDK client, and token provider are deliberately excluded. Attach
resolves credentials again in the receiving process, so a rotated credential can be used without
rewriting the ref. Missing host, Warehouse, or attach-time identity fails closed.
Use `databricksSqlReadTool()` for model-facing analytics reads. It advertises a `read` effect and
fails locally unless input is one `SELECT` (a SELECT-ending CTE is accepted); mutations,
administration keywords, malformed SQL, and multiple statements never reach Statement Execution.
```ts
import {
analyticsCopilotGovernance,
databricksSdk,
databricksSqlReadTool,
} from '@fabric-harness/databricks';
const sdk = databricksSdk({ host, principal });
const sqlRead = databricksSqlReadTool(sdk.statements, {
warehouseId: process.env.DATABRICKS_WAREHOUSE_ID!,
});
const governance = analyticsCopilotGovernance({
stewardAudience: 'analytics-stewards',
catalogs: ['main'],
});
```
The analytics-copilot pack scopes approval routing to the SQL and Genie services. `sql_read` and
ordinary `databricks_genie_ask` calls are reads and stay interactive; policy-bound
`databricksSqlTool()` execution and Genie lifecycle writes still route to the steward audience.
Enabling a different authoring service fails bundle initialization until approval routing covers it.
This Harness check is defense in depth: Unity Catalog privileges under the acting service principal
or OBO user remain authoritative.
Use `databricksSqlTool(client, executionPolicy, options)` only when the model deliberately needs
more than SELECT-only access. Bind exact statements or supply a server-side validator; the
`allowAnyStatement: true` arm is an explicit advanced opt-out. Approval routing remains a separate
defense-in-depth layer. The sandbox is useful when the session's shell abstraction should itself
mean SQL.
See the runnable
[`with-analytics-copilot`](/docs/reference/source-access)
example for service-principal and OBO authentication, expected output, and failure behavior.
## Unity Catalog Volume connector
The connector uses the Databricks Files API and keeps all paths under a configured
`/Volumes///` root.
```ts
import { databricksVolumeSource } from '@fabric-harness/connectors/databricks-volume';
const source = databricksVolumeSource({
host: process.env.DATABRICKS_HOST!,
token: async () => workspaceIdentity(),
volumePath: '/Volumes/main/support/knowledge',
include: (path) => path.endsWith('.md') || path.endsWith('.pdf'),
});
await session.mount('/knowledge', source);
```
`databricksVolumeWriter()` provides scoped `put()` and `delete()` operations. Use
`UcVolumesAttachmentStore` when attachments must participate in Fabric's attachment lifecycle.
The acting principal still needs the relevant `USE CATALOG`, `USE SCHEMA`, and Volume privileges.
## General-purpose code execution
For Python, package installation, bash, repository mutation, or untrusted code, use a code sandbox
such as Docker, Kubernetes, E2B, Daytona, Modal, or another remote backend. Give that sandbox
Databricks tools or scoped OAuth access as needed. This preserves the common sandbox interface
without pretending a SQL Warehouse is a machine shell.
See the focused [Databricks SQL sandbox reference](/docs/ecosystem/sandboxes/databricks-sql) and the
[sandbox matrix](/docs/reference/sandboxes-matrix).
---
# Enterprise Databricks controls
Canonical: https://harness.techfabric.com/docs/databricks/enterprise
Identity propagation, Unity Catalog enforcement, approvals, audit lineage, durable state, cost controls, secret handling, and deployment hardening.
TechFabric Harness adds runtime controls around Databricks calls. Those controls are defense in depth:
Unity Catalog and Databricks resource permissions remain authoritative.
## Identity modes
| Mode | Use | Fabric API |
| --- | --- | --- |
| App service principal | Databricks Apps and unattended production workloads | `appServicePrincipalFromEnv()` |
| OAuth M2M | External services acting as one application | `databricksIdentity({ kind: 'service-principal' })` |
| On-behalf-of user | Preserve the signed-in user's grants | `onBehalfOfFromHeaders()` |
| Personal access token | Local development or controlled single-user testing | `kind: 'pat'` |
Token providers are resolved at call time and refresh before expiry. Identity labels, not tokens, are
placed in actors, submissions, lineage, and cost records.
For Databricks Apps ingress, use `databricksAppsOidcAuthenticator()` as the Node server
`authenticate` hook. It validates workspace JWT signatures, issuer, audience, expiry, and signing-key
rotation before mapping the forwarded user email to both the Fabric actor and Unity Catalog
principal. See [Authentication and RBAC](/docs/operating/auth#databricks-apps).
When Databricks Apps forwards an OBO access token instead of an OIDC identity token, use the narrow
Apps authorization entrypoint. It validates the token against the workspace current-user API,
returns only safe identity and lifetime metadata, assigns an opaque tenant per user, and caches by a
token digest bounded by token expiry:
```ts
import { createDatabricksAppUserAuthenticator } from
'@fabric-harness/databricks/app-user-authorization';
import { startDevServer } from '@fabric-harness/node';
const authenticate = createDatabricksAppUserAuthenticator({
host: process.env.DATABRICKS_HOST!,
});
await startDevServer({ authenticate });
```
Missing forwarded authorization falls through so another configured authenticator may run. Invalid
or unauthorized forwarded tokens fail closed. The raw token is never returned, logged, or used as a
cache key. Applications that need authenticated user identity but do not pass an OBO token to
downstream Databricks APIs can opt into `trustForwardedUserIdentity`. It maps the integrity-protected
`x-forwarded-user`, `x-forwarded-email`, and `x-forwarded-preferred-username` ingress headers to a
natural-person principal. Generated Databricks App bundles enable this trust mode because they are
reachable only through Apps ingress; a forwarded email or preferred username is required so an M2M
caller with only `x-forwarded-user` is not reclassified as a person.
`trustForwardedUserIdentity`, `trustForwardedServicePrincipal`,
`appPrincipalId`, and `appAuthorizedCallerIds` are explicit ingress-trust controls; enable them only
when the server is exclusively reachable through Databricks Apps. See the runnable
`examples/with-databricks-simple/app-auth.ts` path for deterministic output and failure behavior.
## Enforcement flow
```mermaid
flowchart LR
R[Tool request] --> A[Definition and invocation policy]
A --> E{Allowed?}
E -->|No| DENY[Reject and audit]
E -->|Yes| H{Approval required?}
H -->|Yes| WAIT[Durable approval wait]
WAIT --> DEC{Decision}
DEC -->|Deny or expire| DENY
DEC -->|Approve| CALL[Call Databricks]
H -->|No| CALL
CALL --> UC[Unity Catalog and resource ACLs]
UC -->|Denied| DENY
UC -->|Allowed| RESULT[Redacted result]
RESULT --> LINEAGE[Lineage, telemetry, and cost]
classDef decision fill:#fef3c7,stroke:#d97706,color:#422006
classDef allow fill:#dcfce7,stroke:#16a34a,color:#052e16
classDef deny fill:#fee2e2,stroke:#dc2626,color:#450a0a
class E,H,DEC decision
class CALL,UC,RESULT,LINEAGE allow
class DENY deny
```
## Controls to configure
1. Use a least-privilege service principal or a user OBO token for each request.
2. Restrict outbound network policy to the workspace host and any explicitly required services.
3. Add catalog allowlists as defense in depth; do not treat them as a substitute for UC grants.
4. Route write and execute tools to a data-steward approval audience.
5. Persist sessions, submissions, and streams in Lakebase for restart recovery.
6. Send governed tool lineage to an audit sink, MLflow, OpenTelemetry, or Lakebase telemetry tables.
7. Apply estimated and actual-cost tenant budgets, then reconcile against system-table usage.
8. Keep secrets in Databricks App resources, environment-backed secret references, or an external
secret manager. Never put credentials in prompts, tool inputs, or lineage labels.
9. Build with provenance and SBOM output, inspect the v2 manifest, and require an API token on
production artifact routes.
## Lakebase credential exchange
`lakebaseClient()` does not use a workspace OAuth token as the Postgres password. It exchanges the
workspace token for a database credential using the endpoint resource name, refreshes that credential
early with jitter, single-flights concurrent refreshes, and supplies it to the connection pool.
```ts
const app = databricksApp();
const server = await startDevServer({
host: '0.0.0.0',
port: Number(process.env.DATABRICKS_APP_PORT ?? 8080),
...(await app.serverOptions()),
});
```
`serverOptions()` injects the Lakebase session, submission, and conversation-stream stores into the
shared Node server. See [Lakebase](/docs/ecosystem/databases/lakebase) for configuration.
### Schema ownership in automated deployments
A production deployment usually has two database principals:
- the **release principal** used by CI to deploy resources and apply schema changes;
- the **App principal** used by the running Databricks App to read and write durable state.
Keep one release principal as the stable owner of the Harness tables. Apply new migrations with that
principal before rolling out an App version, then grant the App principal only the DML and sequence
permissions it needs. Do not transfer table ownership to every new deployment identity.
```sql
-- Run as the stable schema owner. Quote UUID-shaped service-principal names.
GRANT USAGE ON SCHEMA public TO "";
GRANT SELECT, INSERT, UPDATE, DELETE
ON TABLE fh_submission_telemetry, fh_lineage
TO "";
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public
TO "";
```
At startup, Fabric resolves existing telemetry tables through PostgreSQL's relation catalog. This is
important because the App principal's first `search_path` schema can differ from the schema that owns
the shared tables. Fabric skips owner-only `ALTER TABLE` and `CREATE INDEX` statements when the
resolved objects already exist. If a required object is genuinely absent, migration still fails
rather than pretending the schema is current.
Submission telemetry sinks are failure-isolated: an MLflow or Lakebase observer error is logged but
cannot terminate the App process or change an agent result. Treat missing telemetry as an operations
failure anyway. The protected certification gate must prove that lifecycle events and lineage rows
were actually written before promoting the release.
---
# Workspace compatibility
Canonical: https://harness.techfabric.com/docs/databricks/compatibility
Evidence-based Databricks cloud, region, authentication, API, App, Lakebase, and runtime compatibility.
TechFabric Harness targets Databricks workspaces on AWS, Azure, and Google Cloud through the official
modular TypeScript SDK for stable services. Availability of Apps, Lakebase, serverless Model
Serving, private connectivity, and individual
AI services varies by workspace, region, and account configuration. A successful check in one
workspace is recorded only for that cloud and region.
## Runtime and API contract
Every generated client is exact-pinned to the same reviewed [Databricks modular JavaScript SDK](https://docs.databricks.com/aws/en/dev-tools/sdk-javascript)
release. Fabric does not mix modular SDK versions.
| Surface | Transport owner | Databricks API contract |
| --- | --- | --- |
| Jobs | `@databricks/sdk-jobs` `0.21.0` | Jobs 2.2 |
| Lakeflow Declarative Pipelines | `@databricks/sdk-pipelines` `0.21.0` | Pipelines 2.0 |
| SQL Statement Execution | `@databricks/sdk-statementexecution` `0.21.0` | Statement Execution 2.0 |
| AI Search | `@databricks/sdk-vectorsearch` `0.21.0` | Vector Search 2.0 |
| Model Serving and query | `@databricks/sdk-modelserving` and `@databricks/sdk-modelservingquery` `0.21.0` | Serving 2.0 |
| Genie stable lifecycle and conversations | `@databricks/sdk-genie` `0.21.0` | Genie 2.0 |
| Workspace object permissions, including Genie ACLs | `@databricks/sdk-accessmanagement` `0.21.0` | Permissions 2.0 |
| Unity Catalog catalogs, schemas, tables, volumes, and grants | `@databricks/sdk-uc-*` `0.21.0` | Unity Catalog 2.1 |
| Workspace files and volumes | `@databricks/sdk-files` and `@databricks/sdk-uc-volumes` `0.21.0` | Files 2.0 / Unity Catalog 2.1 |
| Workspace secrets | `@databricks/sdk-secrets` `0.21.0` | Secrets 2.0 |
| Workspace identity | `@databricks/sdk-scim` `0.21.0` | SCIM 2.0 preview path |
| MLflow experiments | `@databricks/sdk-experiments` `0.21.0` | MLflow 2.0 |
| Lakebase | `@databricks/sdk-postgres` `0.21.0` | Postgres/Lakebase generated contract plus credential exchange 2.0 |
| Databricks Apps | Databricks CLI project deployment and Apps runtime contract | Apps 2.0 |
| Internal SDK-gap protocols | Private deny-by-default Fabric adapter | Agent Services 2.1 Beta, Genie Agent Mode Beta, Workspace object import/export 2.0, MLflow Traces 3.0, AI Gateway v2, ResponsesAgent custom schema, and exact Genie/AI Search decoder fallbacks |
| Application native escape hatch | `databricksWorkspaceApi()` | Any workspace-relative `/api/*` or `/serving-endpoints/*` API under the configured identity |
| Runtime | Supported version |
| --- | --- |
| MLflow ResponsesAgent | `>=3.10`; protected Azure gate currently exercises `3.14.0` |
| Node.js | `>=22.0.0` |
| `@fabric-harness/databricks` | `>=7.0.0 <8` |
| `pg` for Lakebase | `^8.11.0` |
`DATABRICKS_SDK_VERSION` is read from the installed official SDK. `DATABRICKS_API_VERSIONS` uses
that same value in certification evidence, and a release test requires every `@databricks/sdk-*`
dependency, this page, and the package README to agree. `DATABRICKS_AUTH_MODES` and
`DATABRICKS_PACKAGE_COMPATIBILITY` keep authentication and runtime checks on the same contract.
## Authentication modes
| Mode | Use |
| --- | --- |
| OAuth M2M | External services and protected CI service principals |
| App service principal | Runtime identity injected into a Databricks App |
| On-behalf-of | Per-user Unity Catalog grant enforcement from an App request |
| PAT | Developer smoke tests and transitional automation |
Production deployments should use OAuth or App identity. The official SDK credential chain resolves
CLI profiles and OAuth M2M credentials. Tokens remain in credential providers and request headers;
they are not written to prompts, lineage, compatibility records, or certification errors.
## Native SDK boundary
Generated clients own stable request/response serialization for Jobs, Statement Execution,
Lakeflow, Vector Search, Model Serving, Files, Secrets, Genie, Access Management, Unity Catalog,
MLflow Experiments, and Lakebase. TechFabric Harness adds agent-specific approval binding, structured resource policy,
lineage, managed-resource ownership, bounded retries, and durable cleanup.
Fabric's raw protocol transport is private to protocols not represented by the modular SDK: Agent
Services, Genie Agent Mode streaming, Workspace object import/export, MLflow 3 traces,
AI Gateway/model-service discovery, and the ResponsesAgent custom request schema. It has an
exhaustive method-and-path allowlist; an unlisted route is rejected before credentials are resolved or
network I/O begins. Callers receive typed capabilities through the Databricks bundle and never
construct the raw transport. SCIM identity, authoring certification verification, and all other stable
services use generated clients. Source-boundary and runtime-denial tests prevent stable modules from
silently adding raw API paths. The exact SDK pin, generated serialization tests, packaged-runtime
import test, and protected workspace lifecycle checks all gate a release. Generated types reduce wire
drift; they do not replace live behavioral certification.
Application code is not limited to the services with a generated client or a Harness adapter.
`databricksWorkspaceApi()` is the public, credential-safe native REST escape hatch. It preserves
arbitrary JSON or raw request bodies, raw responses, cancellation, rotating identity, credential
redaction, and safe retry defaults. It is not automatically model-callable and does not weaken the private
allowlist used by Harness's own SDK-compatibility protocols. See
[native access and platform coverage](/docs/databricks/native-access).
## Multidimensional coverage
The package capability registry separates:
- Databricks upstream maturity;
- native API fidelity;
- lifecycle operations;
- runtime certification;
- identity modes;
- contract clouds versus clouds with retained live evidence; and
- known gaps.
`DATABRICKS_PLATFORM_DOMAINS` includes unsupported product families as well as implemented ones.
An absent first-party adapter therefore cannot be mistaken for an absent Databricks capability.
Use the generated SDK or `databricksWorkspaceApi()` while a higher-level governed adapter is
missing.
There are two narrowly tested SDK 0.21 compatibility exceptions. Its generated Genie query-result
decoder expects object rows while the live API returns array rows. Its shared JSON decoder also tries
to convert a valid floating-point AI Search score to `BigInt`. Fabric retries only those two read-only
responses through the quarantined raw protocol transport when each exact decoder failure is detected;
other errors fail closed. Remove each fallback as soon as the pinned SDK accepts the live payload.
## Certification records
The protected live workflow emits schema-versioned, redacted evidence containing:
- cloud, region, workspace id/host, auth mode, and principal kind;
- Node and package versions plus every REST API family used;
- required and optional check identifiers with duration and result;
- source commit plus SHA-256 digests of the exact npm candidate and App artifact;
- App restart/cascade-deletion evidence and deployment conformance;
- a stable evidence id suitable for partner review.
Tier R release checks cannot become `not-configured`: that result makes certification fail. A Tier O
failure remains visible in the evidence but does not block a Tier R release. An
environment-specific required-check list can add checks but cannot remove Tier R. Tier A
covers the protected Jobs/Lakeflow/AI Search/serving/UC/workspace/secrets/Genie authoring lifecycles;
Tier O covers previews and SKU-specific paths unless a workflow explicitly promotes one. Authoring
certification uses a reverse-order cleanup ledger and fails when any created resource leaks. See the
[Databricks certification page](/docs/databricks/live-certification) for the current public status,
scope, limitations, and machine-readable record.
The runner executes required checks before configured optional checks while preserving declaration
order in the emitted evidence. This keeps short-lived OBO credentials and on-demand App health from
expiring while a long MLflow evaluation or other Tier O probe runs; optional checks still execute
and remain visible in the same record.
```bash
node packages/databricks/dist/certify.js
node scripts/build-databricks-compatibility-matrix.mjs \
artifacts/databricks-certification.json
```
`databricksCompatibilityRecord()` accepts only passing evidence with cloud, region, auth, runtime,
package, and artifact identity. The matrix labels a cloud/region `live-certified` only when such a
record is retained; otherwise it remains a contract target requiring workspace certification.
### Current `7.1.1` protected evidence
Package `7.1.1` is published on npm and passed its exact-version protected gate in
[GitHub Actions run 33909857317](/docs/reference/source-access)
from commit `bc56fa834a26e882953e4824796bd780a5b4efdb`. Evidence
`dbx-cert-ca5de23e2502326192565582e6844c5ee423d0bcaf9d3ada19c96a608ae9743b`
records all 21 Tier R checks as passed in Azure `eastus2` under OAuth M2M. Seven configured Tier O
checks passed; two Tier O probes failed and are recorded below without blocking the release tier.
Tier R covered identity, AI Gateway, Model Serving, SQL, Unity Catalog allow/deny, catalog
preflight denial, governed mutation approval, RAG, Genie, managed MCP under OBO, Volumes,
Lakebase, lineage, System Tables, actual-cost reconciliation, App health, and Lakebase-backed
restart recovery with cascade deletion. The required hook-authored dynamic-agent check also
preserved its persistent state across that restart. The generated App passed a 25-request burst
test with no failures or duplicate run IDs and enforced its rate limit.
This run executed in single-user workspace mode: the two-user isolation checks were skipped rather
than passed, so the record makes no claim about cross-user or cross-tenant protection and carries
no user-isolation evidence digest. Multi-user production deployments should treat that isolation as
uncovered until a two-identity run retains passing evidence.
Configured Tier O probes passed for ResponsesAgent, managed RAG evaluation, AI Search, Genie Agent
Mode, Jobs, Lakeflow, and a classic notebook. Unity Catalog Agent Services registration was rejected
by the workspace API, and Feature Serving did not resolve its configured records; this record
establishes nothing for those two surfaces.
The exact packed candidate digest is
`e6ed0ce22a6d8805e935a78e2310cba3bd7f82a357bc25fa679685ed7331b70e`.
The certified App directory digest is
`0ec00f14ab57f1c9e73e633824d25a3beaac8f54f8a06f6db2fddca86024ea38`.
The bound restart-evidence digest is
`d67f9ab298c671df879815a19a6d8d9fd0c259112d7490c8b865dbafbd09d69b`.
The retained `7.1.1` compatibility matrix labels Azure `live-certified` and AWS/GCP
`requires-workspace-certification`.
The exact tarball also passed the focused same-commit Tier A gate in
[run 33915517941](/docs/reference/source-access). Evidence
`dbx-cert-abd5c9a21d0fe784a8ca44572a0d4955ee18494ce677d90fe9af1fe849abb722`
records all ten required authoring checks as passed. Every destructive lifecycle ended with
`cleanupRequired: false`, both sweepers found no retained resources, and the cross-tier manifest
matched package version, commit, and tarball digest byte-for-byte.
The Tier A record does not establish classic compute, Delta Sync, provisioned throughput, or Genie
Agent Mode authoring. With no passing two-user probe in the current run, the separate 14-day rolling
isolation claim also remains unestablished. This Azure `eastus2` record does not imply AWS, GCP, or
another Azure region.
### Prior `7.0.2` protected evidence
Package `7.0.2` passed its exact-version protected gate in
[run 31053240003](/docs/reference/source-access) from commit
`544cd478d29b26ed7090499c682af0922f4533d1`. Evidence
`dbx-cert-40f9f87ecd2044f529d37e4dcbc260e55707cda9df5c987790078f35df29850a`
records all 21 Tier R checks and nine configured Tier O checks as passed in Azure `eastus2`,
including current-run two-user isolation. Its package digest is
`edf67563b0bf0daa211ada78c5a3996063a39befa464fa2df27ba04e102cb59c`, its App directory digest is
`04f852085e16f6b1a5d04b81e6c7160970282bf03091c7cf249410e56a364e74`, and its same-package Tier A
evidence is retained by [run 31056050505](/docs/reference/source-access) as
`dbx-cert-37f20b775f8f1e676f6b7d36bb17bb61f1ae9c593e560e028d1872764d70693d`.
### Prior `5.0.1` protected evidence
Package `5.0.1` passed its exact-version protected gate in
[run 30501444655](/docs/reference/source-access) from commit
`067694748e1326b3ef039f19c91486ded15b6f70`. Evidence
`dbx-cert-6826f4cc09501d1c112281248c0307589d922ae0fa21adc1c894c929bbf75613`
records the same 20 required Tier R checks and nine configured Tier O checks as passed in Azure
`eastus2`, including current-run two-user isolation. Its package digest is
`ee1eaac3c3e11ff6eaf1918023946d95381facab6107e6b81f7115a3a6660fd8`, its App directory
digest is `2110a13af799b63c5f9047f1959254802419c82143a9eaec7f49c83b794bf808`,
and it did not carry same-package Tier A evidence.
### Prior `4.9.2` protected evidence
Package `4.9.2` is published on npm and passed its exact-version protected gate in
[GitHub Actions run 29812986146](/docs/reference/source-access)
from commit `50996433b34cdedb6d2732949dedfe6a918cdd48`. Evidence
`dbx-cert-89ee6c419eda616be6cbffbcd42b842ed5a0f37d85683510c49c801f7911555b`
records all 20 Tier R checks as passed in Azure `eastus2` under OAuth M2M, including:
- identity, Unity AI Gateway, Model Serving, live RAG, and governed mutation approval;
- SQL plus Unity Catalog allow/deny and catalog preflight denial;
- ordinary Genie consumption, managed Genie MCP under OBO, and an OBO grant differential;
- Volumes, Lakebase, lineage, System Tables, and actual-cost reconciliation; and
- Databricks App health plus Lakebase-backed stop/start recovery and cascade deletion.
The exact packed candidate digest is
`91fb6cd09f2d631b47b6827b4eda535b7c405562fa9f5ed659c66a1c0f5f7258`.
The certified App directory digest is
`828d6ba0309b72430a6e2c024086815c28a762fd8b2adec1ba43100e7bb1c247`,
and the bound restart-evidence digest is
`d5ace515506f18043c48039716cd8c8c5f4d98dd0b11437bebe6f5d49aa49d1a`.
The workflow retained the post-certification App archive, package candidate, recovery record,
compatibility matrix, and passing Databricks App black-box conformance report together.
ResponsesAgent, managed RAG evaluation, AI Search, Feature Serving, Jobs, Lakeflow, and a classic
notebook also passed as visible Tier O checks. Agent Services returned "feature is not available"
and Genie Agent Mode returned `FEATURE_DISABLED`; both remain optional previews and are not implied
by the release claim. This record is retained as historical `4.9.2` evidence.
The capability registry embedded in the published package remains pinned to the `4.9.1` evidence
available when that artifact was built; Jobs 2.2 consumption is stable, while ResponsesAgent stays Beta because
the upstream surface is Beta. This record proves one OBO user-versus-App grant differential, not two-user App
isolation: `DATABRICKS_OBO_TOKEN_B` was not configured for this dispatch. It also does not imply AWS,
GCP or another Azure region. Current-package Tier A evidence is recorded separately below because
its short-lived U2M/OBO identity runs in the dedicated authoring workflow.
### Prior `4.3.x` status
Packages through `4.3.2` are published, but no passing exact-package Tier R record for that line is
retained in this repository. Do not describe the 4.3.x line as live-certified.
### Prior `4.2.0` recertification status
Package `4.2.0` adds deterministic artifact digesting, programmatic Bundle validation and App
deployment, and a narrow Databricks Apps user-authorization entrypoint. Its build, package,
documentation, SDK-boundary, and local contract gates pass, but it does not yet have a passing
exact-package protected workspace record. Do not describe `4.2.0` as live-certified until the Tier R
consumption workflow is rerun with valid OBO and Feature Serving fixtures.
### Prior `4.1.1` recertification record
Package `4.1.1` is published and its build, package, documentation, SDK-boundary, and local contract
gates passed. The exact-package Azure `eastus2` consumption run
[`29689767139`](/docs/reference/source-access) recorded 23
passing checks, including the Databricks App, Lakebase restart path, AI Search/RAG, Genie, SQL and UC,
Volumes, lineage, system tables, and cost reconciliation. It is **not** a passing release
certification record: the short-lived OBO fixture token was rejected as invalid, and the workflow's
explicitly promoted Feature Serving fixture returned `404`. Agent Mode also reported the workspace
preview as disabled, but remained non-blocking Tier O evidence.
Do not substitute that partial run for a passing `4.1.1` record. Rotate the OBO fixture, repair or
remove the promoted Feature Serving fixture, rerun Tier R, and link the resulting evidence before
describing `4.1.1` as live-certified. The older records below remain valid only for their exact
package, artifact, cloud, region, and capability set.
The current retained consumption record is the `7.1.1` Tier R evidence in
[run 33909857317](/docs/reference/source-access): all 21 required checks passed against the exact
published tarball and commit `bc56fa834a26e882953e4824796bd780a5b4efdb`. The same package's
management evidence is retained by [run 33915517941](/docs/reference/source-access) as
`dbx-cert-abd5c9a21d0fe784a8ca44572a0d4955ee18494ce677d90fe9af1fe849abb722`: all ten required
identity, approval, serverless Jobs, Lakeflow, direct-vector AI Search, custom-model Serving, UC OBO,
Workspace, secret-reference, and Genie authoring checks passed. Every destructive lifecycle ended
with `cleanupRequired: false`, both resource sweeps were empty, and the retained tarball digest
`e6ed0ce22a6d8805e935a78e2310cba3bd7f82a357bc25fa679685ed7331b70e` matched independently across
the Tier R and Tier A records.
The `4.0.2` records remain useful historical and second-region authoring evidence:
- `dbx-cert-b2c709717ef3a452234a5b2944311a576dafda03a9eea8246747ab3fa430b183`
from [run 29660486290](/docs/reference/source-access)
covers OBO execution in `eastus2`, serverless Jobs, Lakeflow, direct-vector and Delta Sync AI
Search, custom-model serving, managed-only UC schema/grants, Workspace objects, secret references,
and Genie Agent create/query/update/ACL-read/trash.
- `dbx-cert-d5bdbd276533a8634910e223898337314446561be2a444030fc2d11baa237bec`
from [run 29660486237](/docs/reference/source-access)
covers a policy-bound classic `new_cluster` Jobs definition lifecycle in Azure `westus3`.
Both records use artifact digest
`72f602626b9d79a808f6dd9e30f6ba703061bda19bae2aef983e7e92d7c61a45`; every required check
passed, all destructive lifecycles completed their six phases, and the pre/post sweeps found no
retained resources. Neither record implies AWS, GCP, provisioned throughput, Agent Mode Beta, or
another region passed the same live gate. The earlier `4.0.1` records remain historical evidence and
do not substitute for this exact native-SDK release evidence.
When the Agent Services Beta is enabled, set `DATABRICKS_AGENT_SERVICES_TEST=1` and
`DATABRICKS_AGENT_SERVICE_CONNECTION`. The runner then makes the create/discover/update/grant-read/
delete lifecycle a required workspace check. Add `DATABRICKS_AGENT_SERVICE_TEST_PRINCIPAL` to include
an `EXECUTE` and `READ_METADATA` grant/revoke differential. See
[Unity Catalog Agent Services](/docs/databricks/agent-services).
## App and Lakebase prerequisites
For the reference App, enable Databricks Apps, grant its service principal the selected SQL
Warehouse/serving/UC resources, provision the scoped runtime-token secret, and configure a UC Volume
for attachments. Restart recovery additionally requires a Lakebase Autoscaling endpoint and database
credential-exchange permission. Apply [private networking](/docs/operating/private-networking) when
the workspace uses private connectivity or an enterprise CA.
When CI and the App use different Lakebase principals, keep a stable release principal as table
owner, run schema migrations before App rollout, and grant the App principal DML plus sequence usage.
Fabric resolves shared telemetry relations independently of the App principal's first `search_path`
schema, so an already-migrated deployment does not require table ownership merely to restart. See
[Enterprise Databricks controls](/docs/databricks/enterprise#schema-ownership-in-automated-deployments).
---
# Databricks certification
Canonical: https://harness.techfabric.com/docs/databricks/live-certification
Public certification status, tested capabilities, evidence scope, and limitations for TechFabric Harness on Databricks.
TechFabric Harness tests its Databricks integration in protected workspaces before making live-support
claims. Certification binds the tested package, source commit, generated Databricks App, workspace
profile, and redacted evidence into one retained record. A passing record applies only to that exact
scope; it is not a blanket claim for every cloud, region, preview, or workspace configuration.
## Certification record
- **Current npm package:** `@fabric-harness/databricks@7.1.1`
- **Exact live-certified package:** `@fabric-harness/databricks@7.1.1`
- **Certification date:** September 4, 2026
- **Workspace profile:** Azure Databricks, `eastus2`, OAuth machine-to-machine
- **Consumption checks:** 21 of 21 required Tier R checks passed
- **Additional checks:** 7 configured Tier O checks passed; 2 Tier O checks failed
- **Authoring checks:** 10 of 10 required Tier A checks passed with an empty cleanup ledger
- **Source commit:** `bc56fa834a26e882953e4824796bd780a5b4efdb`
### What the public record shows
Exact-package results and limitations belong in the same view. The image below is a sanitized
rendering of the machine-readable public record; the JSON evidence and protected run remain
authoritative.
Version `7.1.1` is the byte-identical npm package certified by the Tier R record. The protected
publisher independently matched its tarball and generated App digests to the retained evidence
before publishing. The focused Tier A workflow then downloaded that exact retained tarball and
proved the governed authoring lifecycles against the same source commit and package digest.
The public record promotes `7.1.1` only after exact-commit Tier R, same-package Tier A, guarded
publication, public package consumption, and tag verification all passed.
### The repository version is the certified version
The version in this repository's `packages/databricks` manifest is `7.1.1` — the same version the
machine-readable record certifies. There is no outstanding source candidate: the published version,
the certified version, and the repository version are one package, and every live-support claim on
this page belongs to that exact build and source commit. A future candidate appears in the record
only as an explicitly uncertified entry until the complete protected evidence chain below promotes
it — no part of that chain is assumed from a version bump.
Development builds may be published under the npm `dev` dist-tag for single-user evaluation. A
development-tagged package is explicitly uncertified, does not move `latest`, does not advance this
record, and must not be used to claim shared-user isolation or production readiness. Promoting that
exact version to `latest` still requires the complete protected evidence above.
Tier R runs its 25-request concurrency burst in a fresh 60-second production rate-limit window.
This keeps earlier correctness probes from consuming burst capacity while preserving the real
30-request prompt ceiling. Any failed burst request still fails certification, and retained load
evidence aggregates non-secret failure categories such as `admission_http_429` for diagnosis.
## What the protected record proves
The current Tier R record passed:
- workspace identity and OAuth machine-to-machine authentication;
- Unity AI Gateway, Model Serving, SQL, and Unity Catalog allow-and-deny controls;
- governed mutation approval, lineage, Volumes, Lakebase, and System Tables cost reconciliation;
- RAG, Genie, and managed MCP invocation under on-behalf-of identity;
- Databricks App health, Lakebase-backed stop/start recovery, hook-authored dynamic-agent
continuity, cascade deletion, and burst-load bounds.
The certifying run executed in single-user workspace mode, so the two-user isolation probe was
skipped rather than passed; see the capability status below for exactly what that does not cover.
The same `7.1.1` record also passed configured Tier O probes for MLflow ResponsesAgent, managed RAG
evaluation, AI Search, Genie Agent Mode, Jobs, Lakeflow, and a classic notebook. Two Tier O probes
failed in this run: Unity Catalog Agent Services registration was rejected by the workspace API, and
Feature Serving did not resolve its configured records. Tier O failures are recorded visibly and do
not block certification, but this record establishes nothing for those two surfaces.
The same-package Tier A record passed approval provenance plus create, verify, mutate, delete, and
verify-delete lifecycles for serverless Jobs, Lakeflow, direct-vector AI Search, custom-model
Serving, Unity Catalog administration under OBO, Workspace objects, secret references, and Genie
Agents. Every lifecycle ended with `cleanupRequired: false`; the pre-run and post-run sweepers found
no retained certification resources.
## Capability status
### Live-certified
- **Core consumption path:** Required Tier R checks passed in the recorded Azure workspace.
- **Databricks App restart recovery:** Durable state, stream offsets, approvals, and deletion
behavior passed stop/start probes.
### Beta and workspace-dependent
- **MLflow ResponsesAgent — passed Tier O:** The Responses schema, stable streamed output item,
inference tables, and secure App proxy path passed in the current run.
- **Genie Agent Mode — passed Tier O:** Ordered output and a terminal SSE event passed in the
current run.
- **Managed RAG evaluation — passed Tier O:** The configured MLflow 3 evaluation job completed
successfully against the governed golden set.
### Not established by this record
- **Two-user App isolation, current run:** The certifying run executed in single-user workspace
mode, so the user-isolation checks were skipped rather than passed. This record says nothing
about cross-user or cross-tenant protection; multi-user production deployments must treat that
isolation as uncovered until a two-identity run retains passing evidence.
- **Two-user App isolation rolling claim:** With no passing two-user probe in the current run, the
separate 14-day rolling claim remains unestablished.
- **Unity Catalog Agent Services:** The Tier O registration lifecycle probe failed in the current
run, so this record establishes neither registration nor runtime invocation.
- **Feature Serving:** The Tier O primary-key resolution probe failed in the current run.
- **Optional authoring variants:** The current Tier A record does not establish classic-compute
Jobs, Delta Sync indexes, provisioned-throughput Serving, or Genie Agent Mode authoring.
- **AWS and Google Cloud workspaces:** This Azure record does not establish live behavior in another
cloud or region.
`Beta` and `workspace-dependent` are textual status labels, not color-only indicators. Preview
availability can differ by account, region, entitlement, and Databricks rollout.
A controlled single-user development or private evaluation can leave `require_user_isolation`
disabled; that is the workflow default and does not require inventing a second user. A single-user
certification can publish a changed Databricks package — the current `7.1.1` record is exactly that
case — but it carries an explicit limitation: the certification makes no claim about cross-user or
cross-tenant isolation, and multi-user production deployments should treat that isolation as
uncovered. In the evidence record this appears as the user-isolation checks being skipped rather
than passed, so the run makes no claim about cross-tenant protection. Certifying a shared or
user-facing App's isolation behavior, and advancing the public two-user rolling claim, continue to
require two distinct identities and retained isolation evidence.
## How certification is evaluated
The reference Databricks App is an on-demand certification target. The protected live
workflow deploys and starts it for the bounded certification window, uploads the
redacted evidence, and stops its compute in an unconditional cleanup step. A cleanup
failure fails the workflow rather than leaving the App silently active. Operators
should not keep the reference App running between certifications.
Fabric uses three evidence tiers:
- **Tier R — production consumption and runtime behavior:** Every required check must pass before
publishing a changed Databricks package.
- **Tier A — protected resource-authoring lifecycles:** Required for management claims; cleanup must
complete without leaked resources.
- **Tier O — preview, SKU-specific, or optional integrations:** Recorded visibly but non-blocking
unless explicitly promoted for that run.
The release workflow validates the protected run, commit, package digest, generated App digest,
restart evidence, required results, and evidence age before publishing. Direct local publication of
the Databricks package fails closed. An independently versioned, non-Databricks release scope may
publish only its explicit package allowlist without a Databricks run; it cannot include, tag, or
publish the Databricks candidate. This prevents pending Databricks source from blocking an unrelated
connector release without weakening the Databricks gate.
## What customers should verify
Before production rollout, run the documented preflight and smoke tests in the target workspace.
Confirm:
- the intended cloud and region expose Databricks Apps, Lakebase, Model Serving, and required AI
services;
- the application and user principals have only the necessary workspace and Unity Catalog grants;
- on-behalf-of scopes are consented and resolve the expected user;
- private networking, egress, and SQL policies match the deployment;
- restart recovery and tenant isolation pass using representative identities;
- Beta capabilities are enabled for the target workspace before relying on them.
Use [workspace compatibility](/docs/databricks/compatibility) for API and runtime requirements,
[authoring certification](/docs/databricks/authoring-certification) for resource-management
coverage, and [ResponsesAgent](/docs/databricks/responses-agent) for that Beta surface's specific
contract and evidence.
## Evidence references
- Tier R consumption run: `33909857317`
- Tier A authoring run: `33915517941`
- [Machine-readable public status](/evidence/databricks/current.json)
- [Source and protected-run access](/docs/reference/source-access)
- Certified package: `@fabric-harness/databricks@7.1.1`
- Tier R evidence: `dbx-cert-ca5de23e2502326192565582e6844c5ee423d0bcaf9d3ada19c96a608ae9743b`
- Tier A evidence: `dbx-cert-abd5c9a21d0fe784a8ca44572a0d4955ee18494ce677d90fe9af1fe849abb722`
- Required results: 21 of 21 Tier R checks passed
- Authoring results: 10 of 10 Tier A checks passed for the same package artifact
---
# Databricks App Tutorial
Canonical: https://harness.techfabric.com/docs/deployment/databricks-app
Scaffold, test, build, deploy, and persist a TechFabric Harness agent on Databricks Apps.
This tutorial deploys a finite analytics job into Databricks Apps. The App service principal calls
Model Serving and Unity Catalog APIs, while optional Lakebase stores sessions, durable submissions,
and conversation streams.
Read the [Databricks architecture](/docs/databricks/architecture) first when evaluating identity,
governance, or service boundaries. Use this page for the scaffold-to-deployment procedure.
## Prerequisites
- Node.js 22+ and pnpm or npm.
- A Databricks workspace with Apps enabled.
- A Model Serving endpoint.
- A SQL warehouse for the generated analytics tools.
- Optional: a Lakebase Autoscaling endpoint and OAuth Postgres role.
## 1. Scaffold the project
```sh
npx @fabric-harness/cli init --template databricks --dir analytics-agent
cd analytics-agent
npm install
```
The template creates:
```text
.fabricharness/
jobs/databricks-analyst.ts
roles/data-analyst.md
skills/analyze-table/SKILL.md
config.ts
.env.example
AGENTS.md
package.json
```
The generated job uses `defineDatabricksAgent()`:
```ts title=".fabricharness/jobs/databricks-analyst.ts"
import { defineDatabricksAgent } from '@fabric-harness/databricks';
import { schema } from '@fabric-harness/sdk';
import policy from '../policies/databricks.js';
export default defineDatabricksAgent({
name: 'databricks-analyst',
description: 'Answer governed analytics questions with Genie, inspectable SQL, Unity Catalog, and cost context.',
input: schema.object({ question: schema.string() }),
output: schema.string(),
triggers: { webhook: true, manual: true },
model: 'system.ai.gpt-oss-20b',
analyticsCopilot: true,
tools: ['sql-read', 'genie', 'consumption', 'tables', 'table-info'],
sandbox: 'empty',
policy,
});
```
## 2. Run without credentials
The mock path validates discovery, schemas, tool assembly, and the model loop without contacting a
workspace:
```sh
fh agents
fh describe databricks-analyst
fh run databricks-analyst \
--question "What tables are available in main?" \
--mock
```
Mock mode does not validate OAuth scopes, Unity Catalog grants, SQL execution, or deployment.
## 3. Configure a live workspace
Copy the template and fill the required values:
```sh
cp .env.example .env.local
```
```dotenv title=".env.local"
DATABRICKS_HOST=https://
DATABRICKS_CLIENT_ID=00000000-0000-0000-0000-000000000000
DATABRICKS_CLIENT_SECRET=replace-with-secret-reference
DATABRICKS_WAREHOUSE_ID=0123456789abcdef
DATABRICKS_MODEL=system.ai.gpt-oss-20b
DATABRICKS_GENIE_SPACE_ID=0123456789abcdef0123456789abcdef
DATABRICKS_CATALOG=main
DATABRICKS_ANALYTICS_STEWARD_AUDIENCE=analytics-stewards
DATABRICKS_COST_TENANT_ID=acme
DATABRICKS_COST_PER_DAY_USD=50
DATABRICKS_APP_OBO_REQUIRED=1
FABRIC_DATABRICKS_APP_CAPABILITIES=genie,obo,system-tables-cost
```
Run a live source invocation:
```sh
fh run databricks-analyst --question "Describe main.sales.orders"
```
The service principal must have workspace access, permission to invoke the serving endpoint,
warehouse usage, `CAN RUN` on the Genie Agent, System Tables access, and the required Unity Catalog
grants. Missing App capability bindings fail `fh doctor --target databricks-app`; missing runtime
Warehouse, Genie, steward, tenant, or cost settings fail before the first model ask.
## 4. Build the App
```sh
fh build --target databricks-app
```
Expected summary:
```text
Build complete
Output .fabricharness/build/databricks-app
Manifest .fabricharness/build/databricks-app/manifest.json
Jobs databricks-analyst
Agents none
```
Inspect the v2 manifest before deployment:
```sh
jq '{schemaVersion, jobs, agents, entrypoint}' \
.fabricharness/build/databricks-app/manifest.json
```
```json
{
"schemaVersion": 2,
"jobs": [{ "name": "databricks-analyst", "kind": "job" }],
"agents": [],
"entrypoint": "dist/server.mjs"
}
```
The artifact includes `app.yaml`, `databricks.yml`, the shared v2 Node server, and bundled `.mjs`
definitions. It does not regenerate a separate legacy HTTP runtime.
For a monorepo application, run the build from the package that owns `.fabricharness/`. Imports from
workspace-owned agent contracts are bundled into each generated definition, and source-only pnpm
protocols are removed from the runtime `package.json`. Treat the whole output directory as the
deployment unit:
```sh
test -z "$(grep -R -E '\"(workspace|catalog):' .fabricharness/build/databricks-app/package.json || true)"
cp -R .fabricharness/build/databricks-app /tmp/detached-agent-app
```
The detached directory is the handoff boundary for Runway or another deployment system. It does not
need the source repository, its shared-contract package, or TechFabric Harness packages at runtime.
See [Portable agent packages](/docs/deployment/portable-packages) for the full artifact contract,
digest-based handoff, isolation test, and deployment-time binding requirements.
### What developers see in the App
After deployment, inspect the agent identity, readiness, runtime, workspace region, and each bound
resource. The representative state below is sanitized and contains no client workspace identifier,
principal identifier, token, or secret value.
## 5. Deploy with a Declarative Automation Bundle
```sh
cd .fabricharness/build/databricks-app
databricks auth login --host "$DATABRICKS_HOST"
databricks bundle validate
databricks bundle deploy
databricks bundle run
```
Set provider and resource configuration through Databricks App environment/resources rather than
committing secrets. Apps supplies `DATABRICKS_APP_PORT`; the generated `app.yaml` starts
`dist/server.mjs` on that port.
The generated bundle also creates an MLflow experiment, attaches it to the App with `CAN_EDIT`, and
injects its ID as `DATABRICKS_MLFLOW_EXPERIMENT_ID`. This activates submission-correlated MLflow
traces without granting the App access to unrelated experiments. For a direct `databricks apps
deploy` outside the bundle, attach an experiment resource named `fabric-mlflow-experiment` or set an
experiment ID and grant the App service principal `CAN_EDIT` yourself.
Bind an existing Genie Agent as a least-privilege App resource in `.fabricharness/config.ts`:
```ts
export default {
target: 'databricks-app',
databricks: {
app: {
genie: { agentId: '0123456789abcdef0123456789abcdef' },
},
},
};
```
The generated `genie_space` resource defaults to `CAN_RUN` and is injected as
`DATABRICKS_GENIE_AGENT_ID` through `valueFrom`. `CAN_EDIT` or `CAN_MANAGE` requires an explicit
`authoring: true` on that resource declaration. The App service principal still needs access to the
Agent's warehouse and Unity Catalog data.
### Bind any native Databricks App resource
Use `databricks.app.resources` when the App needs native resources beyond the Genie and AI Search
shortcuts. Harness keeps the Databricks Bundle resource shape intact and adds only:
- `name`, the 1-30 character lowercase App resource key used by `valueFrom`;
- `env`, the environment variable exposed to the App; and
- `authoring: true`, an explicit acknowledgement required for write, manage, or owner permissions.
```ts title=".fabricharness/config.ts"
import type { FabricHarnessConfig } from '@fabric-harness/node';
export default {
databricks: {
app: {
resources: [
{
name: 'analytics-warehouse',
env: 'DATABRICKS_WAREHOUSE_ID',
sql_warehouse: {
id: '0123456789abcdef',
permission: 'CAN_USE',
},
},
{
name: 'refresh-job',
env: 'REFRESH_JOB_ID',
job: { id: '12345', permission: 'CAN_MANAGE_RUN' },
},
{
name: 'documents-index',
env: 'DOCUMENTS_INDEX',
uc_securable: {
securable_type: 'TABLE',
securable_full_name: 'main.rag.documents_index',
permission: 'SELECT',
},
},
{
name: 'vendor-token',
env: 'VENDOR_API_TOKEN',
secret: {
scope: 'analytics-agent',
key: 'vendor_api_token',
permission: 'READ',
},
},
],
},
},
} satisfies FabricHarnessConfig;
```
The supported blocks and normal runtime permissions are:
| Native block | Databricks resource | Normal runtime permissions |
| --- | --- | --- |
| `app` | Another Databricks App | `CAN_USE` |
| `database` | Lakebase provisioned database | `CAN_CONNECT_AND_CREATE` |
| `postgres` | Lakebase Autoscaling branch/database | `CAN_CONNECT_AND_CREATE` |
| `experiment` | MLflow experiment | `CAN_READ` |
| `genie_space` | Genie space or Agent | `CAN_VIEW`, `CAN_RUN` |
| `job` | Databricks Job | `CAN_VIEW`, `CAN_MANAGE_RUN` |
| `serving_endpoint` | Model Serving endpoint | `CAN_VIEW`, `CAN_QUERY` |
| `secret` | Databricks secret | `READ` |
| `sql_warehouse` | SQL Warehouse | `CAN_USE` |
| `uc_securable` | UC connection, function, table/AI Search index, or volume | `USE_CONNECTION`, `EXECUTE`, `SELECT`, `READ_VOLUME` |
Management, ownership, secret write, table modification, and volume write permissions fail the
build unless the declaration includes `authoring: true`. This prevents an accidental configuration
change from turning a runtime App into a resource administrator. Databricks remains the final
authorization boundary and can still reject any grant the deployer is not permitted to assign.
Each locator becomes a native Bundle variable such as
`app_resource_analytics_warehouse_id`. Override it with a Bundle target or `--var` to promote the
same artifact without rebuilding:
```sh
cd .fabricharness/build/databricks-app
databricks bundle validate \
--var app_resource_analytics_warehouse_id=
```
The generated `app.yaml` uses `valueFrom`; `databricks.yml` owns the native resource attachment; and
`databricks-app-resources.json` records a non-secret manifest for deployment evidence. A secret
binding stores only its scope and key—never the secret value.
Run preflight after building:
```sh
fh doctor --target databricks-app
```
When a generated artifact is present, doctor invokes the native
`databricks bundle validate --strict --output json` command from that artifact. Without a build, it reports
the build command without failing setup. Build-time validation remains network-free and rejects
duplicate keys, duplicate environment names, reserved Harness bindings, malformed identifiers,
invalid native permission pairs, and unacknowledged elevated access before writing an artifact.
The restricted [`with-databricks-app-resources` source example](/docs/reference/source-access)
contains build, environment override, failure, deployment, and cleanup instructions.
A completed deploy means Databricks accepted the new App revision; the public App URL can briefly
return `502`, `503`, or `504` while routing changes over. Release automation should poll `/api/ready` and
retry only safe `GET` probes during that transition. Do not retry a mutating `POST` unless it carries
the Harness idempotency key expected by that route. The protected recovery workflow applies this
rule for up to ten minutes before it starts state and approval assertions.
For a configuration-preserving production restart, pass the App name explicitly so the Databricks
CLI uses the Apps API and restarts the existing active deployment:
```sh
databricks apps stop "$DATABRICKS_APP_NAME"
databricks apps start "$DATABRICKS_APP_NAME"
```
Running `databricks apps start` without a name inside the generated bundle directory enters project
mode and resolves bundle variables again. Use the explicit-name form for a restart, or rerun
`fh deploy --target databricks-app` when applying new source or configuration. The protected live
workflow verifies the actual platform behavior: a bare stop/start creates a new deployment snapshot
from the same configured source path. It then proves that the managed App resources, pending human
approval, Lakebase sessions, submissions, conversation offsets, and UC Volume attachments remain
usable after the restart. Static, non-secret workspace settings are emitted into `app.yaml`, so the
new snapshot retains its catalog, schema, volume, model endpoint, and SQL warehouse configuration.
Once the App is running, invoke the finite job through its App URL:
```sh
TOKEN="$(databricks auth token --host "$DATABRICKS_HOST" | jq -r .access_token)"
curl -sS "$DATABRICKS_APP_URL/api/jobs/databricks-analyst" \
-H "authorization: Bearer $TOKEN" \
-H 'content-type: application/json' \
-d '{"question":"What were yesterday’s top products?"}'
```
Generated Databricks Apps mount the complete Harness HTTP surface beneath `/api`, which is the path
Databricks supports for OAuth Bearer-token API access. Root routes remain available for platform
health checks and local compatibility, but external clients must use `/api/jobs`, `/api/agents`,
`/api/responses`, and the corresponding `/api` inspection routes.
Databricks validates the OAuth token at the App ingress. M2M requests use App authorization: the
ingress admits the external `/api` request without forwarding the caller token or preserving that
prefix for the App process. The generated `app.yaml` and bundle inject the non-secret
`DATABRICKS_APP_NAME`, which Fabric uses as the App-principal identity when Databricks does not expose
`DATABRICKS_CLIENT_ID` to the process. Interactive OBO requests still bind to the forwarded user and
an isolated user tenant. Generated App servers trust Databricks Apps' integrity-protected
`x-forwarded-user`, `x-forwarded-email`, and `x-forwarded-preferred-username` headers for browser
admission, while the forwarded access token remains available only to downstream OBO clients. A
natural-person email or preferred username is required before Fabric takes this path, so an M2M
caller that supplies only `x-forwarded-user` remains a service principal. This avoids requiring a
SCIM scope merely to admit a user who already granted the App its declared SQL, Genie, and Model
Serving scopes. If Apps omits `x-forwarded-user` for browser traffic, Fabric uses the verified email
or preferred username as the stable identity instead of admitting an undefined principal.
When an external M2M token is forwarded through the same OBO header, Fabric
selects App authorization only if its proxy identity or signed token identity matches
`FABRIC_HARNESS_DATABRICKS_APP_CALLER_IDS`. Generated artifacts initialize that non-secret allowlist
from the deployment service principal's `DATABRICKS_CLIENT_ID`; set
`BUNDLE_VAR_databricks_app_caller_ids` to a comma-separated list when multiple automation callers
need App authorization. Non-allowlisted forwarded tokens fail closed on OBO validation. This
fallback is enabled only in generated Databricks Apps; generic Node servers remain fail-closed.
After granting user authorization, verify the browser path through `/api`, not the root route:
```js
const identity = await fetch('/api/certification/obo').then((response) => response.json());
const sessions = await fetch('/api/sessions').then((response) => response.json());
```
Both calls use the signed-in App user's isolated tenant. A `401` here means the generated App is not
receiving trusted Databricks Apps identity headers; it is not evidence that SQL or Genie consent was
denied. The reference certification route uses token-backed workspace inspection when the token
permits it; otherwise it reports the user principal already authenticated from Apps ingress and
omits token-lifetime fields. It never reflects the forwarded token.
## 6. Add Lakebase durability
Stateless Apps do not need `pg`. The generated server passes `lakebase: false` and omits PostgreSQL
imports unless `.fabricharness/config.ts` sets `databricks.app.lakebase: true` or the build resolves
a managed Lakebase App resource.
Fabric needs both Postgres connection information and the full Lakebase endpoint resource name:
```dotenv
DATABRICKS_LAKEBASE_HOST=ep-id.database.us-west-2.cloud.databricks.com
DATABRICKS_LAKEBASE_DATABASE=databricks_postgres
DATABRICKS_LAKEBASE_USER=00000000-0000-0000-0000-000000000000
DATABRICKS_LAKEBASE_ENDPOINT=projects/project-id/branches/branch-id/endpoints/endpoint-id
PGPORT=5432
```
When Lakebase is enabled, install `pg`. At runtime Fabric:
1. Gets a workspace OAuth token from the App service principal.
2. Exchanges it at `POST /api/2.0/postgres/credentials` using the endpoint resource name.
3. Supplies the database credential through the pool password callback.
4. Refreshes before expiry and single-flights concurrent refreshes.
5. Injects Lakebase session, submission, and conversation-stream stores into the shared server.
6. Binds scheduled runs to the App principal's isolated tenant and coordinates replicas through a
Postgres scheduler lease.
7. Persists the last scheduled occurrence and runs the most recent missed occurrence once after an
App restart. Stateless Apps use a process-local lease, skip missed ticks, and still bind work to
the App principal.
Never place the workspace OAuth token directly in `PGPASSWORD`.
Scheduled background work is intentionally owned by the App principal, not by whichever user first
opens a persistent session. App users retain their own OBO tenants; service-owned schedules and
user-owned conversations cannot silently append to one another.
### Resolve approvals from a deployed App
Databricks App users have tenant-scoped approval permissions without broad `admin:read`. Discover
pending work through the App's `/api` surface:
```sh
TOKEN="$(databricks auth token --host "$DATABRICKS_HOST" | jq -r .access_token)"
export DATABRICKS_OAUTH_TOKEN="$TOKEN"
fh approvals --url "$DATABRICKS_APP_URL/api" --token-env DATABRICKS_OAUTH_TOKEN
fh approve \
--url "$DATABRICKS_APP_URL/api" \
--token-env DATABRICKS_OAUTH_TOKEN
```
Discovery returns only sessions visible to the authenticated Databricks user. The `audience`
attached to an approval remains a host routing label: restrict `approval:write` to the intended
Databricks group or application role.
That user-scoped route cannot discover approval requests raised by App-principal scheduled work;
the tenant separation above is intentional. A schedule that needs a human decision must either
persist a proposal and let a later user-owned session perform the gated action, or publish a
deterministic decision card through a trusted external bridge such as
[Buzz](/docs/ecosystem/channels/buzz). The application's governed approval record associated with
the card receipt must retain the scheduler session and approval ids, map an authenticated response
back to that exact request, and resolve through the normal stored approval/CAS boundary. Do not
grant a user the App tenant or parse free-form chat as approval.
### Bind an AI Search index
Declare the existing index in `.fabricharness/config.ts`:
```ts
export default {
databricks: {
app: {
aiSearch: {
index: 'main.support.kb_index',
},
},
},
};
```
The App build emits a managed `uc_securable` table resource with `SELECT`, references it through
`valueFrom`, and injects the full index name as `DATABRICKS_AI_SEARCH_INDEX`. Databricks grants
the App service principal the required parent `USE CATALOG` and `USE SCHEMA` privileges when the
deployer is authorized to grant them. The generated bundle uses
`var.databricks_ai_search_index`, so targets can bind different indexes without rebuilding the
application code.
### Add non-secret App environment configuration
Declare application configuration that is safe to embed in the build artifact:
```ts
export default {
databricks: {
app: {
env: {
FEATURE_MODE: 'rag',
SUPPORT_QUEUE: 'priority',
},
},
},
};
```
Fabric emits the entries deterministically into both `app.yaml` and the generated bundle. Keys
must be uppercase environment-variable names. Harness-managed names cannot be overridden, and
credential-shaped names such as `*_TOKEN`, `API_KEY`, `PASSWORD`, or `SECRET` are rejected. Bind
credentials through Databricks App resources and `valueFrom`; do not bake them into generated
artifacts.
### Build-time bundle variables
`fh build` and `fh deploy` read the following Databricks bundle inputs while generating the App
artifact:
| Variable | Generated setting | Default |
| --- | --- | --- |
| `BUNDLE_VAR_databricks_catalog` | `DATABRICKS_CATALOG` | `main` |
| `BUNDLE_VAR_databricks_schema` | `DATABRICKS_SCHEMA` | `agents` |
| `BUNDLE_VAR_databricks_volume` | `DATABRICKS_VOLUME` | `fabric_attachments` |
| `BUNDLE_VAR_databricks_serving_endpoint` | `DATABRICKS_MODEL` | `system.ai.gpt-oss-20b` |
| `BUNDLE_VAR_databricks_warehouse_id` | `DATABRICKS_WAREHOUSE_ID` | empty |
| `BUNDLE_VAR_databricks_app_caller_ids` | allowed automation caller IDs | `DATABRICKS_CLIENT_ID`, then empty |
| `BUNDLE_VAR_lakebase_endpoint` | managed Lakebase endpoint resource | unset |
| `BUNDLE_VAR_lakebase_database_resource` | managed Lakebase database resource | unset |
These are build inputs, not a secret store. Rebuild after changing them. The generated
`databricks.yml` also exposes corresponding deployment variables so bundle targets can override
environment-specific catalog, schema, volume, model, warehouse, caller, AI Search, and Lakebase
resource values.
For `fh deploy`, provide the Lakebase **resource names** to the bundle separately from the direct
PostgreSQL connection values shown above:
```sh
export BUNDLE_VAR_lakebase_endpoint='projects/project-id/branches/production/endpoints/primary'
export BUNDLE_VAR_lakebase_database_resource='projects/project-id/branches/production/databases/app-db'
fh deploy --target databricks-app
```
`app-db` is the Lakebase database resource ID. It can differ from the PostgreSQL database name such
as `databricks_postgres`. Use the `name` field returned by
`databricks postgres list-databases projects/project-id/branches/production`.
For a custom server entrypoint, the equivalent wiring is:
```ts
import { databricksApp } from '@fabric-harness/databricks';
import { startDevServer } from '@fabric-harness/node';
const app = databricksApp();
const server = await startDevServer({
port: Number(process.env.DATABRICKS_APP_PORT ?? 8080),
host: '0.0.0.0',
basePath: '/api',
...(await app.serverOptions()),
});
const shutdown = async () => {
await server.close();
await app.close();
};
process.once('SIGTERM', () => void shutdown());
process.once('SIGINT', () => void shutdown());
```
## 7. Add a persistent agent
Create `.fabricharness/agents/copilot.ts`:
```ts
import { createAgent } from '@fabric-harness/sdk';
export default createAgent(({ id }) => ({
name: 'copilot',
model: 'databricks/system.ai.gpt-oss-20b',
instructions: `You are the governed analytics copilot for account ${id}.`,
triggers: { webhook: true },
}));
```
Rebuild. The manifest now reports the finite job under `jobs` and `copilot` under `agents`.
Persistent input returns a durable receipt:
```sh
curl -i "$DATABRICKS_APP_URL/agents/copilot/acct-42" \
-H "authorization: Bearer $DATABRICKS_TOKEN" \
-H 'content-type: application/json' \
-d '{"message":"Summarize this account’s sales."}'
```
Use `@fabric-harness/client` to wait or stream by offset. See
[Persistent agents](/docs/building/persistent-agents).
## 8. Validate before production
Run the repository live suite with a controlled workspace:
```sh
FABRIC_DATABRICKS_TEST=1 \
FABRIC_DATABRICKS_LAKEBASE_TEST=1 \
pnpm --filter @fabric-harness/databricks test
```
The production gate should cover App deployment, service-principal access, user OBO where used,
Unity Catalog denials, Model Serving, SQL, and Lakebase restart recovery. Local contract tests do not
replace that workspace-specific validation.
## See also
- [Databricks integration reference](/docs/deployment/databricks)
- [Build and run artifacts](/docs/deployment/build-artifacts)
- [Enterprise controls](/docs/building/enterprise-controls)
- [Production readiness](/docs/reference/production-readiness)
---
# Databricks SQL sandbox
Canonical: https://harness.techfabric.com/docs/ecosystem/sandboxes/databricks-sql
Use a Databricks SQL Warehouse as a governed query execution sandbox for TechFabric Harness sessions.
**Package:** `@fabric-harness/databricks/sql-sandbox`
**Execution:** Databricks SQL Statement Execution API
`databricksSqlSandbox()` maps `session.shell(statement)` to a SQL Warehouse. It is designed for data
agents whose shell abstraction should mean SQL, while retaining the common TechFabric Harness sandbox
interface.
```ts
import { databricksSqlSandbox } from '@fabric-harness/databricks/sql-sandbox';
import { init } from '@fabric-harness/sdk';
const runtime = await init({
sandbox: databricksSqlSandbox({
host: process.env.DATABRICKS_HOST!,
principal: { kind: 'on-behalf-of', userToken: () => workspaceIdentity() },
warehouseId: process.env.DATABRICKS_WAREHOUSE_ID!,
catalog: 'main',
schema: 'analytics',
resultFormat: 'jsonl',
waitTimeoutSeconds: 30,
}),
});
const session = await runtime.session();
const result = await session.shell('SELECT current_user(), current_catalog()');
```
## Capabilities
| Capability | Behavior |
| --- | --- |
| `exec()` | Executes one SQL statement and serializes result rows as JSONL or CSV |
| Text and binary files | Small in-memory map scoped to the sandbox instance |
| Snapshots and restore | Not supported |
| Network | Managed by the provider |
| Persistence | Session only for the in-memory file map |
| Isolation | Databricks SQL Warehouse plus its configured Unity Catalog access |
The file methods do not access DBFS, workspace files, or Unity Catalog Volumes. Mount a
`databricksVolumeSource()` for governed non-tabular data. Use Docker or another code sandbox for
bash, Python package installation, repository work, or arbitrary code execution.
## Required access
The acting principal needs workspace access, permission to use the SQL Warehouse, and the required
Unity Catalog grants. Prefer a rotating OAuth token provider rather than a long-lived token.
See [Databricks connectors and sandboxes](/docs/databricks/sandboxes-connectors) for the selection
guide and full examples.
---
# Databricks Lakebase
Canonical: https://harness.techfabric.com/docs/ecosystem/databases/lakebase
Durable state for Databricks Apps using exchanged database credentials.
Lakebase is Fabric's preferred durable Postgres path inside Databricks Apps. Configure the database
connection fields and the full endpoint resource name:
```sh
DATABRICKS_LAKEBASE_HOST=ep-id.database.us-west-2.cloud.databricks.com
DATABRICKS_LAKEBASE_DATABASE=databricks_postgres
DATABRICKS_LAKEBASE_USER=00000000-0000-0000-0000-000000000000
DATABRICKS_LAKEBASE_ENDPOINT=projects//branches//endpoints/
PGPORT=5432
```
`databricksApp()` also accepts `PGHOST`, `PGDATABASE`, `PGUSER`, `PGPORT`, and `ENDPOINT_NAME`, which
align with App database resource configuration. The Databricks bundle exchanges the app's workspace
OAuth identity for a short-lived database credential, refreshes it with single-flight protection,
and injects Lakebase-backed session, submission, and conversation-stream stores into the shared
server. A raw workspace token is rejected as a database password.
Use `lakebaseClient()` when assembling a custom bundle. For the standard path, follow [Databricks App deployment](/docs/deployment/databricks-app#6-add-lakebase-durability), including the live restart and credential-exchange smoke. Local contract tests cannot validate workspace permissions or the hosted credential response shape.
---
# Authentication and RBAC
Canonical: https://harness.techfabric.com/docs/operating/auth
Bind authenticated principals, tenants, permissions, and SSO identities to every server operation.
`fh dev` and generated Node builds ship a simple Bearer-token auth surface for solo deployments. Enterprise hosts use
`authenticate` to return a principal with tenant and permission scopes. The server derives the
execution actor from that principal, so clients cannot spoof audit identity in request bodies.
Local/dev mode remains open when no auth is configured. Production mode fails closed: only
`/health` and `/ready` are anonymous, and every other HTTP or WebSocket route returns `401` unless
the bearer token or custom resolver authorizes it.
## Default: Bearer token
```sh
FABRIC_HARNESS_API_TOKEN=changeme fh dev --port 9111
```
```sh
curl -H 'Authorization: Bearer changeme' http://localhost:9111/sessions
```
WebSocket upgrades use the same token via `?token=` query param (browsers can't set headers on `WebSocket` constructor):
```ts
new WebSocket('wss://app.example.com/sessions/abc/ws?token=changeme');
```
## Custom resolver: `extractAuthToken`
When you have an existing identity layer (cookies, JWT, SSO terminator), pass a custom resolver:
```ts
import { startDevServer, parseCookie } from '@fabric-harness/node';
await startDevServer({
extractAuthToken: (req) => {
const cookie = parseCookie(req, 'session');
if (!cookie) return undefined; // fall through to bearer check
return verifySessionCookie(cookie); // your validator returns true/false
},
});
```
The resolver returns:
- `true` — authorize the request
- `false` — reject with 401
- `undefined` — fall through to the built-in bearer-token check
Same hook works for HTTP requests AND WebSocket upgrades.
## Enterprise principals and permissions
Use `authenticate` when the server must enforce tenant isolation and route-level RBAC:
```ts
import { startDevServer } from '@fabric-harness/node';
await startDevServer({
authenticate: async (req) => {
const claims = await verifyCompanyJwt(req.headers.authorization);
if (!claims) return false;
return {
id: claims.sub,
kind: claims.type === 'user' ? 'user' : 'service-principal',
provider: 'company-oidc',
tenantId: claims.organizationId,
displayName: claims.name,
ucPrincipal: claims.databricksPrincipal,
permissions: claims.permissions,
};
},
});
```
The built-in permission scopes are:
| Permission | Operations |
| --- | --- |
| `agent:invoke` | Jobs, persistent prompts, dispatch, and channel ingress |
| `session:read` | Session, run, timeline, metrics, and conversation reads |
| `session:abort` | Abort an active persistent submission |
| `session:delete` | Cascade-delete a settled persistent instance |
| `approval:read`, `approval:write` | Inspect and vote on approvals |
| `artifact:read` | Read artifacts and attachments |
| `build:read` | Read build manifests |
| `mcp:invoke` | Call the server's `/mcp` endpoint |
| `session:replay` | Fork a session from a durable entry |
| `admin:read` | Admin and OpenAPI routes |
`*` grants every permission and `session:*` grants all permissions for one resource. Omitting
`permissions` keeps compatibility with existing custom authenticators and grants unrestricted
access; enterprise authenticators should always return an explicit array.
`tenantId` binds the principal to one tenant. A request for a different `X-Fabric-Tenant` or
`?tenant=` receives `403`. Use `tenantIds` instead when an operator may explicitly select from a
known set. A single-value `tenantIds` allowlist binds implicitly when the request omits a tenant;
an allowlist containing multiple tenants requires an explicit permitted tenant selection and
otherwise returns `403`. Omitting the selection never widens access to every tenant. The resulting
principal and tenant propagate through submissions, entries, approvals, tool attribution, and
telemetry.
Add application-specific checks after the built-in scopes with `authorize`:
```ts
await startDevServer({
authenticate: companyAuthenticator,
authorize: ({ principal, permission, tenantId }) =>
permission !== 'session:delete' ||
(principal.roles?.includes('tenant-admin') === true && tenantId !== undefined),
});
```
HTTP and WebSocket upgrades use the same authentication, tenant, and authorization pipeline.
## OIDC and JWKS validation
Install `jose`, then use the built-in validator for any OpenID Connect provider. It verifies the
signature against a local or remote JWKS, issuer, audience, required claims, algorithms, expiry,
and clock tolerance. Remote JWKS keys are cached and refreshed when providers rotate signing keys.
```sh
pnpm add jose
```
```ts
import { oidcJwtAuthenticator, startDevServer } from '@fabric-harness/node';
await startDevServer({
authenticate: oidcJwtAuthenticator({
issuer: 'https://identity.example.com',
audience: 'fabric-api',
jwksUri: 'https://identity.example.com/.well-known/jwks.json',
provider: 'company-oidc',
claims: {
tenantId: 'organization_id',
roles: 'roles',
groups: 'groups',
permissions: 'permissions',
},
groupRoles: {
'platform-operators': ['operator'],
},
rolePermissions: {
operator: ['session:*', 'approval:*', 'artifact:read'],
},
}),
});
```
The validator returns `undefined` when no supported token is present, so a configured legacy API
token can remain as fallback. An invalid or expired JWT returns `false` and the server responds
with `401`. Use `mapPrincipal` when provider claims need logic beyond declarative claim mapping.
### Microsoft Entra ID
`entraIdAuthenticator()` configures the tenant-specific v2 issuer and rotating JWKS. It maps `oid`,
`tid`, `name`, `roles`, and `groups`, and treats `idtyp: app` as a service principal.
```ts
authenticate: entraIdAuthenticator({
tenantId: process.env.ENTRA_TENANT_ID!,
clientId: process.env.ENTRA_CLIENT_ID!,
groupRoles: { [process.env.ENTRA_OPERATOR_GROUP!]: ['operator'] },
rolePermissions: { operator: ['session:*', 'approval:*', 'admin:read'] },
})
```
### Databricks Apps
`databricksAppsOidcAuthenticator()` validates workspace-issued JWTs and accepts either an
`Authorization: Bearer` token or the Databricks Apps `x-forwarded-access-token` header. It maps the
workspace, email/Unity Catalog principal, roles, and groups into the same server identity.
```ts
authenticate: databricksAppsOidcAuthenticator({
workspaceHost: process.env.DATABRICKS_HOST!,
audience: process.env.DATABRICKS_APP_CLIENT_ID!,
rolePermissions: { 'data-steward': ['session:read', 'approval:*'] },
})
```
If the workspace uses a custom issuer, account-level identity federation, or a proxy JWKS, set
`issuer` and `jwksUri` explicitly. Keep the service-principal path for unattended traffic and use
the forwarded user token only when operations should inherit that user's Databricks grants.
## Delete a persistent instance
Deletion is deliberately restricted to settled work:
```sh
curl -X DELETE \
-H "Authorization: Bearer $TOKEN" \
https://agents.example.com/agents/support/customer-42
```
The operation removes every named session for that agent instance, its settled submissions,
conversation streams, session artifacts, and content-addressed attachments. If work is still
active, the server requests an abort and returns `409`; retry after settlement. Durable custom
stores must implement `SessionStore.delete` and `deleteSessionSubmissions`.
## Recipe: SSO terminator (Cloudflare Access, IAP, Cognito)
When fabric-harness runs behind an SSO terminator, the terminator validates the user and forwards a verified header. Trust the header inside the resolver:
```ts
await startDevServer({
extractAuthToken: (req) => {
const userHeader = req.headers['cf-access-authenticated-user-email'];
if (typeof userHeader === 'string' && userHeader.length > 0) return true;
return undefined;
},
});
```
Make sure the terminator strips the header from inbound requests that didn't pass through it — otherwise clients can spoof.
## Recipe: per-tenant cookies
Combine with `X-Fabric-Tenant` header to scope cookie validation per tenant:
```ts
extractAuthToken: (req) => {
const tenant = req.headers['x-fabric-tenant'];
const cookie = parseCookie(req, `session_${tenant}`);
return cookie ? verifyForTenant(cookie, tenant) : undefined;
},
```
## WebSocket cookies (browsers)
Same-origin WebSocket upgrades carry browser cookies automatically — `extractAuthToken` reads them just like HTTP. Cross-origin? `Sec-WebSocket-Protocol` workarounds exist but are clunky; recommend bearer-via-query-param (`?token=...`) for cross-origin WS.
## See also
- [HTTP server reference](/docs/reference/http-server)
- [Multi-tenancy](/docs/operating/multi-tenancy)
---
# Operator Console
Canonical: https://harness.techfabric.com/docs/operating/operator-console
Inspect sessions, approvals, tenants, and persistent instances from the authenticated Node server.
The Node server includes a compact operator console at `/admin`. It uses the same authentication,
tenant binding, and `admin:read` permission checks as the admin APIs. Its warm-paper surfaces,
burnt-sienna emphasis, compact controls, and light/dark palette follow the same Fabric family
contract as `@fabricorg/ui`. The console remains self-contained so loading `@fabric-harness/node`
never introduces a React or browser-asset dependency.
```sh
FABRIC_HARNESS_API_TOKEN="$API_TOKEN" fh dev --port 4317
```
Open `http://localhost:4317/admin`, enter the API token, and connect. The token is kept in browser
`sessionStorage` and sent as a bearer header; it is not written to the server-rendered page.
## What operators see
The console combines tenant-scoped sessions, durable queue state, persistence health, and only the
actions allowed for the authenticated principal. These screenshots are deterministic, sanitized UI
states; no client data, workspace token, or environment identifier is embedded in the documentation.
The console provides paginated, searchable views for:
- finite jobs and persistent agents;
- sessions, runs, and active durable submissions;
- built-in tools and pending or completed approvals;
- persistence, queue, and worker health;
- session cost metrics, policy decisions, and the audit trail.
Operators with `approval:write` can approve or reject a pending request. Operators
with `session:replay` can fork a session from any durable entry, while
`session:abort` and `session:delete` control abort and cascade deletion. Buttons
are hidden when the authenticated principal lacks the corresponding permission;
the server independently authorizes every action.
An application user does not need `admin:read` merely to handle approvals. Use the tenant-scoped
`GET /approvals` endpoint or `fh approvals --url ` to discover visible requests, then
`fh approve` or `fh reject` against the same remote URL. The `/admin` console remains reserved for
operators who need broader run, health, audit, and replay access.
All collection endpoints use the same pagination shape:
```json
{
"items": [],
"total": 0,
"offset": 0,
"limit": 50,
"nextOffset": null
}
```
Use `?offset=0&limit=50&q=running`. A tenant-bound principal only receives its
own tenant. A principal allowed to operate several tenants can add `?tenant=acme`;
the server validates that selection against the principal before reading data.
Custom SSO deployments should use cookie authentication in `authenticate`, so the console can load
without a separate bearer token. Assign operators `admin:read`, plus the action
permissions they need. The APIs remain the source of truth; the console
does not bypass RBAC or tenant isolation.
---
# Private networking and egress
Canonical: https://harness.techfabric.com/docs/operating/private-networking
Enforce outbound policy at container, cluster, and cloud-provider boundaries, with proxy, DNS, custom CA, and mTLS support.
Fabric applies URL, DNS-answer, and redirect policy before supported HTTP calls. Production
deployments also need a boundary outside the agent process. That second layer prevents a custom
tool, dependency, or direct socket from bypassing the application policy.
```mermaid
flowchart LR
A[Agent process] -->|policy-checked request| P[Internal egress proxy]
A -. direct socket denied .-> X[Public or private service]
P -->|domain and port allowlist| D[Databricks workspace APIs]
P -->|private DNS| E[Private endpoints]
subgraph Boundary[Container, cluster, or provider boundary]
A
P
end
classDef runtime fill:#e8f0fe,stroke:#2563eb,color:#172554
classDef control fill:#fff4d6,stroke:#d97706,color:#451a03
classDef service fill:#dcfce7,stroke:#16a34a,color:#052e16
classDef denied fill:#fee2e2,stroke:#dc2626,color:#450a0a
class A runtime
class P control
class D,E service
class X denied
```
## Require an enforceable boundary
`assertEnforceableNetworkPolicy()` rejects a network policy backed only by process-level checks.
Call it during production startup after creating the sandbox:
```ts
import {
DockerSandboxEnv,
assertEnforceableNetworkPolicy,
type CapabilityPolicy,
} from '@fabric-harness/sdk';
const policy: CapabilityPolicy = {
network: {
mode: 'allowlist',
protocols: ['https:'],
hosts: ['*.azuredatabricks.net', '*.databricks.com'],
resolveDns: true,
},
};
const sandbox = new DockerSandboxEnv({
workspacePath: process.cwd(),
network: 'fabric-agent-internal',
networkBoundary: {
enforcement: 'container',
reference: 'docker-network:fabric-agent-internal+egress-proxy',
},
});
assertEnforceableNetworkPolicy(policy, sandbox, {
deployment: 'production',
});
```
The `networkBoundary` value is an operator assertion, not a network provisioner. Create the Docker
network with `internal: true`, connect the agent only to that network, and connect an independently
configured proxy to both the internal and destination networks. The
[`private-networking` example](/docs/reference/source-access)
contains a runnable Compose topology that proves a proxied request succeeds while a direct request
fails.
## Proxy, custom CA, and mTLS
Use one transport factory for connectors that require enterprise TLS settings:
```ts
import { readFile } from 'node:fs/promises';
import { createPrivateNetworkFetch } from '@fabric-harness/node';
const client = createPrivateNetworkFetch({
proxy: {
url: process.env.HTTPS_PROXY!,
authorization: `Bearer ${process.env.EGRESS_PROXY_TOKEN!}`,
},
tls: {
ca: await readFile('/var/run/fabric-secrets/private-ca.pem'),
cert: await readFile('/var/run/fabric-secrets/client.crt'),
key: await readFile('/var/run/fabric-secrets/client.key'),
},
policy,
});
try {
const response = await client.fetch(`${process.env.DATABRICKS_HOST}/api/2.0/clusters/list`);
if (!response.ok) throw new Error(`Workspace request failed: ${response.status}`);
} finally {
await client.close();
}
```
Keep proxy authorization and PEM material in a secret provider or mounted secret volume. Proxy URLs
with embedded credentials are rejected so credentials cannot appear in URL logs. `rejectUnauthorized`
defaults to `true`; do not disable certificate verification in production.
## Kubernetes and AKS
`createKubernetesEgressNetworkPolicy()` emits deny-by-default egress with only three permitted paths:
1. selected cluster DNS pods on TCP/UDP 53;
2. a selected egress proxy on one TCP port;
3. explicitly listed private CIDRs and ports.
```ts
import {
createKubernetesEgressNetworkPolicy,
kubernetesSandbox,
} from '@fabric-harness/connectors/k8s';
const manifest = createKubernetesEgressNetworkPolicy({
namespace: 'fabric-agents',
podSelector: { 'app.kubernetes.io/name': 'fabric-agent' },
egressProxy: {
namespaceSelector: { 'kubernetes.io/metadata.name': 'networking' },
podSelector: { 'app.kubernetes.io/name': 'fabric-egress-proxy' },
port: 3128,
},
privateCidrs: [{ cidr: '10.40.0.0/24', ports: [443] }],
});
// Apply `manifest` with the cluster API before attaching the pod.
const sandbox = kubernetesSandbox(pod, {
networkPolicy: {
namespace: manifest.metadata.namespace,
name: manifest.metadata.name,
},
});
```
The generator rejects public CIDRs. Route public destinations through the proxy, where FQDN and
certificate policy can be enforced. On AKS, combine this policy with Azure CNI, private clusters,
Private Link, private DNS zones, workload identity, and an Azure Firewall or proxy route. Mount
custom trust bundles and client certificates through the Secrets Store CSI Driver.
## Databricks private connectivity
For Databricks Apps, keep the workspace identity supplied by the platform and configure connectivity
at the workspace/account layer:
- use workspace private connectivity for front-end and back-end API paths;
- route SQL Warehouse, Model Serving, AI Search, Lakebase, and Unity Catalog traffic through
approved private endpoints where the workspace feature supports them;
- use workspace DNS names in the Fabric allowlist and resolve them through the private DNS path;
- exchange the App identity for short-lived Lakebase credentials instead of storing a database
password;
- store attachments in Unity Catalog Volumes and use service-principal or on-behalf-of identity,
never embedded cloud storage keys.
See [Databricks architecture](/docs/databricks/architecture),
[enterprise Databricks controls](/docs/databricks/enterprise), and
[Databricks Apps deployment](/docs/deployment/databricks-app) for the complete identity and data flow.
## Verification checklist
- A direct request with proxy settings disabled cannot reach its destination.
- An allowed proxied request succeeds; a non-allowlisted hostname is denied by the proxy.
- Private DNS resolves only to the expected private ranges.
- An untrusted server certificate fails, while the configured private CA succeeds.
- The server requires and validates the expected client certificate.
- `assertEnforceableNetworkPolicy()` passes with a named container, cluster, or provider boundary.
- Logs, traces, errors, and build artifacts contain no proxy credentials, client keys, or workspace
tokens.
---
# MCP
Canonical: https://harness.techfabric.com/docs/reference/mcp
How Model Context Protocol servers plug into TechFabric Harness, so an agent can use a tool you did not have to wrap yourself.
TechFabric Harness connects Model Context Protocol (MCP) servers to a session as normal Fabric tools. Remote Streamable HTTP, legacy SSE, and local stdio transports are supported.
It can also expose Harness jobs, persistent agents, and governed tools as an authenticated MCP
Streamable HTTP server.
## Expose Fabric through MCP
```ts
import { startDevServer } from '@fabric-harness/node';
await startDevServer({
authenticate: companyAuthenticator,
mcp: {
enabled: true,
exposeJobs: true,
exposeAgents: true,
tools: [lookupAccount],
},
});
```
Connect an MCP client to `https://agents.example.com/mcp`. Finite jobs appear as `job_` tools
using their declared input schema. Persistent agents appear as `agent_` tools accepting
`instanceId`, `message`, and optional `session`. Set `FABRIC_HARNESS_MCP_ENABLED=1` to enable the
same surface in generated Node artifacts.
The `/mcp` route requires `mcp:invoke`. The authenticated principal and tenant propagate into job
runs and persistent submissions, including tool attribution and audit entries. Direct tool calls receive
the actor, tenant, MCP request id, canonical input digest, cancellation signal, and an empty default
sandbox through `ToolContext`. MCP annotations are derived from Fabric tool effects, and tool errors are
redacted before returning to the client.
Custom tools must declare `metadata.effect`. Read-only and no-effect tools can be exposed directly.
`write` and `execute` tools must first be wrapped by server-side governance and declare
`metadata.governed: true`; an unknown effect or ungoverned mutation fails server startup. Resolve approval
grants and policy versions through the trusted `mcp.resolveToolContext` hook or another server-side store.
Never accept an approval grant from model-supplied tool arguments.
```ts
await startDevServer({
authenticate: companyAuthenticator,
mcp: {
enabled: true,
tools: [governedWriteTool],
resolveToolContext: async ({ toolCallId, input, actor, tenantId }) => ({
policyVersion: await policyStore.currentVersion(tenantId),
approval: await approvalStore.findGrant({ toolCallId, input, actor, tenantId }),
}),
},
});
```
If the effect is missing, the mutation has not been governance-wrapped, or the configured resolver cannot
produce the approval expected by the tool's policy, the operation fails closed before provider execution.
## Remote MCP server
Mount an MCP server's tools as session tools:
```ts
import { connectMcpServer } from '@fabric-harness/sdk';
const github = await connectMcpServer('github', {
url: process.env.GITHUB_MCP_URL!,
headers: {
authorization: `Bearer ${process.env.GITHUB_TOKEN}`,
},
allowTools: ['get_*', 'list_*', 'search_*'],
denyTools: ['*delete*'],
});
await session.prompt('Look up project FAB-123', {
tools: github.tools,
});
await github.close();
```
`connectMcpServer()` uses Streamable HTTP by default. Pass `transport: 'sse'` for a server that still exposes the legacy SSE transport. `allowTools` and `denyTools` accept exact names or `*` globs and filter the server catalog before any tool reaches the model. Provider credentials remain in transport or OAuth state and are not added to tool metadata or model context.
The returned connection has `reconnect()` and `close()` lifecycle methods. Prompt and task abort
signals propagate to active MCP calls, including the protocol cancellation notification.
## OAuth client credentials
Use the machine-to-machine helper when the MCP authorization server supports the
`client_credentials` grant. The MCP SDK discovers authorization metadata, obtains a token, and
refreshes it when required.
```ts
import { connectMcpServer, createMcpClientCredentialsAuth } from '@fabric-harness/sdk';
const authProvider = createMcpClientCredentialsAuth({
clientId: process.env.MCP_CLIENT_ID!,
clientSecret: process.env.MCP_CLIENT_SECRET!,
scope: 'tools.read',
});
const connection = await connectMcpServer('enterprise-tools', {
url: process.env.MCP_SERVER_URL!,
authProvider,
allowTools: ['catalog_*'],
});
```
## OAuth authorization code
Authorization code uses PKCE and application-owned state storage. In production, implement
`loadState` and `saveState` with an encrypted server-side session or secret store. Never serialize
this state into a prompt, agent memory, job payload, or client-side transcript.
```ts
import { connectMcpServer, createMcpAuthorizationCodeAuth } from '@fabric-harness/sdk';
const authProvider = createMcpAuthorizationCodeAuth({
redirectUrl: 'https://agents.example.com/oauth/mcp/callback',
clientName: 'TechFabric Harness',
clientId: process.env.MCP_CLIENT_ID!,
clientSecret: process.env.MCP_CLIENT_SECRET,
scopes: ['tools.read'],
onAuthorizationUrl: (url) => redirectUser(url),
loadState: () => encryptedSession.get('mcp-oauth'),
saveState: (state) => encryptedSession.set('mcp-oauth', state),
});
// On the callback request, pass the returned code once. Stored refresh tokens
// are reused by subsequent connections through the same provider state.
const connection = await connectMcpServer('user-tools', {
url: process.env.MCP_SERVER_URL!,
authProvider,
authorizationCode: callbackUrl.searchParams.get('code') ?? undefined,
});
```
## Local stdio server
Use `createStdioMcpClient()` with `createMcpTools()` for a local MCP process:
```ts
import { createMcpTools, createStdioMcpClient } from '@fabric-harness/sdk';
const client = createStdioMcpClient({
command: 'npx',
args: ['-y', '@modelcontextprotocol/server-filesystem', '/workspace'],
});
const tools = await createMcpTools(client, {
prefix: 'workspace',
effect: 'read',
source: 'filesystem',
});
await session.prompt('Summarize the workspace', { tools });
await client.close();
```
Tool names are prefixed and normalized before they enter the model toolset. Apply `toolPolicy` rules to the generated names when an MCP server exposes write or execute effects.
## Why MCP
MCP is becoming a converging standard for tool/resource exchange across agent ecosystems. Wiring Fabric agents and MCP servers together means you can:
- give Fabric agents access to a growing catalog of MCP-served tools,
- use the same Fabric policy and approval controls for local and remote MCP tools,
- switch between remote and stdio transports without changing the session prompt loop,
- expose the same governed job and agent operations to IDEs, automation clients, and agent peers.
---
# Channel integrations
Canonical: https://harness.techfabric.com/docs/ecosystem/channels
Signed event ingress and outbound tools for persistent agents, which is how a webhook becomes a conversation with memory.
Channels verify provider webhooks, normalize an event, derive a stable conversation key, and dispatch it to a persistent Fabric agent. Provider event IDs become dispatch IDs for deduplication, while tenant and actor identity flow into the audit trail.
```sh
pnpm add @fabric-harness/channels @fabric-harness/sdk
```
| Channel | Stable instance key | Support |
| --- | --- | --- |
| [Buzz](/docs/ecosystem/channels/buzz) | Community/channel/thread | First-party preview adapter |
| [Discord](/docs/ecosystem/channels/discord) | Guild/channel/thread | First-party adapter |
| [Facebook Messenger](/docs/ecosystem/channels/messenger) | Page/sender | First-party adapter |
| [GitHub](/docs/ecosystem/channels/github) | Repository/issue or PR | First-party adapter |
| [Google Chat](/docs/ecosystem/channels/google-chat) | Space/thread | First-party adapter |
| [Intercom](/docs/ecosystem/channels/intercom) | Workspace/conversation | First-party adapter |
| [Linear](/docs/ecosystem/channels/linear) | Organization/issue | First-party adapter |
| [Microsoft Teams](/docs/ecosystem/channels/teams) | Tenant/conversation/thread | First-party adapter |
| [Notion](/docs/ecosystem/channels/notion) | Workspace/page or database | First-party adapter |
| [Resend](/docs/ecosystem/channels/resend) | Tenant/email | First-party adapter |
| [Salesforce Marketing Cloud](/docs/ecosystem/channels/salesforce-marketing-cloud) | Business unit/contact or send | First-party adapter |
| [Shopify](/docs/ecosystem/channels/shopify) | Shop/resource | First-party adapter |
| [Slack](/docs/ecosystem/channels/slack) | Team/channel/thread | First-party adapter and example |
| [Stripe](/docs/ecosystem/channels/stripe) | Connected account/customer or payment resource | First-party adapter |
| [Telegram](/docs/ecosystem/channels/telegram) | Bot/chat/topic | First-party adapter |
| [Twilio](/docs/ecosystem/channels/twilio) | Account/sender/recipient | First-party SMS and WhatsApp adapter |
| [WhatsApp](/docs/ecosystem/channels/whatsapp) | Business account/phone/sender | First-party Meta Cloud API adapter |
| [Zendesk](/docs/ecosystem/channels/zendesk) | Account/ticket | First-party adapter |
| Generic webhook | Caller-provided conversation ID | Direct `fh add channel webhook` scaffold |
First-party means Fabric owns the provider adapter, runs it through the shared Node and edge-safe channel contract, and publishes a dependency-aware `fh add channel ` recipe. Buzz is marked preview because its transport depends on the evolving relay protocol; its authentication, replay, identity, and recipe contracts are maintained.
## Package and version policy
Adapters are dependency-free `@fabric-harness/channels/` subpaths implemented with Fetch and
Web Crypto. An adapter becomes a separate package only when a required provider SDK, runtime
incompatibility, or independent release cadence makes that necessary. Import
`channelCompatibility` from `@fabric-harness/channels/compatibility` for the machine-readable API
matrix. Deprecated adapters receive at least 180 days of notice and are removed only in a major
release.
Channels dispatch to `.fabricharness/agents/`, not finite jobs. See [Persistent agents](/docs/building/persistent-agents) for the durable submission model.
---
# Databases
Canonical: https://harness.techfabric.com/docs/ecosystem/databases
Durable session stores and governed database connectivity, covering where an agent's state lives and what it may query.
Fabric uses databases for durable runtime state and exposes separate, policy-gated tools for agent data access. Do not give the model raw database credentials.
| Database | Runtime use | Agent data access |
| --- | --- | --- |
| [libSQL](/docs/ecosystem/databases/libsql) | Unified local or remote bundle through `libsqlPersistence()`. | Scoped tool guide. |
| [MongoDB](/docs/ecosystem/databases/mongodb) | Unified bundle through `mongodbPersistence()`. | First-party collection-bound find tools. |
| [MySQL](/docs/ecosystem/databases/mysql) | Unified bundle through `mysqlPersistence()`. | First-party fixed-statement tools. |
| [Postgres](/docs/ecosystem/databases/postgres) | Unified session, submission, stream, attachment, run/event, and cost persistence through `postgresPersistence()`. | First-party fixed-statement tools. |
| [Redis](/docs/ecosystem/databases/redis) | Project-owned coordination/store implementation. | First-party namespaced get/set tools. |
| [Supabase](/docs/ecosystem/databases/supabase) | Reuse `postgresPersistence()` with the direct Postgres connection. | RLS-aware table/RPC tool guide. |
| [Turso](/docs/ecosystem/databases/turso) | Reuse `libsqlPersistence()` with a remote URL/token. | Parameterized libSQL tool guide. |
| [Valkey](/docs/ecosystem/databases/valkey) | Reuse `redisPersistence()` with a Redis-protocol client. | Namespaced command tool guide. |
| [Databricks Lakebase](/docs/ecosystem/databases/lakebase) | Sessions, submissions, conversation streams, and attachments for Databricks Apps. | Use governed Databricks SQL and Unity Catalog tools separately. |
| [SQLite](/docs/ecosystem/databases/sqlite) | Local durable sessions through Node configuration. | First-party fixed-statement tools. |
See [Session stores](/docs/reference/session-stores) for the complete persistence contract.
Install governed data tools with `pnpm add @fabric-harness/databases` or scaffold a complete provider
module with `fh add database postgres`, `mysql`, `mongodb`, `redis`, or `sqlite`. Postgres and
Lakebase, MySQL, MongoDB, libSQL/Turso, Redis/Valkey, SQLite, and Supabase/Postgres provide durable Fabric stores. Data tools and session persistence remain
separate so model-facing access never receives runtime-store credentials implicitly.
---
# Agent registry and governance
Canonical: https://harness.techfabric.com/docs/building/agent-registry
Register immutable agent versions, bound authority, budgets, and external identities.
`@fabric-harness/agent-registry` provides runtime-neutral governance contracts for agent catalogs.
It does not execute agents or replace Platform actions. It validates definitions, immutable versions,
tenant registrations, capability grants, autonomy ceilings, enrollment, external identities, and
privacy-safe evidence while keeping vertical-specific scope in the owning application.
```sh
pnpm add @fabric-harness/agent-registry zod
```
```ts
import {
agentCapabilityGrantBaseSchema,
agentDefinitionSchema,
clampRoute,
} from '@fabric-harness/agent-registry';
const definition = agentDefinitionSchema.parse({
agentDefinitionId: 'reviewer',
name: 'reviewer',
displayName: 'Governed reviewer',
description: 'Reviews a bounded subject and proposes governed actions.',
inputKinds: ['review.request'],
outputKinds: ['review.result'],
createdAt: new Date().toISOString(),
});
const grant = agentCapabilityGrantBaseSchema.parse({
grantId: 'reviewer-grant',
readTools: ['read-subject'],
proposalActions: ['propose-review'],
executionActions: [],
});
const route = clampRoute('approval-required', {
status: 'auto-executed',
actionInvocationId: 'candidate-invocation',
});
```
The registry deliberately separates read, propose, and execute authority. A registration's autonomy
is a ceiling: policy may narrow it but cannot widen it. Persist only validated, hash-pinned versions;
mint short-lived execution principals per run; and route mutations through governed TechFabric Platform
actions. External channel identities remain untrusted candidates until an application resolves an
active binding and validates the applicable grant.
Use zod `.extend()` in an application package to attach tenant or subject scope. Do not add vertical
domain fields to the shared registry. The runnable credential-free fixture is
`examples/agent-registry` in the restricted source repository.
---
# Mounting agents in your own server
Canonical: https://harness.techfabric.com/docs/building/agent-router
Serve the persistent-agent HTTP surface from an application that already owns its routes, using createFabricAgentRouter.
Harness serves agents from its own runtimes — `fh dev`, the Node build artifact,
the Cloudflare worker. `createFabricAgentRouter()` covers the other direction: an
application that already owns an HTTP surface, and wants Harness agents on it,
under its own paths and behind its own middleware.
```ts
import { createFabricAgentRouter } from '@fabric-harness/sdk';
const router = createFabricAgentRouter(backend, { basePath: '/api' });
const handled = await router.handle(request);
if (handled) return handled;
// …your application keeps everything else.
```
The router is runtime-neutral — `Request`, `Response`, `URL`, and plain objects —
so it runs on Node, Cloudflare, Deno, Bun, or any fetch-capable host, and mounts
in any framework that can hand it a `Request`.
## Two entry points
| Method | Unmatched path | Use when |
| --- | --- | --- |
| `handle(request)` | resolves `undefined` | You compose routes and want to fall through. |
| `fetch(request)` | resolves `404` | Your framework expects a total handler. |
## What the router owns
Everything a client observes: route parsing, method dispatch, request
validation, response shape, status codes, and the `202` admission receipt. The
[streaming protocol](/docs/reference/streaming-protocol) reference documents that
wire contract, and the router is the same implementation the built-in runtimes
follow — mounting agents yourself does not put you on a different protocol.
## What the backend owns
A `FabricAgentRouterBackend` supplies durable submission, conversation reads, and
lifecycle:
```ts
import type { FabricAgentRouterBackend } from '@fabric-harness/sdk';
const backend: FabricAgentRouterBackend = {
async resolveAgent(name) {
return { found: registry.has(name), exposed: true };
},
async admit(input) {
// Capture the stream position BEFORE admitting.
const offset = await streams.nextOffset(input);
const submissionId = await runner.admit(input);
return { submissionId, offset };
},
async readConversation(address) { /* … */ },
async getSubmission(address) { /* … */ },
async abort(address) { /* … */ },
};
```
`loadInstance`, `deleteInstance`, `streamUpdates`, and `handleSchedules` are
optional. A route whose backend method is absent returns `404` rather than a
misleading `405`, and `updatesUrl` appears in the receipt only when
`streamUpdates` exists — the router never points a client at a route the host
does not serve.
## Two invariants worth stating
**Capture the offset before admitting.** The receipt's `offset` is the position
a client resumes from; reading it must return this delivery and the reply.
Returning `"0"` instead makes every client re-read the whole conversation on
every send.
**An unexposed agent must answer like a missing one.** `resolveAgent` returning
`{ found: true, exposed: false }` produces exactly the `404` that
`{ found: false }` does, so probing cannot enumerate private agents.
## Errors
A backend method that throws becomes `500` with a generic body. Storage and
execution details — connection strings, stack traces, internal identifiers —
never reach the caller. Map your own domain failures to responses inside the
backend if you need callers to distinguish them.
## Mount paths
`basePath` is stripped before matching, so the backend never learns where the
router lives. URLs in the receipt are relative to the router, which means the
host composes its own prefix rather than the router guessing at it.
## When not to use this
If you are deploying with `fh build`, you already have this surface — the Node
and Cloudflare targets serve it. Reach for the router when Harness is a
component inside a larger service you own, not when it is the service.
## See also
- [`examples/with-mounted-agent-router`](/docs/reference/source-access) — a runnable version with an in-memory backend.
- [Streaming protocol](/docs/reference/streaming-protocol) — the wire contract this router implements.
- [HTTP applications](/docs/building/http-applications) — adding your own routes to a Harness-owned server instead.
---
# Agent Anatomy
Canonical: https://harness.techfabric.com/docs/building/anatomy
The metadata-first shape of a TechFabric Harness agent — default and strict variants of the same call.
import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
TechFabric Harness finite agents are metadata-first. The canonical default export is `defineAgent({...})`, which provides lazy `prompt`, `skill`, `task`, `shell`, and `session` helpers. The runtime admits these finite definitions as jobs/runs, so they live under `.fabricharness/jobs/` and use `/jobs` routes. Persistent, addressable agents use `createAgent()` under `.fabricharness/agents/`.
- **Default** — `import { defineAgent } from '@fabric-harness/sdk'`. Injects headless defaults (`runtime: 'stateless'`, `sandbox: 'virtual'`, `loopRuntime: pi-agent-core`, `compaction: { enabled: true }`) on every `init()` call. The fast path for prototypes, webhooks, edge agents.
- **Strict** — `import { defineAgent } from '@fabric-harness/sdk/strict'`. Same call shape, **no defaults injected**. Required for Temporal-backed durability (replay determinism) and recommended for compliance/audit workloads.
Both produce the same `AgentDefinition` and run identically through `fh run`, `fh build`, `fh describe`, and any deploy target.
## Recommended finite-agent definition
```ts
import { defineAgent, schema } from '@fabric-harness/sdk';
export default defineAgent({
name: 'ask',
input: schema.object({ question: schema.string() }),
output: schema.string(),
async run({ input, prompt }) {
return prompt(input.question);
},
});
```
## Invoke another finite job
Use the job context's `invoke()` for child work. The configured runtime admits a new run without an HTTP
round trip and propagates the parent run, tenant, and actor. Cycles and nesting beyond 16 jobs are
rejected before admission.
```ts title=".fabricharness/jobs/account-review.ts"
import { defineAgent, schema } from '@fabric-harness/sdk';
export default defineAgent({
input: schema.object({ accountId: schema.string() }),
run: async ({ input, invoke }) => {
const receipt = await invoke({
job: 'collect-account-evidence',
input: { accountId: input.accountId },
idempotencyKey: `evidence:${input.accountId}`,
});
return { evidenceRunId: receipt.runId };
},
});
```
When a definition has a declared or loader-registered name, `invoke(definition, { input })` is also
available. The top-level `invoke()` export remains available in application routes, channels, and
schedules. Use the named form across module boundaries to avoid importing executable job modules.
## Middleware and run identity
Job middleware wraps `run` in declaration order. It receives the same typed context and is suitable
for tracing, shared authorization, metrics, and transaction boundaries. `context.run` contains the
stable run ID, job name, parent chain, tenant, and actor when the job was admitted through a server;
it is absent for a direct handler call in a unit test.
```ts
import { defineAgent, schema } from '@fabric-harness/sdk';
export default defineAgent({
name: 'account-review',
input: schema.object({ accountId: schema.string() }),
middleware: [
async ({ run }, next) => {
console.log({ event: 'started', runId: run?.runId });
const output = await next();
console.log({ event: 'completed', runId: run?.runId });
return output;
},
],
async run({ input, prompt }) {
return prompt(`Review account ${input.accountId}`);
},
});
```
Each middleware may call `next()` once. It may also short-circuit by returning a typed output.
The runtime and default session are initialized on the first helper call. Add `init: { sandbox: 'local' }` to set initialization options, or call `session('review')` when you need a named session.
## Lower-level job definition
```ts
import { defineAgent, schema } from '@fabric-harness/sdk';
export default defineAgent({
name: 'ask',
input: schema.object({ question: schema.string() }),
output: schema.string(),
triggers: { webhook: true },
run: async ({ init, input }) => {
const session = await (await init()).session();
return session.prompt(input.question);
},
});
```
`init()` defaults are injected automatically. Override any of them by passing a value:
`init({ runtime: 'inline', sandbox: 'docker' })`.
```ts
import { defineAgent, schema } from '@fabric-harness/sdk/strict';
export default defineAgent({
name: 'ask',
input: schema.object({ question: schema.string() }),
output: schema.string(),
triggers: { webhook: true },
run: async ({ init, input }) => {
const fabric = await init({
runtime: 'temporal',
sandbox: 'local',
compaction: { enabled: false },
});
const session = await fabric.session();
return session.prompt(input.question);
},
});
```
Every option is declared in source. Nothing implicit. Required for Temporal — auto-compaction would break replay determinism.
The `defineAgent` call shape is identical across both imports. Plain default-exported functions are intentionally rejected because a definition builder keeps jobs discoverable and gives the CLI typed metadata.
Top-level finite-definition `instructions`, `tools`, `policy`, `costBudget`, and approval timeout
are applied to `init()`. A call-provided role overrides definition instructions; definition policies
and budgets remain security floors, while call policy can add denials and approval requirements.
## The `FabricContext`
When the CLI invokes an agent it provides:
```ts
interface AgentRunContext {
payload: TInput;
input: TInput;
init(options?: AgentInit): Promise;
run?: JobInvocationContext;
session(id?: string): Promise;
prompt(text: string): Promise;
skill(name: string): Promise;
task(text: string): Promise;
shell(command: string): Promise;
invoke(request: NamedJobInvocation): Promise;
}
```
Use `input` for typed, schema-validated values inside `run`:
```ts
run: async ({ init, input }) => {
const fabric = await init();
const session = await fabric.session();
return session.prompt(input.question);
}
```
## `init()` options
Agents from either entrypoint call the same `init()` to construct the runtime:
```ts
const fabricAgent = await init({
id: 'agent-1',
model: 'openai/gpt-5.5',
role: 'engineer', // Markdown role file under .fabricharness/roles/
sandbox: 'local', // 'virtual' | 'empty' | 'local' | 'docker' | 'cloudflare' | factory
autonomy: {
mode: 'background',
onMissingInput: 'assume',
onApprovalUnavailable: 'fail',
onCredentialMissing: 'fail',
},
});
```
When using the bare `@fabric-harness/sdk` import you typically omit `runtime`, `sandbox`, and `loopRuntime` — those defaults are injected. Override anything you like; defaults fill the gaps you don't set.
> ⚠️ **Temporal users:** if you set `runtime: 'temporal'` from the bare import, the SDK emits a one-time `console.warn`. Auto-compaction is non-deterministic across Temporal replay. Either switch to `@fabric-harness/sdk/strict` or pass `compaction: { enabled: false }` explicitly.
## Triggers
Declare triggers inside `defineAgent({...})`. The Node and Cloudflare server targets respect:
```ts
export default defineAgent({
name: 'triage',
triggers: {
webhook: true, // POST /jobs/triage
schedule: '*/15 * * * *', // Node scheduler or Cloudflare Cron Trigger
// cli: true, // CLI-only, default true
},
async run(ctx) {
return ctx.input;
},
});
```
## Where to go next
- [SDK entrypoints, runtimes, and targets](/docs/reference/sdk-entrypoints-runtimes-targets) — when to swap to `/strict`.
- [Enterprise controls](/docs/building/enterprise-controls) — definition-level policy, tools, budgets, and approvals.
- [Sessions and prompts](/docs/building/sessions-prompts) — the core call surface.
- [Skills](/docs/building/skills) and [Roles](/docs/building/roles).
- [Tools](/docs/building/tools) and [Commands](/docs/building/commands).
- [Sandboxes](/docs/building/sandboxes).
---
# Approvals
Canonical: https://harness.techfabric.com/docs/building/approvals
Pause an agent mid-run until a person approves a risky action, and resume from the same session state once they do.
Approvals turn an autonomous agent into one that asks for confirmation at risky moments. The framework persists the request, suspends the relevant code path, and resumes when a human (or another system) resolves it.
## Declarative routing — `approvalRules`
Most users want approvals tied to *specific tools*, not scattered through agent code. Declare the rules once in policy and the loop pauses for approval before dispatch:
```ts
const fabric = await init({
policy: {
toolPolicy: {
approvalRules: [
{ pattern: 'submit_*', audience: 'reviewer', reason: 'Review before submission of ${name}' },
{ pattern: 'delete_*', audience: 'project-admin', ttlSeconds: 3600 },
{ pattern: 'finalize_*', audience: 'compliance-team' },
],
},
},
});
```
`audience` is an opaque string id. fabric-harness never decides who an audience maps to — your host application's identity layer (UI, SSO, RBAC) does that mapping. Rules also work on `commandPolicy.approvalRules` for `bash` invocations.
When the agent calls a matching tool, fabric-harness emits `approval_requested` with the audience id, the templated reason, and the TTL. The host UI renders the request to the right humans; on `approval_granted` the loop continues; on denial or TTL it throws. This is sugar over the imperative `session.approval.request()` API — which stays available as the escape hatch for ad-hoc cases.
Approved durable responses expose `ApprovalResponse.grant`. The grant binds the approval id, logical
tool-call id, canonical input digest, executing principal, approvers, decision time, and optional
expiry. Memory, file, SQLite, Postgres, Cloudflare Durable Object, and Temporal paths preserve the
same shape. Denied responses never carry a grant. A crash retry may replay the same bound operation;
different input, call id, or principal fails closed.
## Webhook subscriptions
Agents that should wake on inbound events (event bus, queue, external SaaS webhook) use `defineWebhookSubscription`:
```ts
import { defineAgent, defineWebhookSubscription } from '@fabric-harness/sdk';
export default defineAgent({
name: 'data-validator',
triggers: { webhook: true },
subscriptions: [
defineWebhookSubscription<{ recordId: string }>({
id: 'on-record-created',
events: ['record.created'],
handler: async ({ payload, idempotencyKey, headers }) => {
// ...invoke whatever your host application exposes.
},
}),
],
async run() { /* ... */ },
});
```
`fh dev` and generated Node builds expose each subscription at
`POST /agents//subscriptions/`. Set `FABRIC_HARNESS_WEBHOOK_SECRET` to require an
`X-Fabric-Signature: sha256=` header on every delivery. The server dedupes by
`Idempotency-Key` for at-least-once event buses and emits a `webhook_received` event into the
session log for audit.
`fh subscriptions [agent]` lists registered subscriptions across the workspace.
## Request an approval
`session.approval.request()` returns `true` when the approver allows the action.
It throws a `FabricError` (`APPROVAL_DENIED` or `APPROVAL_REQUIRED`) on denial
or timeout — wrap in try/catch when you want to handle rejection gracefully.
```ts
import { FabricError } from '@fabric-harness/sdk';
try {
await session.approval.request({
reason: 'About to push changes to GitHub',
subject: 'git push origin main',
risk: 'high',
timeoutMs: 24 * 60 * 60 * 1000, // 24h
});
await session.shell('git push origin main');
} catch (error) {
if (error instanceof FabricError && error.code === 'APPROVAL_DENIED') {
throw new Error(`Push blocked: ${error.message}`);
}
throw error;
}
```
The available fields are:
| Field | Type | Description |
|---|---|---|
| `reason` | `string` | Human-readable reason shown to the approver. |
| `subject` | `string?` | Short subject line for UIs. Defaults to `'custom'`. |
| `risk` | `'low' \| 'medium' \| 'high'?` | Drives escalation policy. |
| `timeoutMs` | `number?` | Override the session's default approval timeout. |
| `onApproval` | `ApprovalCallback?` | Per-call approval handler. Falls back to session/agent `onApproval`. |
## Resolve from the CLI
```sh
fh approvals --pending
fh approve --actor preetham
# or
fh reject --actor preetham --reason "Wrong branch"
```
For a deployed Node or Databricks App, use the tenant-scoped remote surface. Omitting the session id
discovers approvals across only the sessions visible to the authenticated tenant:
```sh
fh approvals \
--url "$FABRIC_HARNESS_APP_URL/api" \
--token-env DATABRICKS_OAUTH_TOKEN \
--tenant acme
fh approve \
--url "$FABRIC_HARNESS_APP_URL/api" \
--token-env DATABRICKS_OAUTH_TOKEN \
--tenant acme
```
The `audience` value is a routing label, not an authorization grant. The host must map that label to
an authenticated group or role and grant `approval:write` only to eligible approvers. The server
always enforces tenant isolation and records the authenticated voting principal.
## Notify approvers
Attach `approvalNotificationHandler()` to the agent's `onEvent` callback. It receives requested,
escalated, and resolved events, retries transient failures, deduplicates deliveries, and records
delivery outcomes through the audit hook. Notification failures never resolve the approval.
```ts
import {
approvalNotificationHandler,
defineAgent,
slackApprovalNotifier,
} from '@fabric-harness/sdk';
import { redisApprovalNotificationStore } from '@fabric-harness/node';
const notifyApproval = approvalNotificationHandler({
notifier: slackApprovalNotifier({
webhookUrl: process.env.SLACK_APPROVAL_WEBHOOK_URL!,
}),
baseUrl: 'https://agents.example.com',
store: redisApprovalNotificationStore({
client: redis,
namespace: 'production',
}),
retries: 4,
deadLetter: async (record) => {
await approvalDeadLetters.insert(record);
},
audit: async (record) => {
await auditLog.append('approval_notification', record);
},
onError: (error, notification) => {
logger.error({ error, notificationId: notification.id });
},
});
export default defineAgent({
name: 'release-agent',
onEvent: notifyApproval,
async run({ prompt }) {
return prompt('Prepare the approved release.');
},
});
```
Use `webhookApprovalNotifier()` for an internal workflow service, PagerDuty bridge, or custom
notification worker. Slack and generic webhook destinations receive a bounded public payload: raw
tool input, environment values, actor credentials, and destination secrets are excluded. Deep links
contain only the session, approval, and tenant identifiers; `/admin` still enforces authentication,
tenant isolation, and approval permissions when opened.
`inMemoryApprovalNotificationStore()` is suitable for local development. Use
`redisApprovalNotificationStore()` across replicas or implement `ApprovalNotificationDeliveryStore`
with an atomic claim in the system that owns your notification outbox. A dead-letter callback is
strongly recommended in production.
## Durable waits
On the Temporal worker target, the approval wait runs as a dedicated workflow and can wait without
holding a process. The workflow records its ID on the approval request so `fh approve` and
`fh reject` signal the owning workflow. The first denial wins; approvals are deduplicated by actor
and resume only after `policy.approvals.requiredApprovals` is reached. An escalation deadline emits
`approval_escalated` with the configured notify audience and risk, while the overall timeout remains
authoritative. Worker restarts do not lose the timer, votes, or pending request. On the inline
runtime, the wait uses the configured session store or callback.
To make agents safe on either runtime, declare autonomy fallbacks:
```ts
await init({
autonomy: {
onApprovalUnavailable: 'fail', // or 'assume-rejected' | 'assume-approved'
},
});
```
## Logged identities
Approvals record both the agent identity and the human actor. Combined with the two-identity actor schema (Entra Agent ID + on-behalf-of user), the audit trail answers "which agent acted, on whose behalf, who approved?"
## See also
- [Policies and approvals](/docs/reference/policies-approvals)
- [`fh approvals`](/docs/cli/approvals)
---
# Artifacts
Canonical: https://harness.techfabric.com/docs/building/artifacts
Publish files that belong to a session so a later step, another agent or a human reviewer can pick them up by reference.
An **artifact** is a session-bound file. Agents publish artifacts as the durable, reviewable output of their work — Markdown reports, JSON results, CSVs, images.
## Publish an artifact
```ts
await session.artifact('report.md', markdown, { contentType: 'text/markdown' });
```
The content can be a string or an in-memory buffer:
```ts
await session.artifact('summary.json', JSON.stringify({ ok: true }), {
contentType: 'application/json',
});
```
Agents that generate a file in the sandbox first can read it back and publish the bytes:
```ts
const bytes = await session.fs.readBytes('/workspace/report.md');
await session.artifact('report.md', bytes, { contentType: 'text/markdown' });
```
## Read from the CLI
```sh
fh artifacts
fh artifact get report.md --out ./reports/report.md
```
## What gets persisted
The session store records artifact metadata (id, name, content type, byte size, created time) alongside the session entry that produced it. The bytes are stored separately:
- **File store** (default): files under `.fabricharness/sessions//artifacts/`.
- **SQLite store**: BLOB column.
- **Postgres store**: BYTEA column or external blob URL.
- **Cloud stores** (designed): Azure Blob / ADLS, S3, R2.
## Patterns
- **Reports.** A triage agent publishes `triage-report.md` for reviewers.
- **Generated code.** A migration agent publishes the proposed diff as an artifact and asks for approval before applying it.
- **Datasets.** A data agent publishes profiled CSVs alongside the prompt that produced them.
See also: [`fh artifacts`](/docs/cli/artifacts), [`fh artifact get`](/docs/cli/artifacts).
---
# Channels
Canonical: https://harness.techfabric.com/docs/building/channels
Turn a signed platform webhook from Slack, GitHub or Stripe into an agent dispatch, without writing an ingress service yourself.
Channels are how agents get **triggered** by the outside world. A webhook hits `/channels/:name/*`,
the handler verifies the signature, normalizes the platform event, and **dispatches** to a persistent
agent keyed by a stable conversation id. Outbound actions are tools bound at agent init.
The core seam lives in `@fabric-harness/sdk`; 17 vendor adapters live in
`@fabric-harness/channels` behind subpath exports, so platform SDK code stays out of core. Handlers
are written against the Web `Request`/`Response` API and `crypto.subtle`, so the same channel runs on
Node and Cloudflare.
## Why channels
- **Stateless.** A channel is a route container plus a conversation-id (de)serializer. Session
continuity falls out of the key — the same Slack thread → same key → same session.
- **Exactly-once.** The platform event id (Slack `event_id`, GitHub `X-GitHub-Delivery`) becomes the
`dispatchId`, so a webhook redelivery collapses to a single agent turn via the persistent-run
idempotency marker.
- **Identity propagation.** The platform user becomes the dispatch `actor` and the team/org becomes
the `tenantId` — flowing into the audit trail and into on-behalf-of governance (e.g. Databricks UC).
- **Policy-gated outbound.** Outbound actions are `defineTool` tools (`effect: 'write'`), so the
agent's `CapabilityPolicy` can gate "can this agent post to Slack".
## Slack
A channel lives in `.fabricharness/channels/.ts` and exports a `channel`:
```ts title=".fabricharness/channels/slack.ts"
import { createSlackChannel } from '@fabric-harness/channels/slack';
export const channel = createSlackChannel({
signingSecret: process.env.SLACK_SIGNING_SECRET!,
agent: 'assistant', // dispatch app mentions / threaded messages here
});
```
The dev server mounts it at `POST /channels/slack/events`. Point your Slack app's **Event
Subscriptions → Request URL** there and subscribe to `app_mention`. The handler verifies the `v0`
HMAC signature, answers the URL-verification challenge, and dispatches the event keyed by the thread.
Bind the reply tool to the thread at agent init — the instance id *is* the thread key:
```ts title=".fabricharness/agents/assistant.ts"
import { createAgent } from '@fabric-harness/sdk';
import { parseSlackConversationKey, replyInSlackThread } from '@fabric-harness/channels/slack';
export default createAgent(({ id }) => {
const thread = parseSlackConversationKey(id);
return {
model: 'anthropic/claude-haiku-4-5',
tools: [replyInSlackThread(thread, { botToken: process.env.SLACK_BOT_TOKEN! })],
};
});
```
## GitHub
Same shape, different signature scheme (`X-Hub-Signature-256`) and key (`owner/repo//`):
```ts title=".fabricharness/channels/github.ts"
import { createGitHubChannel } from '@fabric-harness/channels/github';
export const channel = createGitHubChannel({
secret: process.env.GITHUB_WEBHOOK_SECRET!,
agent: 'triage',
});
```
Mounted at `POST /channels/github/webhook`. It acknowledges `ping`, dispatches issue / PR / comment
events (with `X-GitHub-Delivery` as the dedupe key, the owner as tenant, the sender as actor), and
ignores bot senders to avoid loops. Outbound: `commentOnGitHubIssue(ref, { token })`.
## Other first-party channels
Scaffold the provider module, environment template, dependency set, and starter test:
```sh
fh add channel discord
fh add channel teams
fh add channel telegram
fh add channel twilio
fh add channel whatsapp
fh add channel google-chat
fh add channel linear
fh add channel stripe
fh add channel zendesk
```
Each first-party adapter verifies the provider's native request authentication, assigns the provider
event ID as `dedupeKey`, derives a stable conversation key, and propagates tenant/actor identity.
Provider setup and outbound tool examples are in the [channel catalog](/docs/ecosystem/channels).
Microsoft Teams includes Bot Connector key discovery and validates issuer, App ID audience, token
lifetime, service URL, and channel endorsement without requiring an application JWT library.
The subpath exports include typed webhook payloads and conversation refs for custom routing. The
built-in normalization covers the event families an agent commonly needs:
| Provider | Normalized events |
| --- | --- |
| Slack | mentions, messages, edited messages |
| GitHub | opened, edited, reopened, synchronized, and issue/PR comments |
| Discord | commands and component callbacks |
| Teams | messages, message updates, and deletes |
| Telegram | messages, edited messages, channel posts, and callback queries |
| Twilio | SMS/WhatsApp messages and delivery statuses |
| WhatsApp | text messages and delivery/read statuses |
| Google Chat | messages and interactions |
| Linear | issue and project resource changes |
| Notion | workspace entity changes |
| Stripe | account and financial resource events |
| Zendesk | ticket events |
| Intercom | conversation and ticket notifications |
| Shopify | versioned commerce webhooks |
| Messenger | messages and postbacks |
| Resend | inbound and delivery email events |
| Salesforce Marketing Cloud | signed ENS event batches |
Delivery-status dedupe keys include both message ID and status, so a `sent` event cannot suppress a
later `delivered` or `read` event.
## Authoring your own channel
A channel is `defineChannel({ routes, conversationKey, parseConversationKey })`. The SDK provides the
building blocks: `verifyHmacSha256` (constant-time), `hexToBytes`, `conversationKey` /
`parseConversationKey` (url-safe), and `readJsonBody` (raw bytes for HMAC **and** parsed JSON from a
single read). Inside a route handler, call `ctx.dispatch(agent, { instanceId, input, dedupeKey,
tenantId, actor })`.
## See also
- Example: [`examples/with-slack-channel`](/docs/reference/source-access)
— Slack mention → agent → in-thread reply, end to end.
- Example: [`examples/with-channel-adapters`](/docs/reference/source-access)
— all 18 verified ingress adapters and their governed outbound tools in one runnable workspace.
- [Persistent agents](/docs/building/persistent-agents) — channels dispatch to `createAgent(...)` instances.
- [Triggers](/docs/reference/triggers) — gating which agents accept inbound events.
---
# Commands and Capabilities
Canonical: https://harness.techfabric.com/docs/building/commands
Scope which shell commands an agent may run and which secrets it can see, per session rather than per deployment.
A **command** in TechFabric Harness is a shell-level capability you explicitly grant to a session — not a generic "run anything" door. Use `defineCommand` from `@fabric-harness/sdk` to declare one.
## Declaring commands
```ts
import { defineCommand } from '@fabric-harness/sdk';
const npm = defineCommand('npm');
const git = defineCommand('git');
const gh = defineCommand('gh', {
env: {
GH_TOKEN: process.env.GH_TOKEN,
},
});
```
A `defineCommand` declaration captures:
- the binary name,
- environment variables to inject,
- working directory defaults,
- timeouts and stdin handling.
## Granting commands per call
```ts
await session.prompt('Fix the failing tests', {
commands: [npm, git, gh],
});
```
The model can only run shell commands whose binary matches one of the declared commands. Anything else fails with a capability error.
## Secrets
Use `secret()` to mark a value as a credential reference. The runtime resolves it at exec time and never echoes it into model context. Exported from `@fabric-harness/sdk`.
```ts
import { defineCommand, secret } from '@fabric-harness/sdk';
const gh = defineCommand('gh', {
env: { GH_TOKEN: secret('GH_TOKEN') },
});
```
`secret()` is intentionally a token, not the value: it can be passed around without leaking its content into logs, traces, or the LLM context.
## Capability policy
Use `policy` to enforce filesystem, command, tool, network, timeout, and approval rules. Policies can be set on `init()`, a session, or an individual prompt; narrower call-level policy is combined with definition-level controls.
```ts
await session.prompt('Fix the tests', {
policy: {
filesystem: {
read: ['/workspace/**'],
write: ['/workspace/src/**', '/workspace/tests/**'],
},
commandPolicy: {
allow: ['npm test', 'git diff', 'git status'],
requireApproval: ['git push', 'npm publish', 'terraform apply'],
},
network: {
mode: 'allowlist',
hosts: ['api.github.com'],
},
maxCommandTimeoutMs: 120_000,
},
});
```
The runtime applies policy at both the tool layer and the sandbox boundary. See [Policies and approvals](/docs/reference/policies-approvals) for deny rules, approval routing, and composition behavior.
---
# Connector catalog
Canonical: https://harness.techfabric.com/docs/building/connector-catalog
Sandbox, MCP, knowledge-base, data, Azure, and Databricks connector options.
TechFabric Harness connectors are intentionally modular. The core SDK owns the stable agent/session/tool/sandbox contracts; provider packages and project-local adapters own provider SDKs, credentials, lifecycle, and live compatibility.
There are three connector forms:
| Form | Use when | Example |
|---|---|---|
| Package helper | Fabric already ships a dependency-free helper or REST client. | `daytonaSandbox`, `databricksSqlTool`, `foundryAgentTool` |
| Recipe via `fh add` | Provider SDK shape or project conventions need local code. | `fh add daytona --print` |
| Direct SDK primitive | No connector file is needed. | `fumadocsSource`, `connectMcpServer` |
Discover scaffold and guide recipes from one catalog:
```sh
fh add
fh add --json
```
The JSON form returns stable IDs such as `scaffold:channel:slack` and `guide:mcp:github-mcp`, plus the exact command for each entry.
Catalog entries also include declared dependencies. Direct scaffolds install missing dependencies with the detected package manager; use `--no-install` for manifest-only changes. Guide recipes remain side-effect free unless `--install-deps` is supplied.
## What `fh add` installs
Named recipes either add a maintained package adapter or generate a project-local connector that uses
the provider SDK directly. The command preview lists every dependency and file before installation:
```sh
fh add slack --print
fh add notion-channel --print
fh add --json
```
Use the [ecosystem catalog](/docs/ecosystem) to open the provider-specific setup, identity mapping,
webhook verification, policy, and validation instructions.
## Quickstart for any provider
If no pre-built adapter exists, use `fh add --category ` and pipe to your coding agent:
```sh
fh add https://e2b.dev --category sandbox | claude
fh add https://api.notion.com --category data | cursor-agent
fh add https://docs.linear.app --category mcp | codex
```
The CLI emits the canonical category spec (`sandbox.md`, `mcp.md`, or `data.md` shipped with `@fabric-harness/sdk`) along with a "build me a connector for this URL" header. The agent reads the provider's docs, follows the spec literally, and writes a single TypeScript file at `./connectors/.ts`. No PR to fabric-harness needed — the file lives in your project.
When fabric-harness ships a first-party recipe for the provider, prefer the named slug:
```sh
fh add daytona | claude
fh add linear-mcp | claude
```
### Auto-PR mode
`fh add daytona --pr` opens a draft GitHub PR with a stub connector file (committed to a `connector/` branch) and the canonical spec embedded in the PR body. You (or your coding agent) push commits that flesh out the stub. Requires the `gh` CLI and a configured remote.
```sh
fh add daytona --pr
fh add https://your-provider.example.com --category sandbox --pr --pr-branch=connector/your-provider
```
### Validating connectors
`fh doctor --connectors` walks `connectors/` and `.fabricharness/connectors/` and flags files that are missing exports, lack the `@fabric-harness/*` import, or are otherwise malformed:
```sh
fh doctor --connectors
# ✓ connector:daytona.ts: ./connectors/daytona.ts
# ✗ connector:broken.ts: missing @fabric-harness/* import
# ✓ connectors: 1/2 healthy
```
## Install packages
Core connector helpers:
```sh
npm install @fabric-harness/sdk @fabric-harness/connectors
```
Cloud-specific packages:
```sh
npm install @fabric-harness/azure
npm install @fabric-harness/databricks
```
Provider SDKs stay in your app, not in `@fabric-harness/sdk`:
```sh
npm install @daytona/sdk
npm install @e2b/code-interpreter
```
## Sandbox connectors
Sandbox connectors adapt remote execution providers to Fabric's `SandboxEnv` contract: file operations, shell execution, cwd scoping, cleanup, and optional snapshots.
| Connector | Package helper | Recipe | Validation | Notes |
|---|---|---|---|---|
| Daytona | `daytonaSandbox()` / `daytonaSandboxFactory()` from `@fabric-harness/connectors` | `fh add daytona` | `FABRIC_DAYTONA_TEST=1` | Wraps an initialized Daytona sandbox. |
| E2B | `e2bSandbox()` | `fh add e2b` | `FABRIC_E2B_TEST=1` | Maps E2B command and file APIs. |
| Modal | `modalSdkSandbox()` | `fh add modal` | `FABRIC_MODAL_TEST=1` | Native Modal TypeScript SDK process/filesystem adapter. |
| Vercel Sandbox | `vercelSandbox()` | `fh add vercel` | `FABRIC_VERCEL_TEST=1` | Native SDK command/filesystem adapter with reconnect and streamed output. |
| Custom provider | `remoteSandbox()` / `remoteSandboxEnv()` | `fh add --category sandbox --print` | `validateSandboxAdapter(env)` | Keep provider SDK and credentials in project code. |
### Daytona example
```ts
import { Daytona } from '@daytona/sdk';
import { daytonaSandbox } from '@fabric-harness/connectors';
const client = new Daytona({ apiKey: process.env.DAYTONA_API_KEY });
const remote = await client.create();
const fabric = await init({
sandbox: daytonaSandbox(remote, { cleanup: true }),
});
```
### Validate any sandbox adapter
```ts
import { validateSandboxAdapter } from '@fabric-harness/connectors';
await validateSandboxAdapter(env);
```
The validator runs a smoke contract: `mkdir`, `writeFile`, `readFile`, `exec`, and `rm`.
See [Sandbox connectors](/docs/building/sandbox-connectors) for the full adapter guide.
## MCP connectors
MCP connectors expose remote tools to agents. Use `connectMcpServer(name, options)` from `@fabric-harness/sdk` (or `/strict` if you prefer).
| Connector | Recipe | Pattern |
|---|---|---|
| GitHub MCP | `fh add github-mcp --print` | Hosted MCP with bearer token. |
| Mintlify MCP | `fh add mintlify-mcp --print` | Streamable HTTP MCP endpoint for a docs site when available. |
| Linear MCP | `fh add linear-mcp --print` | Hosted MCP with API token. |
| Slack MCP | `fh add slack-mcp --print` | Hosted or self-hosted bridge. |
| Custom MCP | `fh add --category mcp --print` | Reads provider docs and generates a connector. |
Example:
```ts
import { connectMcpServer } from '@fabric-harness/sdk';
const github = await connectMcpServer('github', {
url: 'https://mcp.github.com/mcp',
transport: 'streamable-http',
headers: { Authorization: `Bearer ${process.env.GITHUB_TOKEN}` },
});
try {
const session = await (await init({ tools: github.tools })).session();
await session.prompt('Find open issues tagged bug.');
} finally {
await github.close();
}
```
## Knowledge-base connectors
Knowledge-base connectors mount docs/content into a sandbox as files. Agents then use `read`, `grep`, and `glob` instead of a retrieval pipeline.
| Source | Helper | Recipe | Notes |
|---|---|---|---|
| Local Markdown/MDX | `localDirectorySource()` | none | General local docs. |
| Fumadocs | `fumadocsSource()` | `fh add fumadocs --print` | Strips frontmatter by default. |
| Mintlify local checkout | `mintlifySource()` | none | Includes `docs.json` nav. |
| HTTP docs | `httpFilesystemSource()` | none | Fetch URLs into files. |
| Cloudflare R2 | `r2FilesystemSource()` from `@fabric-harness/sdk/cloudflare` | docs example | Worker/R2 deployments. |
| S3-compatible | `s3FilesystemSource()` from `@fabric-harness/connectors` | none | Structural object-store client. |
| Azure Blob | `azureBlobFilesystemSource()` from `@fabric-harness/connectors` | none | Structural blob client. |
Example:
```ts
import { defineAgent, fumadocsSource, withFilesystemSources } from '@fabric-harness/sdk';
export default defineAgent<{ question: string }>({
run: async ({ init, input }) => {
const sandbox = withFilesystemSources('virtual', [{
mountAt: '/workspace/docs',
source: fumadocsSource('./apps/docs/content/docs'),
}]);
const session = await (await init({ sandbox })).session();
return await session.prompt(`Answer using /workspace/docs: ${input.question}`);
},
});
```
## Data connectors
Data connectors should expose narrow tools, not raw credentials or arbitrary network access.
| Connector | Package/helper | Recipe | Notes |
|---|---|---|---|
| Postgres | project-local `ToolDef`/command | `fh add postgres --print` | Prefer read-only credentials and row/time limits. |
| Notion | hosted MCP or REST command | `fh add notion --print` | Hosted MCP when available; REST adapter otherwise. |
| Databricks SQL | `databricksSqlReadTool()` / `databricksSqlTool()` | see [Databricks](/docs/deployment/databricks) | SELECT-only reads by default; broader statements require an exact allowlist or server-side validator. |
| Databricks Jobs | `databricksRunJobTool()` | see [Databricks](/docs/deployment/databricks) | Trigger existing jobs; requires a `runPolicy` allowlist. |
| Databricks Lakeflow | `databricksLakeflowTools()` | see [Databricks](/docs/deployment/databricks) | List/status by default; start/stop require a pipeline allowlist. |
| Databricks AI Functions | `databricksAiQueryTool()` | see [Databricks](/docs/deployment/databricks) | In-warehouse inference against explicitly allowed serving endpoints. |
| Unity Catalog | `unityCatalogTablesTool()` | see [Databricks](/docs/deployment/databricks) | Read catalog/table metadata. |
| MLflow | `databricksMlflowLogMetricTool()` / `databricksMlflowLogParamTool()` | see [Databricks](/docs/deployment/databricks) | Write telemetry only to construction-time allowed run IDs. |
## Azure and Foundry connectors
Azure helpers live in `@fabric-harness/azure`:
| Helper | Purpose |
|---|---|
| `AzureOpenAIModelProvider` | Azure OpenAI model calls. |
| `createAzureBlobArtifactStore()` | Blob-backed artifacts. |
| `createAzureKeyVaultSecretResolver()` | Runtime secret resolution. |
| `FoundryAgentServiceClient` | Foundry Agent Service REST client. |
| `foundryAgentTool()` | Invoke a Foundry agent from a Fabric agent. |
| `foundryAgentLifecycleTools()` | Create/update/delete Foundry agents as gated tools. |
| `azureContainerAppsJobTool()` | Start existing Container Apps Jobs. |
| `azureAksRunCommandTool()` | AKS Run Command. |
| `azureContainerInstanceExecTool()` | ACI exec-session creation. |
See [Azure](/docs/deployment/azure) and [Foundry Hosted Agents](/docs/deployment/foundry-hosted-agent).
## `fh add`
List recipes:
```sh
fh add
```
Print one:
```sh
fh add daytona --print
fh add github-mcp --print
fh add postgres --print
```
Generate from provider docs:
```sh
fh add https://example.com/provider/docs --category sandbox --print
```
Categories:
| Category | Output shape |
|---|---|
| `sandbox` | `SandboxEnv` / `SandboxFactory` |
| `mcp` | `connectMcpServer()` wrapper |
| `kb` | `FilesystemSource` |
| `data` | `ToolDef`, `Command`, or MCP wrapper |
## Live tests
Connector live tests are opt-in. See [Live tests](/docs/reference/live-tests) for the full matrix.
```sh
FABRIC_DAYTONA_TEST=1 DAYTONA_API_KEY=... pnpm --filter @fabric-harness/connectors test
FABRIC_AZURE_FOUNDRY_TEST=1 AZURE_FOUNDRY_PROJECT_ENDPOINT=... AZURE_TOKEN=... pnpm --filter @fabric-harness/azure test
FABRIC_DATABRICKS_TEST=1 DATABRICKS_HOST=... DATABRICKS_TOKEN=... pnpm --filter @fabric-harness/databricks test
```
## Security checklist
- Keep provider credentials in env, managed identity, Key Vault, or secret manager.
- Never put tokens in prompts, payloads, session history, artifacts, or build manifests.
- Prefer read-only credentials for data tools.
- Treat sandbox execution, cloud control-plane actions, and data writes as `execute`/`write` effects.
- Gate destructive actions with Fabric policy approvals.
- Add unit tests with fake provider objects and live tests behind env gates.
---
# Dynamic Agents and Hooks
Canonical: https://harness.techfabric.com/docs/building/dynamic-agents
Build persistent agents whose instructions, tools, model, integrations, and environment evolve from durable state.
Dynamic agents use the existing `createAgent()` persistent-agent builder. Hooks compose capabilities
inside its synchronous render function; no additional builder or runtime mode is required. The host
renders the function before every delivered interaction using the instance's durable state snapshot.
```ts title=".fabricharness/agents/assistant.ts"
import {
createAgent,
schema,
useModel,
usePersistentState,
useSandbox,
useSkill,
useTool,
} from '@fabric-harness/sdk';
import { reviewChecklist } from '../skills/review';
import { advancedAnalysis, searchIssues } from '../tools/github';
function TriageAgent() {
const [phase] = usePersistentState('phase', 'triage', {
schema: schema.enum(['triage', 'implementation']),
});
useModel(phase === 'triage' ? 'anthropic/claude-haiku-4-5' : 'anthropic/claude-sonnet-4-6');
useSandbox(phase === 'triage' ? 'virtual' : 'docker', { cwd: '/workspace' });
useSkill(reviewChecklist);
useTool(searchIssues);
if (phase === 'implementation') useTool(advancedAnalysis);
return `Complete the ${phase} phase, use the mounted evidence,
and do not skip verification.`;
}
export default createAgent(TriageAgent, {
durability: { maxAttempts: 5, timeoutMs: 2 * 60 * 60_000 },
});
```
The named function is the agent's capability render and its return value is the current instruction.
State changes become visible on the next render, so the model, sandbox, and conditional tool switch at
an interaction boundary. Resource additions, removals, and definition changes are recorded and narrated
to the model. Native-loop hosts resolve the provider that owns the newly selected model; a custom host
can supply `modelProviderResolver` when routing cannot be derived from the standard provider registry.
Declare retry and wall-clock durability outside the render so admission and recovery can enforce it even
when the agent cannot render:
```ts
export default createAgent(renderAssistant, {
durability: { maxAttempts: 5, timeoutMs: 2 * 60 * 60_000 },
});
```
Static definition fields—including `policy`, durability, triggers, and initial-data validation—remain
in force across every dynamic re-render. Hook output may change per turn, but a refresh cannot silently
remove the agent's static authorization boundary.
## The render model
Think of the `createAgent()` function as a pure capability render:
1. Fabric loads the addressed instance and its durable state.
2. The function declares the model, instructions, resources, and lifecycle callbacks for this turn.
3. Fabric runs the model and tools under the normal policy, identity, budget, and cancellation limits.
4. Successful state writes are persisted and affect the next render.
Hooks must be called synchronously and in composition order. Network calls and other asynchronous work
belong in tools, lifecycle callbacks, or lazy MCP connectors. State cannot be written during render.
Custom hooks are ordinary functions, so related capabilities can live together:
```ts
function useResearchMode(enabled: boolean) {
if (!enabled) return;
useInstruction('Investigate competing explanations before deciding.');
useTool(searchSources);
useSkill(researchSkill);
}
export default createAgent(() => {
const [researchEnabled] = usePersistentState('researchEnabled', false);
useResearchMode(researchEnabled);
return 'Resolve the request and explain the evidence.';
});
```
## Complete hook surface
| Hook | Purpose |
| --- | --- |
| `useInstruction()` | Compose instruction fragments in call order. |
| `useModel()` | Select the model, thinking level, and compaction policy for this render. |
| `useSandbox()` | Select a portable sandbox and working directory. |
| `useTool()` | Mount a typed tool, including conditional and durable tools. |
| `useSkill()` | Mount Markdown-first expertise without granting authority. |
| `useSubagent()` | Add a named specialist for delegated session tasks. |
| `useMcpConnection()` | Lazily connect an MCP tool source for this interaction. |
| `usePersistentState()` | Read a durable snapshot and write from tools or lifecycle callbacks. |
| `useDelivery()` | Read the normalized user or signal delivery that caused this render. |
| `useInitialData()` | Read immutable JSON data captured on first contact. |
| `useDispatchMessage()` | Enqueue another real delivery for the same addressed instance. |
| `useDataWriter()` | Write a named, model-invisible client data part. |
| `useResponseStart()` | Attach synchronous metadata before model work begins. |
| `useAgentStart()` | Run awaited intake work after input is durable. |
| `useAgentFinish()` | Inspect the result and optionally continue the same response. |
| `useResponseFinish()` | Attach final synchronous response metadata. |
`defineMcpConnection()` and `defineSubagent()` create reusable declarations.
`GeneralSubagent` supplies a general-purpose delegate when a custom specialist is unnecessary.
## Models, instructions, skills, and sandboxes
Every render can select a different model or environment. A persistent state transition is a useful
way to trade up only when a task earns the additional cost or isolation:
```ts
export default createAgent(() => {
const [phase] = usePersistentState('phase', 'triage', {
schema: schema.enum(['triage', 'implementation']),
});
useModel(
phase === 'triage' ? 'anthropic/claude-haiku-4-5' : 'anthropic/claude-sonnet-4-6',
{ thinkingLevel: phase === 'triage' ? 'low' : 'high' },
);
useSandbox(phase === 'triage' ? 'virtual' : 'local', { cwd: '/workspace' });
useInstruction(`Current workflow phase: ${phase}.`);
useSkill({ name: 'release', content: releaseInstructions });
return 'Complete the current phase without skipping its verification gate.';
});
```
Capability policy still applies after composition. A skill adds context; it does not widen filesystem,
shell, network, credential, connector, or tool permissions.
`useModel()` also takes the per-render compaction policy. The default `@fabric-harness/sdk` import
enables auto-compaction; disable it or tune the thresholds when a phase needs verbatim history:
```ts
useModel('anthropic/claude-sonnet-4-6', {
compaction: { enabled: true, keepRecentEntries: 40 },
});
// or: compaction: false
```
See [Compaction](/docs/reference/compaction) for the full `CompactionSettings` contract and the
`/strict` default.
## Durable state and tools
State setters work inside tracked tools and lifecycle callbacks. A successful tool commits its state
writes with the tool result. Failed tools do not publish a partial state transition.
Mark a tool `durable: true` to receive a named step journal:
```ts
const [phase, setPhase] = usePersistentState('phase', 'draft', {
schema: schema.enum(['draft', 'published']),
});
useTool({
name: 'publish_report',
input: schema.object({ reportId: schema.string() }),
durable: true,
async run({ data, step, log }) {
const receipt = await step.do(`publish:${data.reportId}`, () =>
publisher.publish(data.reportId, { idempotencyKey: data.reportId }),
);
log.info('Report published', { reportId: data.reportId });
setPhase('published');
return { output: receipt };
},
});
```
If a worker stops after one step, recovery re-enters the logical tool call, replays completed steps,
and executes unfinished ones. External operations still need a stable idempotency key because a process
can stop after the external effect but before its checkpoint lands.
On restart, resource-change narration may be recorded before recovery finishes. Harness keeps that
narration out of the provider transcript until the recovered tool result is paired with its original
tool call. If reauthorization or execution fails, the correlated failure is projected as the required
tool result, so Anthropic-compatible and other strict providers never receive an orphaned tool use.
The recovered call/result pair belongs to the resumed response and is visible to `useAgentFinish()`,
preventing required-tool guards from scheduling the same external operation again.
Lifecycle metadata, message data, and lifecycle markers retain their submission and attempt identity
even when the generated host bundles an agent definition separately from the submission runner.
Set `harness: true` on a hook-authored tool only when it needs the runtime-scoped sandbox:
```ts
useTool({
name: 'inspect_workspace',
harness: true,
async run({ harness }) {
return { output: await harness.sandbox.readdir('.') };
},
});
```
## Delivery, initial data, and dispatch
`useDelivery()` exposes the normalized event that triggered the render. `useInitialData()` exposes the
first JSON seed admitted for the instance and never replaces it with later values.
```ts
interface AccountSeed {
accountId: string;
plan: 'standard' | 'enterprise';
}
export default createAgent(() => {
const delivery = useDelivery();
const account = useInitialData();
const dispatch = useDispatchMessage();
useAgentStart(async () => {
if (delivery.kind === 'user' && account.plan === 'enterprise') {
await dispatch({
kind: 'signal',
type: 'account.priority',
body: `Prioritize account ${account.accountId}.`,
});
}
});
return `Handle this ${delivery.kind} delivery for ${account.accountId}.`;
}, {
initialData: schema.object({
accountId: schema.string(),
plan: schema.enum(['standard', 'enterprise']),
}),
});
```
Signal deliveries remain typed canonical conversation records with their submission, tenant, and
actor correlation intact. They render into model context once using their signal tag, but do not
become visible user prompts; the conversation projection classifies them as diagnostic dispatch
messages. Durable replay recognizes the correlated signal by submission ID and never applies it a
second time. The runtime rejects any contradictory typed delivery and rendered prompt before either
is persisted. User deliveries continue to persist as visible user prompts.
Self-dispatch is bound to the host's durable submission queue for each render. This scoped binding
survives packaged agents that bundle their own SDK copy and process restarts; Node, Databricks Apps,
and Cloudflare Durable Objects inject it automatically, while Temporal activities accept the
durable queue explicitly. None depend on process-global queue state inside the agent module.
Seed the first interaction over HTTP:
```sh
curl -X POST 'http://localhost:4317/agents/assistant/account-7?wait=true' \
-H 'content-type: application/json' \
-d '{"message":"Begin","initialData":{"accountId":"account-7","plan":"enterprise"}}'
```
Or pass the same option through `@fabric-harness/client` or `@fabric-harness/react`:
```ts
await client.agent({ agent: 'assistant', id: 'account-7' }).send('Begin', {
initialData: { accountId: 'account-7', plan: 'enterprise' },
});
```
Do not put credentials in initial data, persistent state, tool results, metadata, or data parts. These
surfaces are serializable and may be retained, projected to clients, or replayed.
## Response metadata, client data, and lifecycle
The response lifecycle runs in this order:
```text
useResponseStart → useAgentStart → model and tools → useAgentFinish → useResponseFinish
```
All `useAgentStart()` declarations run concurrently. Each receives the interaction `signal`, a scoped
`harness`, a redacted progress `log`, and `append()`. Fabric commits their state writes and signal
appends together after every callback settles, flattening appends in declaration order. Put dependent
work in one callback rather than depending on callback scheduling. `useAgentFinish()` declarations run
sequentially at the would-stop boundary.
`useDataWriter()` produces typed client-facing data without adding it to model context. Its names are
part of the message shape and must be declared unconditionally on every render.
```ts
const writeProgress = useDataWriter('progress', {
schema: schema.object({ completed: schema.number(), total: schema.number() }),
});
useResponseStart(() => ({ startedAt: Date.now() }));
useTool({
name: 'complete_item',
run: () => {
writeProgress({ completed: 3, total: 8 });
return 'recorded';
},
});
useResponseFinish(({ metadata, response }) => ({
startedAt: metadata.startedAt,
toolCalls: response.toolCalls.length,
finishedAt: Date.now(),
}));
```
Use `useAgentFinish()` for an invariant that must hold before a response settles:
```ts
useAgentFinish(({ response, append }) => {
const verified = response.toolCalls.some(
(call) => call.tool === 'verify_release' && !call.error,
);
if (!verified) {
append({
kind: 'signal',
type: 'release.verification-required',
body: 'Run verify_release before finishing.',
});
}
});
```
The signal continues the same response rather than creating a new user delivery. Finish continuations
are capped at eight cycles and remain subject to model-turn, tool-call, token, cost, cancellation, and
wall-clock limits.
## MCP connections and Databricks
MCP declarations are lazy. Fabric connects only when the current render mounts the declaration, applies
the optional remote-tool allowlist, and closes the connection after the interaction.
```ts
const documentation = defineMcpConnection({
name: 'documentation',
url: 'https://docs.example.com/mcp',
auth: () => process.env.DOCS_MCP_TOKEN,
tools: ['search', 'read'],
});
export default createAgent(() => {
const [researchEnabled] = usePersistentState('researchEnabled', false);
if (researchEnabled) useMcpConnection(documentation);
return 'Use approved documentation sources when research mode is active.';
});
```
Databricks managed MCP uses the same hook through the existing governed connector:
```ts
import { connectDatabricksManagedMcpServer } from '@fabric-harness/databricks';
import { createAgent, defineMcpConnection, useMcpConnection } from '@fabric-harness/sdk';
const catalog = defineMcpConnection({
name: 'catalog',
connect: () => connectDatabricksManagedMcpServer({
host: process.env.DATABRICKS_HOST!,
tokenProvider: async () => process.env.DATABRICKS_TOKEN,
server: {
name: 'catalog',
endpoint: { kind: 'functions', catalog: 'main', schema: 'agent_tools' },
defaultEffect: 'execute',
},
}),
});
export default createAgent(() => {
useMcpConnection(catalog);
return 'Use governed Unity Catalog functions when required.';
});
```
For Databricks Apps, prefer an OBO or least-privilege M2M token provider resolved at request time.
Never persist the workspace credential. See [Databricks integrations](/docs/databricks/integrations)
for effect classification, grants, and connection cleanup.
## Subagents
Declare specialists once and mount them conditionally:
```ts
const reviewer = defineSubagent({
name: 'reviewer',
description: 'Reviews release evidence before publication.',
model: 'anthropic/claude-sonnet-4-6',
thinkingLevel: 'high',
agent: () => {
useInstruction('Reject unsupported claims and list missing evidence.');
useSkill(reviewChecklist);
return 'Review the proposed release independently.';
},
});
export default createAgent(() => {
const [readyForReview] = usePersistentState('readyForReview', false);
if (readyForReview) useSubagent(reviewer);
return 'Prepare a release and delegate review when the evidence is ready.';
});
```
Subagents share the parent environment and cannot own persistent state, sandbox selection, public
response metadata, or root lifecycle hooks. Use [session tasks](/docs/building/subagents) to invoke a
named specialist under the normal task-depth, cancellation, checkpoint, and policy limits.
Addressable Node and Databricks App runtimes expose the same path without inventing a model-callable
delegation tool:
```sh
curl -X POST 'http://localhost:4317/agents/assistant/demo/tasks' \
-H 'content-type: application/json' \
-d '{"task":"Review the release evidence","agent":"reviewer","session":"release","timeoutMs":120000,"maxIterations":4}'
```
The endpoint invokes `session.task()` against the freshly rendered hook configuration. It is
tenant-checked, checkpoints before and after work, caps timeouts at five minutes and iterations at
16, propagates disconnect cancellation, and rejects unknown subagent names.
## Integrations
Dynamic composition works with the existing Fabric ecosystem rather than creating hook-specific
adapters:
| Integration | Dynamic-agent pattern | Guide |
| --- | --- | --- |
| Slack, Teams, GitHub, Discord, and 13 more channels | Read normalized channel input with `useDelivery()`; mount governed outbound tools with `useTool()`. | [Channels](/docs/ecosystem/channels) |
| Postgres, MySQL, MongoDB, Redis, SQLite, libSQL, Turso, Supabase, Valkey, and Lakebase | Construct the adapter outside render, then mount its tools conditionally. | [Databases](/docs/ecosystem/databases) |
| E2B, Daytona, Modal, Vercel, Cloudflare, Databricks SQL, and other sandboxes | Select a backend with `useSandbox()` while retaining capability discovery and policy. | [Sandboxes](/docs/ecosystem/sandboxes) |
| OpenTelemetry, Braintrust, Sentry, Jetty, and evaluations | Keep telemetry configured at the runtime boundary; lifecycle metadata and data parts enrich the trace. | [Tooling](/docs/ecosystem/tooling) |
| Remote MCP servers | Mount lazy, authenticated, allowlisted connections with `useMcpConnection()`. | [MCP](/docs/reference/mcp) |
| Databricks data, AI, Jobs, Genie, AI Search, Unity Catalog, Lakebase, and MLflow | Compose existing governed tools and managed MCP while preserving OBO or M2M identity. | [Databricks](/docs/databricks) |
Use `fh add ` to install managed integration wiring. Recipes remain ordinary public Fabric
tools, channels, stores, and sandboxes, so the same integration can be used by finite `defineAgent()`
jobs and dynamic `createAgent()` instances.
## Run locally
Create the agent file under `.fabricharness/agents/`, then start the development server:
```sh
pnpm add @fabric-harness/sdk @fabric-harness/node @fabric-harness/cli
pnpm exec fh dev
```
Send an interaction to the file name and a stable instance ID:
```sh
curl -X POST 'http://localhost:3000/agents/assistant/demo?wait=true' \
-H 'content-type: application/json' \
-d '{"message":"Start the work"}'
```
The complete source example is in `examples/dynamic-agent` in the source distribution.
## Build and publish an agent
Put public trigger metadata in the optional static `createAgent()` argument. Hosts inspect this
metadata without rendering interaction-dependent hooks:
```ts
export default createAgent(Assistant, {
description: 'Handles durable support work.',
triggers: { webhook: true },
});
```
Choose a deployment target without changing the agent definition.
`triggers.schedule` is intentionally `never` on `createAgent()`: a cron expression cannot identify
the instance, session, or message to deliver. Put the schedule on a finite dispatcher job (see
[Schedule triggers](/docs/reference/triggers#schedule-triggers)), or, on Cloudflare, use the
per-instance schedules API backed by Durable Object alarms (see
[Per-instance schedules](/docs/deployment/cloudflare#per-instance-schedules-durable-object-alarms)).
### Node or Docker
```sh
pnpm exec fh build --target node
NODE_ENV=production \
FABRIC_HARNESS_API_TOKEN="$FABRIC_HARNESS_API_TOKEN" \
node .fabricharness/build/node/dist/server.mjs
```
Ship the complete `.fabricharness/build/node` directory. See [Node deployment](/docs/deployment/node)
or build an OCI image with the [Docker target](/docs/deployment/docker).
### Cloudflare
```sh
pnpm exec fh build --target cloudflare
cd .fabricharness/build/cloudflare
pnpm install
pnpm exec wrangler deploy
```
Persistent instances are serialized and stored in Durable Objects. Configure production bindings and
run the credentialed smoke described in [Cloudflare deployment](/docs/deployment/cloudflare).
### Databricks Apps
```sh
pnpm exec fh build --target databricks-app
cd .fabricharness/build/databricks-app
databricks bundle validate
databricks bundle deploy
databricks bundle run
```
The generated App uses the shared Node server, Databricks App identity, and the same dynamic render
contract. Bind model endpoints, warehouses, Lakebase, MCP Services, and Unity Catalog resources through
App resources or bundle variables—not committed identifiers or tokens. Follow the complete
[Databricks App tutorial](/docs/deployment/databricks-app), including `fh doctor`, OBO/M2M selection,
restart testing, and cleanup.
### Temporal
```sh
pnpm exec fh build --target temporal-worker
node .fabricharness/build/temporal-worker/dist/worker.mjs
```
The generated worker bundles the workspace's persistent-agent definitions. A dynamic prompt carries
only a JSON-safe descriptor (agent name, instance, delivery, actor/tenant correlation, and resource
fingerprint) through workflow history. The activity resolves that bundled definition, verifies it
matches the addressed persistent session, and re-renders hooks at the trusted worker boundary. It also
recomputes the model/tool/MCP resource fingerprint and rejects any mismatch before initializing the
agent, so a different worker deployment cannot silently execute a changed resource set.
Completed activity results are durably memoized so an activity retry does not repeat a completed
model response.
For a hand-composed worker, provide `resolveDynamicAgent` to `createLocalTemporalActivities()` and set
`dynamicAgents: true` on `temporalSessionRuntime()` only after the matching task queue is deployed.
Without both sides, Harness fails closed. Hook functions, resolved MCP credentials, tool
implementations, and secret values must never enter workflow input.
For a production target, use a durable store, enable authentication and tenant isolation, verify the
target's capability matrix, exercise cancellation and restart recovery, and retain the generated
manifest as release evidence.
## Limits and compatibility
- Dynamic hooks are supported by persistent `createAgent()` definitions. Finite `defineAgent()` jobs
keep their explicit bounded run lifecycle.
- Named state is durable and may be declared conditionally. Named `useDataWriter()` identities must be
declared on every render.
- Root lifecycle, dispatch, persistent state, sandbox, MCP, and response-output hooks are unavailable
inside a subagent render.
- An async legacy initializer is accepted, but hooks after its first `await` fail because render has
ended. Keep hooks synchronous.
- Dynamic agents inherit the same policy, secret handling, identity isolation, retry, timeout,
cancellation, store, and deployment contracts as other persistent agents.
- Dynamic Temporal execution uses the coarse prompt activity so re-rendering and all nondeterministic
I/O stay outside workflow code. A worker that does not advertise and implement the dynamic-agent
resolver is rejected before delegation.
See the [runnable dynamic-agent example](/docs/reference/source-access),
[persistent-agent lifecycle](/docs/building/persistent-agents), and generated
[API reference](/docs/reference/api) for the complete types.
---
# Enterprise Controls
Canonical: https://harness.techfabric.com/docs/building/enterprise-controls
Apply tools, policy, approvals, budgets, and instructions at the job definition boundary.
Finite jobs can declare their runtime controls once at the definition boundary. The wrapper applies
them whenever `run()` calls `init()`, including through CLI, development server, and built artifacts.
## Complete governed job
Create `.fabricharness/jobs/governed-report.ts`:
```ts
import { defineTool, defineAgent, schema } from '@fabric-harness/sdk';
const lookupAccount = defineTool({
name: 'lookup_account',
description: 'Read an account summary.',
inputSchema: {
type: 'object',
additionalProperties: false,
properties: { accountId: { type: 'string' } },
required: ['accountId'],
},
execute: async ({ accountId }: { accountId: string }) => ({
accountId,
status: 'active',
}),
});
export default defineAgent({
name: 'governed-report',
description: 'Create a read-only account report.',
instructions: 'Use only approved account data. Never infer missing values.',
input: schema.object({ accountId: schema.string() }),
output: schema.string(),
triggers: { webhook: true, manual: true },
tools: [lookupAccount],
policy: {
toolPolicy: {
allow: ['lookup_account'],
deny: ['write', 'edit', 'bash'],
requireApproval: ['export_*'],
},
network: { mode: 'none' },
approvals: {
requiredApprovals: 2,
risk: 'high',
defaultTimeoutMs: 5 * 60_000,
},
},
costBudget: {
perCall: 0.05,
perSession: 0.50,
onExceed: 'throw',
},
approvals: { timeoutMs: 5 * 60_000 },
async run({ init, input }) {
const agent = await init();
const session = await agent.session();
return session.prompt(`Create the report for ${input.accountId}.`);
},
});
```
Run and inspect it:
```sh
fh describe governed-report
fh run governed-report --account-id acct-42
```
### What operators see
When an effect requires approval, Harness retains the exact request, session, vote count, decision,
and authorized controls. The console does not grant permission: the server independently enforces
the principal, tenant, policy, expiry, and exact-input approval scope.
## Precedence rules
Definition controls are defaults with security-aware composition:
| Control | Behavior when `init({...})` also supplies a value |
| --- | --- |
| `instructions` | A call-provided `role` wins. Otherwise instructions become the agent role. |
| `tools` and `skills` | Definition and call values are combined. |
| `policy` | Definition policy is a security floor. Calls may add denials and approvals, but cannot replace definition allowlists. |
| `costBudget` | The lower per-call and per-session limits win. A call cannot raise the definition budget. |
| `approvals.timeoutMs` | The shorter definition/call timeout wins. |
For example, this call can narrow the job but cannot enable network or raise its budget:
```ts
const agent = await init({
policy: {
toolPolicy: { deny: ['lookup_account'] },
},
costLimit: { perSession: 0.25 },
});
```
## Adversarial controls
For outbound HTTP, enable DNS answer verification in Node deployments so an allowed hostname cannot
resolve to loopback, link-local, metadata, or RFC1918 space:
```ts
const policy = {
network: {
mode: 'allowlist',
hosts: ['api.example.com'],
protocols: ['https:'],
resolveDns: true,
},
};
```
`policiedFetch()` checks the original URL, each redirect, IP literals, and DNS answers. Production
containers should also enforce egress at the network layer because process-level policy cannot
prevent code that deliberately bypasses the configured fetch implementation.
Tool aliases declare `policyAlias` so policy evaluates their canonical capability. For example, a
`save_file` tool with `policyAlias: 'write'` remains subject to every `write` deny and approval rule.
Raw access through `session.sandbox` cannot bypass `requireApproval`; only the exact operation inside
the resolved approval scope receives a temporary grant.
Durable attachment materialization defaults to 20 attachments, 10 MiB per decoded attachment, and
25 MiB total. `createSubmissionRunner({ attachments: { ... } })` can set lower
`maxCount`, `maxAttachmentBytes`, and `maxTotalBytes` values for a route or tenant.
## Approval handlers
The definition declares when approval is required and how long it may wait. The host supplies the
operator integration:
```ts
const agent = await init({
onApproval: async (request) => {
const decision = await approvalService.waitForDecision(request);
return decision; // approved or denied
},
});
```
Temporal-backed sessions persist approval waits. Inline sessions require the process to remain
available. See [Approvals](/docs/building/approvals) for CLI and API resolution.
## Tenant-wide budgets
Use a shared budget store when limits must survive restarts or span replicas:
```ts
import { tenantCostLimit } from '@fabric-harness/sdk';
import { postgresCostBudgetStore } from '@fabric-harness/node';
const costLimit = tenantCostLimit('acme', {
perDayUsd: 50,
store: postgresCostBudgetStore({ client: pool }),
});
const agent = await init({ tenantId: 'acme', costLimit });
```
Definition budgets protect one job/session. Shared budget stores enforce organization or tenant
ceilings across processes. Postgres and SQLite use atomic reservations: concurrent calls that would
cross the ceiling are rejected without committing an over-budget total.
Bind scheduled background jobs to a service principal instead of running them without identity:
```ts
await startDevServer({
scheduler: {
identity: async (jobName) => ({
tenantId: 'acme',
actor: { principal: { id: `scheduler:${jobName}`, type: 'service' } },
}),
},
});
```
Channel adapters derive tenant and actor from the verified provider payload. HTTP principals cannot
list, read, abort, delete, or vote on resources owned by another tenant.
## See also
- [Policies and approvals](/docs/reference/policies-approvals)
- [Cost attribution](/docs/operating/cost-attribution)
- [Agent anatomy](/docs/building/anatomy)
- [Security hardening](/docs/reference/security-hardening)
---
# Evaluations
Canonical: https://harness.techfabric.com/docs/building/evals
Test a job against deterministic scorers and model-based judges, so a prompt change shows up as a number before it ships.
Fabric eval suites are TypeScript modules named `*.eval.ts`. Each suite supplies cases, a runner, one or more scorers, and an optional pass threshold. `fh test` discovers and runs them locally or in CI.
## Install
```sh
pnpm add -D @fabric-harness/evals @fabric-harness/node
```
## Evaluate a job
```ts title="evals/hello.eval.ts"
import { containsTextScorer, defineEvalSuite } from '@fabric-harness/evals';
import { runAgent } from '@fabric-harness/node';
export default defineEvalSuite({
name: 'hello-quality',
cases: [
{ id: 'named-user', input: { name: 'Ada' }, expected: 'Ada' },
{ id: 'default-user', input: { name: 'World' }, expected: 'World' },
],
runner: async ({ case: evalCase }) => {
const run = await runAgent({
agent: 'hello',
payload: evalCase.input,
mock: true,
});
return run.result;
},
scorers: [containsTextScorer()],
passThreshold: 1,
});
```
Run every suite:
```sh
fh test
```
Run a subset or produce machine-readable output:
```sh
fh test evals --suite hello-quality
fh test --threshold 0.9 --json
```
The command exits nonzero when any suite fails, so it can be used directly as a CI quality gate.
## Choose scorers
| Scorer | Use it for |
| --- | --- |
| `exactMatchScorer()` | Deterministic structured or scalar output. |
| `containsTextScorer()` | Required text in an answer. |
| `regexMatchScorer()` | IDs, formats, and constrained strings. |
| `jsonShapeMatchScorer()` | Required JSON fields and primitive types. |
| `llmAsJudgeScorer()` | Semantic criteria that deterministic assertions cannot express. |
Prefer deterministic scorers first. For an LLM judge, use a separate provider/model, write a narrow rubric, and set `passThreshold` explicitly. Never pass provider secrets through eval case input.
For long-lived comparisons and stored grading trajectories, see [Jetty](/docs/ecosystem/tooling/jetty). For every exported type and scorer, see the [eval library reference](/docs/reference/eval-library).
---
# HTTP applications
Canonical: https://harness.techfabric.com/docs/building/http-applications
Mount authenticated Fetch routes and middleware around Fabric jobs, agents, channels, health, and identity.
Use the application surface when one deployed service needs Fabric routes plus product-specific HTTP
endpoints. Routes and middleware live in `.fabricharness/config.ts`, so `fh dev` and Node-derived
build artifacts run the same application.
```mermaid
flowchart LR
REQ[HTTP request] --> PUB[Public middleware]
PUB --> SYS[Health, rate limit, and authentication]
SYS --> RBAC[Principal, tenant, and permission]
RBAC --> APP[Authenticated middleware]
APP --> CHOICE{Route owner}
CHOICE -->|Application| CUSTOM[Fetch route handler]
CHOICE -->|Fabric| CORE[Job, agent, channel, MCP, or admin]
CUSTOM --> RES[HTTP response]
CORE --> RES
RES --> APP
APP --> PUB
classDef public fill:#f3f4f6,stroke:#6b7280,color:#111827
classDef identity fill:#fef3c7,stroke:#d97706,color:#422006
classDef fabric fill:#dbeafe,stroke:#2563eb,color:#172554
classDef custom fill:#dcfce7,stroke:#16a34a,color:#052e16
class PUB public
class SYS,RBAC identity
class CORE fabric
class CUSTOM custom
```
## Define routes
```ts title=".fabricharness/config.ts"
import { defineApplication, type FabricHarnessConfig } from '@fabric-harness/node';
const application = defineApplication({
routes: [
{
method: 'POST',
path: '/api/customers/:customerId/message',
permission: 'agent:invoke',
async handler(request, context) {
const body = await request.json() as {
message: string;
deliveryId?: string;
};
const receipt = await context.dispatch(
{
agent: 'support',
id: context.params.customerId!,
message: { kind: 'user', body: body.message },
},
body.deliveryId ? { idempotencyKey: body.deliveryId } : undefined,
);
return Response.json(receipt, { status: 202 });
},
},
{
method: 'POST',
path: '/api/reports',
permission: 'agent:invoke',
async handler(request, context) {
const input = await request.json();
return Response.json(
await context.invoke({ job: 'report', input }),
{ status: 202 },
);
},
},
],
});
export default {
application,
} satisfies FabricHarnessConfig;
```
Route paths support named `:parameters` and a trailing `*` wildcard. The handler uses the standard
Web `Request` and returns a Web `Response`. Its context contains:
| Field | Purpose |
| --- | --- |
| `principal`, `actor`, `tenantId` | Server-validated identity and tenant; request bodies cannot replace them. |
| `params` | Decoded named path parameters. |
| `dispatch()` | Durable admission to a persistent agent with optional idempotency key. |
| `invoke()` | Ambient finite-job admission with tenant, actor, and parent correlation. |
| `stores` | The configured session, submission, conversation, and attachment stores. |
| `signal` | Aborts when the client aborts the request. |
| `requestId` | Correlation id for application logs and response headers. |
Custom routes inherit the normal authentication, tenant binding, rate limit, and authorization
pipeline. Set `permission` to the closest built-in permission. It defaults to `session:read` for
`GET` and other non-Fabric paths. Application hooks do not accept identity from payload fields.
## Add middleware
Authenticated middleware receives the complete request context and wraps both custom and built-in
routes. Calling `next()` once continues the chain. For a custom route, `next()` returns its
`Response`, so middleware can add headers or replace it. For a streaming or built-in Node route,
`next()` completes when the response finishes.
```ts
middleware: [async (request, context, next) => {
const started = performance.now();
const response = await next();
await audit.record({
requestId: context.requestId,
principalId: context.principal.id,
tenantId: context.tenantId,
method: request.method,
path: new URL(request.url).pathname,
durationMs: performance.now() - started,
});
if (!response) return;
const headers = new Headers(response.headers);
headers.set('x-fabric-request-id', context.requestId);
return new Response(response.body, { status: response.status, headers });
}],
```
`publicMiddleware` wraps health, rate limiting, and authentication. It intentionally receives no
principal or stores. Use it for request timing, trusted ingress normalization, and global response
headers, not authorization decisions.
```ts
publicMiddleware: [async (request, context, next) => {
const response = await next();
console.log(context.requestId, request.method, new URL(request.url).pathname);
return response;
}],
```
Middleware and handlers share one cached request body. Reading `request.clone().json()` in
middleware does not consume the bytes later used by job, channel, webhook-signature, or custom route
handling. The configured `maxBodyBytes` limit still applies.
## Route ownership
Fabric reserves `/health`, `/ready`, `/jobs`, `/agents`, `/channels`, `/sessions`, `/runs`, `/mcp`,
`/admin`, `/builds`, `/dispatch`, and `/openapi.json`. A custom route beneath a reserved root or a
duplicate method/path fails before the server listens. This prevents an application update from
silently replacing approval, health, or agent behavior.
The complete runnable workspace is in
[`examples/application-routes`](/docs/reference/source-access).
---
# Model Providers
Canonical: https://harness.techfabric.com/docs/building/model-providers
Configure which LLM providers are available and pick one per agent, per session or per individual call at runtime.
TechFabric Harness uses an explicit `provider/model-id` reference everywhere a model is selected. There is no implicit "default OpenAI" — you opt into a provider by configuring credentials and naming the model.
## Reference format
```
provider/model-id
```
Examples:
```
openai/gpt-5.5
anthropic/claude-sonnet-4-6
gemini/gemini-2.5-pro
bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
```
## Setting credentials
Put provider keys once in a repo/workspace `.env.local`; TechFabric Harness auto-loads `.env` and `.env.local` files, while shell env still wins:
```sh
cp .env.example .env.local
# OPENAI_API_KEY=...
# ANTHROPIC_API_KEY=...
# AZURE_OPENAI_ENDPOINT=https://....openai.azure.com
# AZURE_OPENAI_API_KEY=...
```
Use explicit `--env ` only for overrides. Never paste API keys into source files or session artifacts.
## Selecting the model
The first non-empty wins, in this order:
1. CLI flag: `fh run ask --model openai/gpt-5.5`
2. Environment: `FABRIC_MODEL=openai/gpt-5.5`
3. `.fabricharness/config.ts` → `run.model` or `agent.model`
4. Agent-declared default: `defineAgent({ model: 'openai/gpt-5.5' })`
Per-call override:
```ts
await session.prompt('Summarize', { model: 'openai/gpt-5.5' });
```
## Mock provider
For local development and tests, pass `--mock` to `fh run` or `fh dev`. The flag substitutes the
deterministic mock provider regardless of the declared model reference and honors typed-result
schemas where possible. Without `--mock`, `openai/gpt-5.5` makes a real provider call and requires
`OPENAI_API_KEY`.
```ts
export default defineAgent({
// ...
model: process.env.FABRIC_MODEL ?? 'openai/gpt-5.5',
});
```
## Provider env names
TechFabric Harness knows the standard env names for common providers:
- `OPENAI_API_KEY`
- `ANTHROPIC_API_KEY`
- `OPENROUTER_API_KEY`
- `DEEPSEEK_API_KEY`
- `MOONSHOT_API_KEY`
- `XAI_API_KEY`
- `GEMINI_API_KEY`
- `GOOGLE_API_KEY`
- `AZURE_OPENAI_API_KEY` + `AZURE_OPENAI_ENDPOINT`
- `GROQ_API_KEY`
- `MISTRAL_API_KEY`
- `COHERE_API_KEY`
## Direct OpenAI-compatible presets
[DeepSeek](https://api-docs.deepseek.com/),
[Moonshot/Kimi](https://platform.kimi.ai/docs/api/overview), and
[xAI](https://docs.x.ai/developers/quickstart) expose OpenAI-compatible Chat Completions APIs.
Fabric provides named presets so direct-key users do not need to repeat the provider base URL:
```sh
FABRIC_MODEL=deepseek/deepseek-v4-flash # DEEPSEEK_API_KEY
FABRIC_MODEL=moonshot/kimi-k2.6 # MOONSHOT_API_KEY
FABRIC_MODEL=xai/grok-4.5 # XAI_API_KEY
```
`kimi/...` is accepted as an alias for `moonshot/...`. Override endpoints with
`DEEPSEEK_BASE_URL`, `MOONSHOT_BASE_URL`, or `XAI_BASE_URL`, or configure
`providers.deepseek`, `providers.moonshot`, and `providers.xai` explicitly.
These are transport presets, not provider-specific SDK implementations. They retain the typed
Fabric model contract, tool calls, streaming, redacted HTTP failures, and cost-budget behavior.
Missing keys fail before a network request and name the required variable. Fabric never falls back
to OpenRouter or mock inference when a direct preset fails.
Built-in prices are effective-dated and intentionally incomplete. DeepSeek V4 and the xAI Grok 4.5
short-context tier have verified rows; Moonshot and unlisted models leave `usage.costUsd` undefined.
Register deployment-specific or contract pricing with `registerModelPrices()` rather than treating
an unknown price as zero.
## Cloudflare Workers AI binding
When deploying to Cloudflare Workers with `fh build --target cloudflare`, you can route inference through the platform binding (`env.AI.run()`) instead of HTTP — no API tokens, no egress, runs at the edge.
```ts
import { CloudflareWorkersAIModelProvider } from '@fabric-harness/cloudflare/workers-ai';
export default {
async fetch(request: Request, env: Env) {
const fabric = await init({
modelProvider: new CloudflareWorkersAIModelProvider({
binding: env.AI,
defaultModel: '@cf/meta/llama-3.1-8b-instruct',
}),
});
// ...
},
};
```
`wrangler.toml`/`jsonc`:
```toml
[ai]
binding = "AI"
```
Handles modern `{ choices: [...] }` and legacy `{ response: '...' }` Workers AI shapes. Optional Cloudflare AI Gateway routing supports enterprise logging/routing knobs:
```ts
new CloudflareWorkersAIModelProvider({
binding: env.AI,
defaultModel: '@cf/meta/llama-3.1-8b-instruct',
gateway: {
id: 'prod-gateway',
skipCache: false,
cacheTtl: 3600,
collectLog: true,
eventId: request.headers.get('x-request-id') ?? undefined,
metadata: { tenant: 'acme', environment: 'prod' },
},
models: {
'@cf/meta/llama-3.1-8b-instruct': {
contextWindowTokens: 8192,
maxOutputTokens: 2048,
supportsTools: true,
},
},
});
```
Model metadata feeds context-budgeting/auto-compaction and admin UIs. The provider includes built-in metadata for common Workers AI chat models and accepts `models` / `defaultModelInfo` overrides for private or newly released models.
## OpenAI-compatible gateways
Many AI gateway products speak the OpenAI Chat Completions request/response shape: Vercel AI Gateway, Helicone, Portkey, LiteLLM (self-hosted), internal corp proxies. Wire any of them with the generic `aiGateway()` helper:
```ts
import { aiGateway, init } from '@fabric-harness/sdk';
// Helicone
const fabric = await init({
modelProvider: aiGateway({
baseUrl: 'https://oai.helicone.ai/v1',
apiKey: process.env.OPENAI_API_KEY!,
headers: { 'Helicone-Auth': `Bearer ${process.env.HELICONE_API_KEY}` },
defaultModel: 'gpt-4o',
name: 'helicone',
}),
});
// Self-hosted LiteLLM
const fabric = await init({
modelProvider: aiGateway({
baseUrl: 'http://litellm.internal:4000/v1',
apiKey: process.env.LITELLM_KEY!,
defaultModel: 'azure/gpt-5.5',
}),
});
```
### Vercel AI Gateway preset
[Vercel AI Gateway](https://vercel.com/docs/ai-gateway/sdks-and-apis/openai-compat) gets a thin preset with the gateway URL pre-baked:
```ts
import { vercelAIGateway, init } from '@fabric-harness/sdk';
const fabric = await init({
modelProvider: vercelAIGateway({
apiKey: process.env.AI_GATEWAY_API_KEY!,
defaultModel: 'anthropic/claude-sonnet-4-6',
}),
});
```
`baseUrl` defaults to `https://ai-gateway.vercel.sh/v1`; override for staging or self-hosted.
## Foundry runtime (Azure)
On Azure compute such as an ACA Job, AKS pod, or VM with managed identity, `FoundryRuntimeModelProvider` calls the Foundry-managed Azure OpenAI surface using a Bearer token instead of an API key:
```ts
import { FoundryRuntimeModelProvider } from '@fabric-harness/azure/foundry-runtime';
import { init } from '@fabric-harness/sdk';
const fabric = await init({
modelProvider: new FoundryRuntimeModelProvider({
defaultModel: 'gpt-4o',
}),
});
```
Set `AZURE_OPENAI_ENDPOINT` and `AZURE_OPENAI_DEPLOYMENT`. Supply `FOUNDRY_AGENT_TOKEN` when your host injects one, or install the optional `@azure/identity` peer dependency to use `DefaultAzureCredential` with the workload's managed identity.
## Spend caps
Per-call and per-session USD ceilings prevent runaway spend. Wired through `init({ costLimit })`:
```ts
const fabric = await init({
costLimit: {
perCall: 0.10, // throw if a single model call exceeds $0.10
perSession: 1.00, // throw if cumulative session spend exceeds $1.00
onExceed: 'throw', // 'throw' (default) | 'approve'
},
});
```
When `onExceed: 'approve'` the loop pauses on a violation and emits `approval_requested` with `kind: 'cost-limit'`. Approve via your existing approval UI (or `fh approve `) to release the loop; deny to throw `CostLimitExceededError`.
Limits evaluate after each call's cost lands on `usage.costUsd`. Forks and replays start with a fresh budget — replay is a debug action, not production work.
### Cross-process aggregation
For "tenant X spends ≤ $50/day" or "company-wide ≤ $100/hour" caps, pair `perScope + scopeKey + store` with a cross-process `CostBudgetStore`:
```ts
import { init, inMemoryCostBudgetStore } from '@fabric-harness/sdk';
import { postgresCostBudgetStore } from '@fabric-harness/node';
const fabric = await init({
costLimit: {
perScope: 50.00,
scopeKey: `tenant:${tenantId}:day:${todayIso}`,
store: postgresCostBudgetStore({ client: pgClient }), // or inMemoryCostBudgetStore() for single-process
onExceed: 'throw',
},
});
```
The store is the source of truth — multiple agents / multiple processes share the running total. fabric-harness never interprets `scopeKey`; you pick the convention (per-tenant, per-day, per-org). Reset semantics (daily rollover, billing period close) are also yours — call `store.reset(scopeKey)` from a scheduled task.
## Anthropic prompt caching
When an Anthropic response includes `cache_read_input_tokens` / `cache_creation_input_tokens`, fabric-harness records them on `usage.cachedInputTokens` and `usage.cacheWriteTokens`, and the cost calculator discounts billed input tokens by the cached read amount (and adds the cache-write surcharge when present). `fh metrics` shows a new `Cache: read=N write=N` line so you can see how much you're saving.
```text
$ fh metrics ask-1f4f...
Tokens: input=120000 output=2400 total=122400
Cache: read=96000 write=0
Cost: $0.046800
```
Cache-read tokens are billed at ~10% of the standard input rate on Claude models. Long, stable system prompts → big savings.
## Per-call cost telemetry
Every model call is enriched with a USD estimate from a static price table (mainline OpenAI, Anthropic, Gemini, Bedrock, Cohere). Cost shows up in `fh metrics` and on OpenTelemetry spans as `gen_ai.usage.cost_usd` — see [CLI → metrics](/docs/cli/sessions#fh-metrics).
Override or extend the catalog at runtime when you have custom-rate contracts:
```ts
import { registerModelPrices } from '@fabric-harness/sdk';
registerModelPrices([
{ provider: 'openai', model: 'gpt-4o', inputPerMTok: 1.5, outputPerMTok: 6, effectiveAt: '2026-05-08', notes: 'Enterprise contract' },
]);
```
## Reasoning effort
Reasoning-capable models accept a `thinkingLevel` controlling how much the model thinks before answering:
```ts
type ThinkingLevel = 'off' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh';
```
It is configurable at three scopes, most-specific wins:
```ts
const agent = await init({ model: 'cloudflare/@cf/openai/gpt-oss-120b', thinkingLevel: 'medium' });
const session = await agent.session('s1', { thinkingLevel: 'high' }); // per-session override
await session.prompt('think hard about this', { thinkingLevel: 'xhigh' }); // per-call override
```
The level is **capability-gated**: reasoning-capable providers map it to their native control, others ignore it (no error). `'off'` (or unset) requests no reasoning.
- **Default loop (pi-agent-core):** works for every provider; pi-ai handles capability detection + per-provider mapping and clamps the level to what each model supports.
- **Native Fabric providers:** Cloudflare Workers AI binding & OpenAI-compatible map to `reasoning_effort`; Anthropic to `thinking.budget_tokens` (with `max_tokens` raised above the budget); Gemini/Vertex to `thinkingConfig.thinkingBudget`. Gated to known reasoning families (o-series / gpt-5 / gpt-oss, Claude 3.7/4.x, Gemini 2.5).
---
# Persistent Agents & Dispatch
Canonical: https://harness.techfabric.com/docs/building/persistent-agents
Long-lived, addressable agent instances with cross-call sessions, async dispatch, and a streaming WebSocket conversation.
TechFabric Harness has two authoring surfaces. A **job** is a finite, run-once execution; a **persistent agent** is a long-lived, URL-addressable instance whose sessions continue across calls. (Fabric names the finite surface *job* because *workflow* is reserved for Temporal durable execution.)
| | Finite **job** | Persistent **agent** |
| --- | --- | --- |
| Author with | `defineAgent({ run })` | `createAgent(AgentFunction, staticConfig?)` |
| Directory | `.fabricharness/jobs/` | `.fabricharness/agents/` |
| Invoke | `POST /jobs/:name` → `{ result, runId }` | `POST /agents/:name/:id` with `{ message, session? }` |
| Lifetime | one run, returns a result | long-lived instance; sessions persist across calls |
| Async / streaming | tracked under `/runs` | `dispatch()` + `GET /agents/:name/:id` WebSocket |
The directory and route split is enforced. Finite definitions live in `.fabricharness/jobs/` and
use `POST /jobs/:name`; persistent definitions live in `.fabricharness/agents/` and use
`POST /agents/:name/:id`.
## Defining a persistent agent
A persistent agent module default-exports `createAgent(...)`. Prefer a named synchronous agent
function: hooks compose its current capabilities and its return value becomes the instruction.
```ts title=".fabricharness/agents/support.ts"
import { createAgent, useModel, useSandbox } from '@fabric-harness/sdk';
function SupportAgent({ id }: { id: string }) {
useModel('anthropic/claude-sonnet-4-6');
useSandbox('virtual');
return `Resolve the support request for conversation ${id}.
Use the conversation history and state what the user should do next.`;
}
export default createAgent(SupportAgent, {
durability: { maxAttempts: 5, timeoutMs: 2 * 60 * 60_000 },
triggers: { webhook: true },
});
```
The function receives the addressed instance `id` and platform `env`, then renders again for each
interaction. `useModel()`, `useSandbox()`, `useTool()`, `useSkill()`, `useSubagent()`, MCP, lifecycle,
and persistent-state hooks compose the live harness.
Declare policy, durability, triggers, and initial-data validation in the second, host-readable
argument. Those fields are validated at registration and remain enforceable even when rendering
crashes. The lower-level initializer form that returns a `PersistentAgentConfig` remains supported
for programmatic adapters, but hook-composed functions are the recommended authoring surface.
See [Dynamic Agents and Hooks](/docs/building/dynamic-agents) for state-driven capability changes.
Persistent agents may declare `triggers.webhook` or `triggers.manual`. They do not accept
`triggers.schedule`, because a cron expression does not specify which instance, session, and
message should be invoked. Use a scheduled finite job that calls `dispatch()` with those values;
see [Triggers and Public Route Gating](/docs/reference/triggers#schedule-triggers).
## Direct prompts
`POST /agents/:name/:id` durably admits a message for the instance's named session and returns `202 { submissionId, streamUrl, offset }`. Processing is FIFO per session and continues off the request. Add `?wait=true` only when a synchronous compatibility response is required:
```sh
# Same instance "u1" → one continuing conversation
curl -XPOST localhost:4317/agents/assistant/u1 -d '{"message":"remember my name is Ada"}'
curl -XPOST localhost:4317/agents/assistant/u1 -d '{"message":"what is my name?"}'
```
Read settlement at `/agents/:name/:id/submissions/:submissionId`, catch up through
`/conversation?offset=`, or tail `/stream?offset=`. Stream checkpoints carry an incarnation; a
changed incarnation resets stale offsets after deletion/recreation. Concurrent messages to one
instance session queue FIFO instead of returning `409`.
### What operators see
The session address, tenant, entry count, queue state, and permitted recovery actions remain visible
together. The screenshot uses deterministic, sanitized fixture data; a real console derives every
row from the authenticated principal's scope.
For conditional admission, `uid: null` means create only, a string means continue exactly that
incarnation, and omission is unconditional. A string `uid` cannot carry `initialData` because an
existing-incarnation condition forbids creation.
## Async dispatch
`dispatch()` hands an input to an instance for asynchronous processing, returning a receipt immediately:
```ts
import { dispatch } from '@fabric-harness/sdk';
const receipt = await dispatch({ agent: 'assistant', id: 'u1', input: 'summarize today' });
// { dispatchId, acceptedAt }
```
Over HTTP, `POST /agents/:name/:id/dispatch` returns `202` + the receipt:
```sh
curl -XPOST localhost:4317/agents/assistant/u1/dispatch -d '{"input":"summarize today"}'
```
Dispatch processing is **idempotent by `dispatchId`** (a re-delivered dispatch is applied at most once). The `fh dev` server uses an in-process queue; with `runtime: 'temporal'`, durable delivery uses `temporalDispatchQueue` — dispatches survive worker/process restarts.
## Streaming conversation (WebSocket)
`GET /agents/:name/:id` upgrades to a conversational WebSocket:
```ts
const ws = new WebSocket('ws://localhost:4317/agents/assistant/u1');
// server → { type: 'ready', target: 'agent', name, instanceId }
ws.send(JSON.stringify({ type: 'prompt', requestId: 'r1', message: 'hello' }));
// server → { type: 'started', requestId }
// → { type: 'event', requestId, event } (streamed, repeated)
// → { type: 'result', requestId, result, session }
```
Send `{ type: 'ping' }` for a `pong` heartbeat. Prompts on one connection are serialized.
## Jobs and persistent agents
Use a finite job for one typed invocation and a persistent agent for an addressable conversation.
Both share the same session, tool, policy, sandbox, and deployment surfaces.
---
# React applications
Canonical: https://harness.techfabric.com/docs/building/react
Build agent conversations and finite-run interfaces with the public Fabric client protocol.
`@fabric-harness/react` provides `FabricProvider`, `useFabricClient`, `useFabricAgent`, and
`useFabricJob`. The package talks only to the public authenticated HTTP protocol, so the same UI can
target local development, a Node artifact, Databricks Apps, or another compatible deployment.
```sh
pnpm add @fabric-harness/react @fabric-harness/client react
```
## Configure the provider
Create one client for the application. Authentication headers stay in the application transport and
are not placed in agent messages or model context.
```tsx title="src/main.tsx"
import { FabricProvider } from '@fabric-harness/react';
export function Root({ children }: { children: React.ReactNode }) {
return (
{children}
);
}
```
For server-rendered frameworks, pass a request-scoped client through `client`. Hooks use stable
external-store server snapshots and start network observation only after the component mounts.
## Agent conversations
`useFabricAgent` catches up from offset zero, streams text, reasoning, and tool-argument deltas over
SSE, replaces transient parts with the canonical durable assistant message, deduplicates reconnect
replay, resets stale projections when the stream incarnation changes, and applies truncation
records. Render the stable `messages` projection; `transcript` remains available
for diagnostics and compatibility. Optimistic messages remain separate in `pending` until their
submission-correlated input is observed.
```tsx
import { useFabricAgent } from '@fabric-harness/react';
function SupportChat({ customerId }: { customerId: string }) {
const chat = useFabricAgent({ agent: 'support', id: customerId, session: 'default' });
async function send(text: string) {
const submissionId = await chat.send(text);
const settled = await chat.wait(submissionId, { timeoutMs: 30_000 });
console.log(settled.outcome);
}
return (
<>
{chat.messages
.filter((message) => message.display === 'visible')
.map((message) => (
))}
{chat.pending.map((message) => (
))}
>
);
}
```
While a tool call is still streaming, its message part has
`{ type: 'dynamic-tool', state: 'input-streaming', inputText }`. Treat `inputText` as untrusted,
partial JSON; only `input-available` contains the provider's completed parsed input.
Send image attachments with the normal delivered-message shape:
```ts
await chat.send({
kind: 'user',
body: 'Inspect this screenshot',
attachments: [{ type: 'image', mimeType: 'image/png', data: base64 }],
});
```
## Finite runs
`useFabricJob` admits an asynchronous run, follows its offset events, publishes the terminal run
envelope, and exposes retry and abort. Starting a newer invocation supersedes the previous
observation, so a late terminal response from the old run cannot replace the new run's state. Use
`invokeSync` only for short request/response work.
```tsx
const report = useFabricJob<{ accountId: string }, { url: string }>('account-report');
await report.invoke(
{ accountId: 'account-42' },
{ idempotencyKey: 'account-report:account-42' },
);
console.log(report.status, report.run?.output?.url);
await report.abort();
```
Agent observation reconnects from the latest durable offset with bounded exponential backoff and
resets the delay after receiving an update. Set `live: 'poll'` in `createFabricAgentStore` when SSE
is intentionally unavailable; the default is automatic SSE with polling fallback. Stores with no
subscribers or in-flight work are evicted, which keeps long-lived
applications with dynamic agent and job addresses bounded.
## Runnable application
[`examples/react-chat`](/docs/reference/source-access)
starts `fh dev --mock` and Vite with one `pnpm dev` command. It includes multiple persistent agent
instances, image attachments, optimistic retry, abort, canonical transcript streaming, and a finite
job panel. Its UI renders the public conversation-message projection, including streaming text and
tool states. The example consumes `@fabricorg/ui` for shared Fabric semantic tokens and operator
controls; this UI package remains an application dependency and is not required by the Harness SDK
or runtime. Its automated browser check captures desktop/mobile light and dark layouts and fails on
browser errors or mobile horizontal overflow.
---
# Roles
Canonical: https://harness.techfabric.com/docs/building/roles
System-prompt overlays you can apply per agent, session or call, so one agent definition serves several jobs.
Roles live under `.fabricharness/roles/.md`. They are **system-prompt overlays**, not persisted user messages.
## Format
```md
---
description: Senior backend engineer focused on safe, minimal changes.
model: openai/gpt-5.5
---
You are a senior backend engineer. Prefer small, well-tested changes.
Do not make broad refactors unless required.
```
## Precedence
```
call role > session role > agent role
```
```ts
// Agent default
const fabricAgent = await init({ role: 'engineer' });
// Session-scoped override (first arg is the session id; pass undefined to auto-generate)
const session = await fabricAgent.session(undefined, { role: 'reviewer' });
// Call-scoped override
await session.prompt('Review this PR', { role: 'reviewer' });
```
## Why a role and not just a prompt
A role is reusable, scoped, and never appears in the user-message history that the model sees. That keeps the conversation clean and prevents role text from being summarized away during compaction.
---
# Sandbox connectors
Canonical: https://harness.techfabric.com/docs/building/sandbox-connectors
Modular remote sandbox adapters for Daytona, E2B, Modal, and custom providers.
TechFabric Harness treats every execution backend as a `SandboxEnv`: a small interface for shell execution, file IO, path scoping, cleanup, and optional snapshots. Agent/session/runtime code does not need to know whether work is running in the virtual sandbox, local process, Docker, Daytona, E2B, Modal, Cloudflare, Foundry, Kubernetes, or another provider.
## Contract
A remote provider adapter implements `RemoteSandboxApi` and wraps it with `createRemoteSandboxEnv`:
```ts
import { createRemoteSandboxEnv } from '@fabric-harness/sdk';
import type { RemoteSandboxApi } from '@fabric-harness/sdk';
const api: RemoteSandboxApi = {
async exec(command, options) { /* provider shell call */ },
async readFile(path) { /* UTF-8 file read */ },
async readFileBuffer(path) { /* binary file read */ },
async writeFile(path, content) { /* file write */ },
async stat(path) { /* file stat */ },
async readdir(path) { /* list names */ },
async exists(path) { /* existence check */ },
async mkdir(path, options) { /* mkdir */ },
async rm(path, options) { /* remove */ },
};
export const sandbox = createRemoteSandboxEnv(api, { cwd: '/workspace' });
```
Provider SDK objects and credentials stay in your app code. Fabric only receives file paths, bytes, commands, cwd, env, and timeout values.
## Package adapters
`@fabric-harness/connectors` ships dependency-free, structural adapters. Your app owns the provider SDK dependency and passes an initialized sandbox object into Fabric.
```sh
npm install @fabric-harness/connectors
```
The connector package declares provider SDKs as optional peers, so applications install only the provider they use.
### Why maintained adapters instead of copied templates
A project-installed template is a useful starting point, but the generated adapter becomes application
code as soon as it is copied. Fabric keeps the common provider mapping in a versioned package while
still letting the application own sandbox creation, credentials, resource limits, and lifecycle.
| Concern | Maintained Fabric adapter | Copied project template |
|---|---|---|
| Provider SDK changes | Bounded peer ranges and one shared compatibility table | Each application must detect and port SDK changes |
| Runtime behavior | One `SandboxEnv` contract across Node, Temporal, Cloudflare, and Databricks | Behavior can drift between generated copies |
| Recovery | Portable references and decoder registration support cross-process reattachment | Reattachment must be designed per project |
| Cancellation | Shared timeout, abort, orphan-settlement, and cleanup semantics | Every copied adapter must preserve the runtime rules itself |
| Security | Credentials stay in the provider client; capabilities and tenant ownership remain enforceable by the runtime | Security depends on each generated copy remaining current |
| Verification | The same conformance runner checks binary files, cwd/env, streaming, timeout, abort, reconnect, and cleanup | Tests and retained evidence are application-owned |
| Customization | Structural interfaces accept provider objects without importing their SDK into Harness core | Direct editing is flexible but creates a permanent fork |
This does not make provider infrastructure implicit. The application still provisions the sandbox and
chooses whether Fabric owns cleanup. Use a project-local `remoteSandbox()` adapter when an organization
needs a provider SDK version or lifecycle policy outside the maintained compatibility range.
| Provider | Compatible SDK range | Contract-tested version |
|---|---|---|
| Daytona | `@daytona/sdk >=0.195.0 <1` | `0.195.0` |
| E2B | `@e2b/code-interpreter >=2.6.1 <3` | `2.6.1` |
| Modal | `modal >=0.9.0 <1` | `0.9.0` |
| Vercel | `@vercel/sandbox >=1.10.1 <2` | `1.10.1` |
| Kubernetes | `@kubernetes/client-node >=0.21.0 <0.22` | `0.21.0` |
| Cloudflare Sandbox | `@cloudflare/sandbox >=0.9.2 <1` | `0.9.2` |
| Cloudflare Shell | `@cloudflare/shell >=0.3.7 <0.4` | `0.3.7` |
### Daytona
```ts
import { Daytona } from '@daytona/sdk';
import { daytonaSandbox } from '@fabric-harness/connectors';
const client = new Daytona({ apiKey: process.env.DAYTONA_API_KEY });
const remote = await client.create({ image: 'ubuntu:latest' });
const fabric = await init({
sandbox: daytonaSandbox(remote, { cleanup: true }),
});
```
The adapter maps Daytona filesystem/process calls to Fabric's `SandboxEnv` and uses Daytona's workdir when available through `daytonaSandboxFactory()`.
### E2B
```ts
import { Sandbox } from '@e2b/code-interpreter';
import { e2bSandbox } from '@fabric-harness/connectors';
const remote = await Sandbox.create();
const fabric = await init({
sandbox: e2bSandbox(remote, { cleanup: true }),
});
```
If your E2B package exposes a different class, adapt it to the structural `E2BSandboxLike` shape or wrap it in `remoteSandboxEnv()`.
### Modal
Fabric maps the native Modal TypeScript SDK sandbox directly. The structural `modalSandbox()` helper
remains available for custom provider handles.
```ts
import { ModalClient } from 'modal';
import { modalSdkSandbox } from '@fabric-harness/connectors/modal';
const client = new ModalClient();
const app = await client.apps.fromName('fabric-harness', { createIfMissing: true });
const image = client.images.fromRegistry('node:22-alpine');
const remote = await client.sandboxes.create(app, image, { workdir: '/workspace' });
const fabric = await init({ sandbox: modalSdkSandbox(remote, { cleanup: true }) });
```
## Generic adapter
Use `remoteSandbox()` to produce a reusable `SandboxFactory`:
```ts
import { remoteSandbox } from '@fabric-harness/connectors';
export function providerSandbox(client): SandboxFactory {
return remoteSandbox({
exec: (command, options) => client.exec(command, options),
readFile: (path) => client.readFile(path),
readFileBuffer: (path) => client.readFileBuffer(path),
writeFile: (path, content) => client.writeFile(path, content),
stat: (path) => client.stat(path),
readdir: (path) => client.readdir(path),
exists: (path) => client.exists(path),
mkdir: (path, options) => client.mkdir(path, options),
rm: (path, options) => client.rm(path, options),
}, { workspacePath: '/workspace' });
}
```
## Stream command output
`SandboxExecOptions` exposes the same output callbacks for local, Docker, and remote sandboxes:
```ts
const result = await env.exec('npm test', {
timeout: 120_000,
onStdout: (chunk) => process.stdout.write(chunk),
onStderr: (chunk) => process.stderr.write(chunk),
});
```
E2B, Vercel, and Kubernetes forward provider output as it arrives. Daytona's compatible command API returns collected output, so the adapter invokes the callbacks immediately before the command promise settles. Custom and Modal adapters emit incremental chunks through the same options.
Resource and egress enforcement belongs in the provider's sandbox creation call. Configure CPU, memory, image, network blocking, and domain allowlists before passing the provider object to Fabric; use Fabric `policy` for tool, command, filesystem, and application-level network decisions inside the session.
## Certification helper
Use the full contract runner before deploying a provider adapter:
```ts
import { assertSandboxCertification } from '@fabric-harness/connectors';
const env = daytonaSandbox(remote, { cleanup: true });
const report = await assertSandboxCertification(env, {
provider: 'daytona',
sdkPackage: '@daytona/sdk',
sdkVersion: '0.195.0',
credentialed: true,
reconnect: async (ref) => daytonaSandbox(
await client.get((ref.providerData as { workspaceId: string }).workspaceId),
),
verifyCleanup: async () => { /* assert the workspace was deleted */ },
});
```
The runner verifies:
1. POSIX shell execution and output
2. exact binary file round trips
3. working directory and environment forwarding
4. timeout exit `124` and abort exit `130`
5. stdout/stderr callbacks
6. JSON-safe portable references and cross-client reconnect
7. provider cleanup verification
The returned `SandboxCertificationReport` is safe to retain as CI evidence: it records provider, SDK version, timings, and check outcomes without credentials or workspace content.
## Verify a provider connection
Live tests are skipped unless enabled:
```sh
FABRIC_DAYTONA_TEST=1 DAYTONA_API_KEY=... pnpm --filter @fabric-harness/connectors test
FABRIC_E2B_TEST=1 E2B_API_KEY=... pnpm --filter @fabric-harness/connectors test
FABRIC_MODAL_TEST=1 MODAL_TOKEN_ID=... MODAL_TOKEN_SECRET=... pnpm --filter @fabric-harness/connectors test
```
## Connector recipes
`fh add` still prints markdown recipes for project-local adapters:
```sh
fh add
fh add daytona | claude
fh add https://e2b.dev --category sandbox | claude
```
Recipes are useful when the provider SDK version or organization conventions require custom code. Package adapters are better when your provider object matches the structural interfaces.
## Provider adapter checklist
- Scope every path to the provider workspace root.
- Honor `cwd`, `env`, and `timeout` on `exec`.
- Convert text and binary content correctly.
- Keep API keys and provider SDK objects outside model context/history.
- Enforce provider-specific network/resource limits before launching work.
- Implement `cleanup` for temporary sandboxes.
- Implement `snapshot`/`restore` only when the provider supports it truthfully.
- Add a unit test with a fake provider object.
- Add a live test behind an env gate.
---
# Sandboxes
Canonical: https://harness.techfabric.com/docs/building/sandboxes
Where shell commands and tool calls actually run — virtual, local, docker, cloudflare, daytona, modal, foundry-hosted, and remote SDK-backed targets.
import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
Every session has a sandbox — an isolated environment exposing filesystem, shell, and metadata APIs. TechFabric Harness defines a backend-agnostic `SandboxEnv` interface and ships several implementations.
## The interface
```ts
export interface SandboxEnv {
exec(command: string, options?: {
cwd?: string;
env?: Record;
timeout?: number;
}): Promise;
readFile(path: string): Promise;
readFileBuffer(path: string): Promise;
writeFile(path: string, content: string | Uint8Array): Promise;
stat(path: string): Promise;
readdir(path: string): Promise;
exists(path: string): Promise;
mkdir(path: string, options?: { recursive?: boolean }): Promise;
rm(path: string, options?: { recursive?: boolean; force?: boolean }): Promise;
cwd: string;
resolvePath(path: string): string;
snapshot?(): Promise;
restore?(snapshot: SandboxSnapshot): Promise;
cleanup(): Promise;
}
```
## Built-in sandboxes
Start simple: the bare `@fabric-harness/sdk` `defineAgent` wrapper defaults to `sandbox: 'virtual'`, so a first agent does not need Docker, Cloudflare, or a remote sandbox provider.
| Backend | Use first when | Capability notes |
| --- | --- | --- |
| `virtual` (default for bare import) | You want the fastest no-container path for support bots, routing, tests, and lightweight filesystem/search work. | In-memory filesystem + bash-like shell via `just-bash`; no network; process-level isolation; no snapshot/restore. |
| `local` | CI/repo automation intentionally needs host tools like `git`, `gh`, `npm`, or `python`. | Host process execution confined to the workspace root (see below). Scope commands and secrets carefully. |
| `docker` | The agent runs untrusted shell, data analysis, package installs, or generated code. | Container isolation; preferred production pilot sandbox for risky shell workloads. |
| `cloudflare` | You deploy to Workers and need Cloudflare Sandbox containers per session. | Provider-managed container isolation with Durable Object session storage. |
| `empty` | You only need model calls and custom tools; no filesystem or shell. | No shell/filesystem effects. |
## Local sandbox confinement
The `local` backend is scoped to the workspace root. Filesystem operations
(`readFile`/`writeFile`/`mkdir`/`rm`/…) resolve paths against the workspace, follow symlinks, and
reject anything outside it. `session.shell()` commands are scanned before they run: absolute
paths, `~` expansion, `..` traversals, and redirection targets (`>`, `>>`, `<`) that resolve
outside the workspace root are denied with a `SANDBOX_ESCAPE_BLOCKED` `FabricError` — the same
observable policy-denial path as other policy violations, never a silent skip or a crash.
In-workspace symlinks that point outside the workspace are denied too.
Workloads that legitimately need other directories (for example `/tmp` scratch space) opt them in
explicitly with the `allowedPaths` escape hatch:
```ts
import { LocalSandboxEnv } from '@fabric-harness/sdk';
const sandbox = new LocalSandboxEnv({
workspacePath: process.cwd(),
allowedPaths: ['/tmp/fabric-scratch'], // additional allowed roots
});
await init({ sandbox });
```
`allowedPaths` is also available as `metadata.allowedPaths` on
`createSandboxEnv({ backend: 'local', … })`, and forked sandboxes inherit it.
Confinement is a static guardrail, not a hard security boundary. Paths constructed dynamically at
runtime (shell variables, encoded payloads, substitution tricks that hide the path characters)
cannot be statically analyzed, and the check runs before exec, so a symlink swapped in afterwards
(a TOCTOU race) is not caught. Quoted string arguments that merely *begin* with an outside
absolute path can be over-denied; add the root to `allowedPaths` or rephrase. For untrusted
workloads, use `docker` or a provider container/microVM, which enforce a real isolation boundary.
## Decision matrix
| Workload | Recommended sandbox |
| --- | --- |
| Hello world, support FAQ, routing, typed extraction | `virtual` |
| GitHub issue triage in CI | `local` + scoped `defineCommand()` commands |
| Data analysis over uploaded files | `docker` |
| Full coding agent with Linux tools | Docker, Daytona, E2B, Modal, Kubernetes, or Cloudflare Sandbox |
| Edge/serverless support agent | Cloudflare target + virtual/filesystem source, or Cloudflare Sandbox when real shell/container support is required |
Treat `virtual` as a DX/performance feature, not a hard security boundary. Use Docker or a provider container/microVM for untrusted code.
### What developers verify
A successful command is only one part of sandbox evidence. Inspect declared capabilities first,
then prove binary files, working directory and environment, streaming, timeout, abort, portable
references where supported, reconnect, and cleanup.
## Selecting a sandbox
The default import injects `sandbox: 'virtual'` automatically. Pick another by passing a value:
```ts
import { defineAgent } from '@fabric-harness/sdk';
import { getSandbox } from '@cloudflare/sandbox';
import { createCloudflareSandboxEnv } from '@fabric-harness/cloudflare';
export default defineAgent({
run: async ({ init, env }) => {
const sandbox = createCloudflareSandboxEnv(getSandbox(env.Sandbox, 'session-1'));
const session = await (await init({ sandbox })).session();
return await session.prompt('hello edge');
},
});
```
**Example:** `examples/with-cloudflare-sandbox/`. Edge-deployed via `fabric-harness build --target cloudflare`.
```ts
import { defineAgent, schema } from '@fabric-harness/sdk/strict';
export default defineAgent({
name: 'long-running',
input: schema.object({ jobId: schema.string() }),
run: async ({ init, input }) => {
const fabric = await init({
runtime: 'temporal',
sandbox: 'local',
compaction: { enabled: false },
});
return await (await fabric.session()).prompt(`Process job ${input.jobId}`);
},
});
```
**Example:** `examples/with-temporal/`. Uses `/strict` because auto-compaction is non-deterministic across replays.
```ts
import { defineAgent } from '@fabric-harness/sdk';
import { AzureOpenAIModelProvider, createAzureKeyVaultSecretResolver } from '@fabric-harness/azure';
const provider = new AzureOpenAIModelProvider({
endpoint: process.env.AZURE_OPENAI_ENDPOINT!,
apiKey: process.env.AZURE_OPENAI_API_KEY!,
deployment: 'gpt-5.5',
});
export default defineAgent({
run: async ({ init, input }) => {
const fabric = await init({ modelProvider: provider });
return await (await fabric.session()).prompt(input.prompt);
},
});
```
**Example:** `examples/with-azure/`. Build with `fabric-harness build --target foundry-hosted-agent`.
```ts
import { defineAgent } from '@fabric-harness/sdk';
import { daytonaSandbox } from '@fabric-harness/connectors';
import { Daytona } from '@daytona/sdk';
const client = new Daytona({ apiKey: process.env.DAYTONA_API_KEY });
export default defineAgent({
run: async ({ init, input }) => {
const remote = await client.create({ image: 'node:22' });
const fabric = await init({ sandbox: daytonaSandbox(remote, { cleanup: true }) });
return await (await fabric.session()).prompt(input.prompt);
},
});
```
**Example:** `examples/with-daytona/`. Daytona credentials stay in the Daytona client.
```ts
import { defineAgent } from '@fabric-harness/sdk';
import { ModalClient } from 'modal';
import { modalSdkSandbox } from '@fabric-harness/connectors/modal';
export default defineAgent({
run: async ({ init, input }) => {
const client = new ModalClient();
const app = await client.apps.fromName('fabric-harness', { createIfMissing: true });
const remote = await client.sandboxes.create(app, client.images.fromRegistry('node:22-alpine'));
const sandbox = modalSdkSandbox(remote, { cleanup: true });
try {
const fabric = await init({ sandbox });
return await (await fabric.session()).prompt(input.prompt);
} finally {
await sandbox.cleanup();
client.close();
}
},
});
```
**Example:** `examples/with-modal/`. Modal credentials remain in `ModalClient`.
```ts
import { defineAgent } from '@fabric-harness/sdk';
export default defineAgent({
run: async ({ init, input }) => {
const fabric = await init({
sandbox: { backend: 'docker', image: 'node:22', cleanup: true },
});
return await (await fabric.session()).prompt(input.prompt);
},
});
```
**Example:** `examples/with-docker/`. Per-session container isolation; good fit for coding agents.
## Sandbox vs runtime vs target
These are three orthogonal axes:
| Axis | Controls | Where it's set |
|---|---|---|
| **Sandbox** | Where shell commands and tool calls run. | `init({ sandbox })`. |
| **Runtime** | How sessions persist. `stateless` / `inline` / `temporal`. | `init({ runtime })`. |
| **Target** | The build artifact: Node process, Cloudflare Worker, Temporal worker, Foundry hosted agent. | `fabric-harness build --target `. |
A Cloudflare Worker target most naturally pairs with the Cloudflare sandbox. A Temporal target most naturally pairs with `runtime: 'temporal'` and the `/strict` import. But the axes don't lock — you can deploy a Node target with a Daytona sandbox if you want.
## Remote and platform sandboxes
`RemoteSandboxApi` keeps provider SDK types out of `@fabric-harness/sdk`, so application code can select a backend without changing the agent/session contract.
| Integration path | Use it for |
|---|---|
| [`@fabric-harness/connectors`](/docs/building/sandbox-connectors) | Daytona, E2B, Modal, and provider-owned remote sandbox objects. |
| [`@fabric-harness/azure/aks-sandbox`](/docs/deployment/azure#aks-sandbox-sandboxenv) | Pod-backed execution on AKS. |
| [Databricks SQL sandbox](/docs/ecosystem/sandboxes/databricks-sql) | Governed SQL Warehouse execution; `session.shell()` executes SQL rather than bash. |
| `remoteSandboxEnv()` | Kubernetes, microVM, or proprietary execution services that implement the common remote contract. |
Use `fh add` when a provider needs project-specific lifecycle or SDK mapping. The [sandbox capability matrix](/docs/reference/sandboxes-matrix) shows how to inspect capabilities at runtime.
---
# Session memory
Canonical: https://harness.techfabric.com/docs/building/session-memory
Persistent key/value recall across sessions, scoped by tenant. Distinct from session entries (audit log).
`session.memory` lets agents remember facts across sessions — borrower preferences, prior outcomes, learned task history. It's a typed key/value store backed by `SessionMemory`, scoped automatically by the session's `tenantId`.
## Memory vs. entries
| | `session.memory` | session entries (`fh export-audit`) |
|---|---|---|
| Purpose | Recall — facts the agent should remember | Audit — what the agent did |
| Persistence | User-controlled key/value | Append-only log |
| Visibility to model | Only when the agent reads + injects | Through compaction / replay |
| Size | Bounded per key | Grows with each action |
| Typical reads | Targeted: `memory.get('borrower:preferences')` | Full session view: `fh inspect`, `fh logs` |
Memory writes do NOT land in the session log. Audit is unaffected.
## API
```ts
const fabric = await init({
tenantId: 'tenant-acme',
memory: postgresSessionMemory({ client: pgClient }), // or inMemorySessionMemory()
});
const session = await fabric.session();
// Get / set typed values
await session.memory.set({
key: 'borrower:42:preferences',
value: { contactChannel: 'sms', timezone: 'America/Phoenix' },
ttlSeconds: 60 * 60 * 24 * 30, // optional 30-day TTL
});
const entry = await session.memory.get<{ contactChannel: string }>('borrower:42:preferences');
// { key, value, tenantId, updatedAt, expiresAt?, metadata? }
// List with prefix + recency filters
const recent = await session.memory.list({ keyPrefix: 'borrower:42:*', limit: 10 });
await session.memory.delete('borrower:42:preferences');
```
`session.memory` auto-scopes every operation to the session's `tenantId`. Two tenants writing to the same key get isolated values.
## Backends
| Backend | Where | Use when |
|---|---|---|
| `inMemorySessionMemory()` | `@fabric-harness/sdk` | Tests, single-process pilots, ephemeral agents. |
| `postgresSessionMemory({ client })` | `@fabric-harness/node` | Production. Survives restarts, shares across a fleet. |
Postgres schema:
```sql
CREATE TABLE fabric_harness_session_memory (
tenant_id TEXT NOT NULL DEFAULT '',
key TEXT NOT NULL,
value JSONB NOT NULL,
metadata JSONB,
expires_at TIMESTAMPTZ,
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (tenant_id, key)
);
```
Created automatically on construction. Pass `initialize: false` if you manage migrations separately.
## Patterns
**Borrower preferences (long-term).** Set on first contact; read on subsequent sessions to skip re-asking.
```ts
const prefs = await session.memory.get<{ channel: 'sms' | 'email' }>('borrower:42:contact');
if (!prefs) {
await session.prompt('Ask the borrower how they prefer to be contacted.');
await session.memory.set({ key: 'borrower:42:contact', value: { channel: 'sms' } });
}
```
**Task history (medium-term).** Use TTL so old context doesn't accumulate.
```ts
await session.memory.set({
key: `task:${taskId}:summary`,
value: { result, durationMs },
ttlSeconds: 60 * 60 * 24 * 7, // 7 days
});
```
**Cross-tenant guards.** Memory operations scope by `tenantId` automatically. The SDK never mixes tenants — `set({ key: 'k' })` from `tenant-a` and `tenant-b` produce isolated rows.
## Reset / cleanup
Memory has no built-in expiry sweeper — TTL'd rows are filtered out on read. For high-volume deployments, run a periodic `DELETE FROM fabric_harness_session_memory WHERE expires_at < NOW()` from your scheduler. The `expires_at` column is indexed for that purpose.
## See also
- [Multi-tenancy](/docs/operating/multi-tenancy)
- [Audit export](/docs/operating/audit-export)
---
# Sessions and Prompts
Canonical: https://harness.techfabric.com/docs/building/sessions-prompts
How agent.session() and session.prompt() work, and what the harness loop does between the model and your tools.
A session is a persisted message/context thread. Inside a session, you can run prompts, skills, tasks, and shell commands.
## Create a session
```ts
const fabricAgent = await init();
const session = await fabricAgent.session(); // new id
const resumed = await fabricAgent.session('s-001'); // resume by id
const scoped = await fabricAgent.session('s-002', {
role: 'engineer',
model: 'openai/gpt-5.5',
cwd: 'project',
});
```
## `session.prompt(text, options?)`
Run one harness loop turn — a single user prompt that may produce assistant messages, tool calls, and shell commands until the model returns a final answer.
```ts
const answer = await session.prompt('What is Temporal?');
```
### Typed results
Use `result` to validate and type the return value. The framework asks the model for a typed object and validates it against the schema.
> `schema` is also exported from `@fabric-harness/sdk` — same API in both.
```ts
import { schema } from '@fabric-harness/sdk';
const triage = await session.prompt('Triage this issue', {
result: schema.object({
severity: schema.enum(['low', 'medium', 'high', 'critical']),
summary: schema.string(),
recommendedLabels: schema.array(schema.string()),
}),
});
```
### Streaming
```ts
for await (const event of session.stream('Tell me about Temporal')) {
if (event.type === 'text_delta') process.stdout.write(event.text);
}
```
### Working directories
Set `cwd` at agent, session, prompt/skill/task, or shell scope. Relative `cwd` values are resolved inside the sandbox and cannot escape the sandbox workspace.
```ts
const fabricAgent = await init({ cwd: 'project' });
const session = await fabricAgent.session();
await session.shell('npm install'); // runs in /workspace/project
await session.shell('npm test', { cwd: 'ui' }); // runs in /workspace/project/ui
await session.prompt('Inspect the UI package', { cwd: 'ui' });
await session.skill('review', { cwd: 'api', args: { focus: 'routes' } });
await session.task('Refactor tests', { cwd: 'packages/core' });
```
File tools (`read`, `write`, `grep`, `glob`) use the same scoped cwd during prompts/skills/tasks, so coding agents can work in a repository subdirectory without repeating absolute paths.
### Tools and commands
```ts
import { defineCommand } from '@fabric-harness/sdk';
const npm = defineCommand('npm');
const git = defineCommand('git');
await session.prompt('Run the failing tests and propose a fix', {
commands: [npm, git],
// tools: optional override of built-in tools
});
```
See [Tools](/docs/building/tools) and [Commands](/docs/building/commands).
## Reasoning streams
Reasoning-capable models (Anthropic Claude with extended thinking, OpenAI o-series, Gemini 2.5 with thought summaries) emit a separate "thinking" content channel. TechFabric Harness surfaces it on `ModelResponse.thinking` and emits a `text_delta` event with `kind: 'thinking'` so streaming consumers can render it apart from the final answer.
```ts
const stream = session.stream('Plan the migration in detail.');
for await (const event of stream) {
if (event.type === 'text_delta' && event.data?.kind === 'thinking') {
renderThinking(event.data.delta); // grey/italic in your UI
} else if (event.type === 'text_delta') {
renderOutput(event.data?.delta); // normal model output
}
}
```
Provider notes:
- **Anthropic**: thinking is emitted when the request includes `extended_thinking: true` (set via `headers` on `AnthropicModelProvider`).
- **OpenAI o-series**: emitted automatically as `reasoning_content` on the chat completion message.
- **Gemini 2.5**: emitted when `thinking_config` is set on the request; consumed via the same `thinking` field.
Other providers leave `ModelResponse.thinking` undefined; consumers should fall back to `text_delta` without the `thinking` kind.
### Token-level streaming
`OpenAICompatibleModelProvider` and `AnthropicModelProvider` both expose a `stream()` method that returns an `AsyncIterable`. The loop uses it automatically when present, emitting `text_delta` events as tokens arrive (instead of buffering until the response completes). UIs see word-by-word output; per-call cost telemetry still lands on the final aggregated chunk.
Consumers don't need code changes — `session.stream()` already understands `text_delta`:
```ts
for await (const event of session.stream(prompt)) {
if (event.type === 'text_delta') process.stdout.write(event.data?.delta ?? '');
}
```
Mid-stream failures fall through to the loop's normal error path. Retries are NOT automatic for streamed calls — partial state can't be safely re-applied.
## `session.skill(name, options?)`
Invoke a Markdown skill from `.fabricharness/skills//SKILL.md`. Args interpolate into the skill body, and `result` validates the typed return value.
```ts
const triage = await session.skill('triage-issue', {
args: { issueNumber: 42, repository: 'octocat/repo' },
commands: [gh],
result: schema.object({
severity: schema.enum(['low', 'medium', 'high', 'critical']),
}),
});
```
## `session.task(text, options?)`
Spawn a child task. Tasks are durable when the session runs on the Temporal worker target.
```ts
const result = await session.task('Refactor the authentication middleware', {
id: 'refactor-auth',
result: schema.object({ filesChanged: schema.array(schema.string()) }),
});
```
## `session.shell(command, options?)`
Execute a shell command in the session's sandbox.
```ts
const out = await session.shell('npm test', { cwd: '/workspace' });
```
## `session.fs` and `agent.fs`
Use `session.fs` for host-side filesystem plumbing that should not appear in model history: staging input files, collecting scratch output, or checking whether generated artifacts exist. It uses the same sandbox backend and cwd as the session.
```ts
const fabric = await init({ sandbox: 'virtual' });
const session = await fabric.session('build-123');
await session.fs.writeText('input/ticket.md', ticketBody);
const exists = await session.fs.exists('input/ticket.md');
const files = await session.fs.list('input');
const text = await session.fs.readText('input/ticket.md');
```
`agent.fs` is also available for quick setup scripts. It is backed by a lazily-created default session; prefer `session.fs` when you need explicit session identity, audit correlation, or lifecycle control.
Filesystem aliases follow familiar sandbox conventions: `readFile`, `readFileBuffer`, `writeFile`, `readdir`, and `rm`.
## Session history
```ts
const history = await session.history();
// { id, createdAt, updatedAt, entries, events }
```
The CLI's `fh inspect`, `fh logs`, and `fh metrics` are wrappers around this same data.
---
# Skills
Canonical: https://harness.techfabric.com/docs/building/skills
Reusable Markdown procedures an agent loads on demand, so a long instruction lives in one file rather than in every prompt.
Skills live under `.fabricharness/skills//SKILL.md`. They are Markdown-first: the body is the instructional prompt, the frontmatter declares metadata.
## Format
```md
---
name: triage-issue
description: Triage a GitHub issue and recommend severity, labels, and next action.
model: openai/gpt-5.5
---
You are triaging a GitHub issue.
Steps:
1. Read the issue using the `gh` command.
2. Inspect relevant files if available.
3. Determine severity.
4. Recommend labels.
5. Decide whether a fix can be proposed.
```
## Invocation
```ts
const result = await session.skill('triage-issue', {
args: { issueNumber: 42, repository: 'octocat/repo' },
commands: [gh],
result: schema.object({
severity: schema.enum(['low', 'medium', 'high', 'critical']),
summary: schema.string(),
recommendedLabels: schema.array(schema.string()),
fixSuggested: schema.boolean(),
}),
});
```
The framework:
1. Loads the skill by name.
2. Interpolates `args` into the prompt (template variables in the body, plus a structured arguments preamble).
3. Runs the harness loop with the chosen tools/commands.
4. Validates the typed result.
## Import a skill directly
A skill does not have to live under `.fabricharness/skills/`. Import its
`SKILL.md` and the whole directory compiles at build time:
```ts
import severityTriage from "../../skills/severity-triage/SKILL.md";
import { defineAgent } from "@fabric-harness/sdk";
export default defineAgent({
name: "incident-triage",
async run({ init }) {
const fabric = await init({ model: "openai/gpt-5.5", skills: [severityTriage] });
const session = await fabric.session();
return await session.skill("severity-triage", { args: { report } });
},
});
```
The Markdown body becomes the prompt, the frontmatter becomes `name`,
`description`, `model`, and metadata, and every sibling file becomes a packaged
resource the model reads on demand — the same shape directory loading produces,
so an imported skill and a workspace skill are indistinguishable at runtime.
Importing buys two things over loading by name:
- **Portability.** The skill sits next to the code that uses it, so it can ship
from a shared package instead of a per-project directory.
- **Build-time failure.** A wrong path fails the build; a wrong name in
`session.skill('...')` fails at model time, after the run has started.
Both forms are supported, and `.fabricharness/skills/` is unchanged.
### Markdown text imports
Any other `.md` import compiles to a string. Compose it with `defineSkill()`
when the file is not a `SKILL.md`:
```ts
import rubric from "../../skills/severity-triage/rubric.md";
import { defineSkill } from "@fabric-harness/sdk";
const closure = defineSkill({
name: "closure-summary",
description: "Summarize how an incident was closed out.",
instructions: `Summarize the closure in three bullets.\n\n${rubric}`,
});
```
`defineSkill()` also accepts `model`, `metadata`, and `packaged`. A skill
carrying `packaged` registers that directory with the `read` tool, so every
resource it advertises is readable.
### Type declarations
`.md` imports are typed by an opt-in ambient declaration, referenced once per
project:
```ts title=".fabricharness/fabric-env.d.ts"
///
```
`fh init` writes this file. It is opt-in rather than automatic because `*.md` is
a global wildcard that would otherwise collide with projects that already type
Markdown through Vite, MDX, or their own loader.
See [`examples/with-imported-skills`](/docs/reference/source-access) for a
runnable version.
## Why skills, not just prompts
- **Reuse.** Multiple agents can call the same skill.
- **Versioning.** Skills are tracked in source control and can be distributed as packaged skill directories.
- **Auditability.** Skill instructions are visible and reviewable, separate from agent code.
- **Model overrides.** A skill can declare its own model (for cheap classifiers, big-context reviewers, etc.).
---
# Subagents
Canonical: https://harness.techfabric.com/docs/building/subagents
Give persistent agents named specialist roles and delegate finite work through tasks or invoke.
TechFabric Harness uses two explicit delegation primitives instead of introducing a second agent
runtime:
- Persistent agents declare `subagents`, which become named role overlays available to their
sessions.
- Finite agents delegate work with `task()` or invoke another registered finite agent with
`invoke()`.
## Persistent specialist roles
```ts title=".fabricharness/agents/research.ts"
import { createAgent } from '@fabric-harness/sdk';
export default createAgent(({ id }) => ({
name: `research-${id}`,
instructions: 'Coordinate research and return sourced conclusions.',
subagents: [
{ name: 'analyst', content: 'Inspect data and quantify findings.' },
{ name: 'reviewer', content: 'Challenge assumptions and identify missing evidence.' },
],
}));
```
`subagents` use the same role representation and precedence rules as normal roles. They are system
prompt overlays, not persisted user messages, and they do not create another process or hidden
workflow runtime.
## Delegate finite work
```ts title=".fabricharness/jobs/research.ts"
import { defineAgent } from '@fabric-harness/sdk';
export default defineAgent<{ topic: string }>({
name: 'research',
run: async ({ input, task, invoke }) => {
const evidence = await task(`Collect evidence about ${input.topic}`, {
id: 'collect-evidence',
});
const review = await invoke({
job: 'review-evidence',
input: { topic: input.topic, evidence },
});
return { evidence, reviewRunId: review.runId };
},
});
```
Use `task()` when the child work belongs to the current session. Use `invoke()` when the delegated
work has its own registered definition, run identity, and admission policy. Temporal turns tasks
into child workflows; local and Node runtimes preserve the same public contract.
Failures are explicit: unknown invoked agents, task-depth overflow, policy denial, cancellation,
and child-run failure reject the parent call. See [Tasks](/docs/building/tasks) and
[Persistent agents](/docs/building/persistent-agents) for durability and inspection behavior.
---
# Tasks
Canonical: https://harness.techfabric.com/docs/building/tasks
Durable child runs and delegated agents, which survive a restart and report back to the session that started them.
A **task** is a child or delegated agent run. Tasks let an agent split work into named, durable sub-runs that can be inspected, cancelled, and (on the Temporal target) replayed independently.
## Spawn a task
```ts
const result = await session.task('Refactor the authentication middleware', {
id: 'refactor-auth',
result: schema.object({
filesChanged: schema.array(schema.string()),
summary: schema.string(),
}),
});
```
Options:
| Option | Purpose |
| --- | --- |
| `id` | Stable id (good for resuming, dedup, and CLI inspection). |
| `result` | Schema for typed output validation. |
| `model` | Override the model for this task. |
| `commands` | Scope shell commands available inside the task. |
| `cwd` | Working directory for the child session's shell/file tools. |
| `checkpoint` | Persist a checkpoint before and after the task (boolean or label). |
## Nesting limit (`MAX_TASK_DEPTH = 4`)
A task can spawn its own task, which can spawn its own — up to **four levels deep**. The fifth nested call throws:
```ts
// session.task → child.task → grandchild.task → great-grandchild.task ✓
// great-great-grandchild.task → throws "Task nesting exceeds MAX_TASK_DEPTH (4)"
```
The cap prevents runaway recursion. Each [`task_start` / `task_end`](/docs/reference/events) event carries a `depth: number` and a `parentSessionId`, so subscribers can render a tree:
```ts
onEvent: (event) => {
if (isEvent(event, 'task_start')) {
console.log(`${' '.repeat(event.data.depth)}↳ ${event.data.taskId}`);
}
}
```
## Inspect or cancel from the CLI
```sh
fh tasks
fh task refactor-auth
fh cancel-task refactor-auth --actor preetham --reason "Replaced by manual fix"
```
## Durable tasks on the Temporal target
When the session runs on the Temporal worker target, each task becomes a child workflow. That gives you:
- crash-safe execution,
- retries for model and tool calls,
- replayable history,
- durable cancellation signals.
## Checkpoints inside a task
Tasks can mark their own progress with checkpoints:
```ts
await session.checkpoint.create({ label: 'before-fix' });
// ... risky work ...
await session.checkpoint.create({ label: 'after-fix' });
```
If the sandbox supports snapshots, the checkpoint records a snapshot ref so you can restore filesystem state along with the conversation state.
---
# Test Without Credentials
Canonical: https://harness.techfabric.com/docs/building/test-without-credentials
The complete no-credential story — mock model, single-file runs, stub sessions, mock sandboxes, and evals.
Everything on this page runs offline with zero API keys. Reach for a real model only when the behavior under test depends on the model itself.
## Mock model from the CLI
`--mock` injects the deterministic mock provider into `fh run` and `fh dev`. Discovery, input/output schema validation, the model loop, and HTTP routing all still execute:
```sh
fh run hello --name Preetham --mock
# Mock response: Say hello to Preetham.
fh dev --mock --port 3000
curl http://localhost:3000/jobs/hello \
-H 'content-type: application/json' \
-d '{"name":"Preetham"}'
# {"result":"Mock response: Say hello to Preetham.","runId":"…"}
```
A `mock/`-prefixed configured model resolves the same way without any flag — `fh init --model mock/test-model` writes `model: "mock/test-model"` into `.fabricharness/config.ts`, and plain `fh run hello` (or the generated project's `npm run run`) executes against the mock. The production guard still applies: mock models throw when `FABRIC_ENV`/`NODE_ENV` is `production` unless `FABRIC_ALLOW_MOCK_MODEL=1` is set.
## Scripted mock responses
For deterministic multi-turn scripts — including tool loops — pass an aimock-style fixture file to `fh run --mock-script ` (implies `--mock`, no credentials, wins over every other model selection):
```json title="fixtures/ask.json"
{
"fixtures": [
{
"match": { "userMessage": "order", "hasToolResult": false },
"response": {
"toolCalls": [{ "name": "lookup_order", "arguments": { "id": "A-1" }, "id": "call-1" }]
}
},
{
"match": { "toolCallId": "call-1" },
"response": { "content": "Order A-1 ships tomorrow." }
}
]
}
```
```sh
fh run support --mock-script ./fixtures/ask.json --question "where is my order?"
```
Each entry is `{ match, response }`. The first entry whose `match` is satisfied wins; requests that match nothing fall back to the echo behavior. Match fields (all specified fields must match):
| Field | Matches when |
| --- | --- |
| `userMessage` | Substring of the latest user message. |
| `toolCallId` | A tool result with this call id is in the request history. |
| `toolName` | A tool result from this tool is in the request history. |
| `toolResultContains` | Substring of any tool result content. |
| `hasToolResult` | The request already contains tool results (`true`/`false`). |
| `sequenceIndex` | Zero-based model-turn index (0 on the first call, 1 after a tool-call round trip). |
Responses are `{ "content": "..." }` and/or `{ "toolCalls": [{ "name", "arguments", "id?" }] }`. Scripted tool calls execute for real — the mock emits the call, the loop runs your tool, and a later entry (matched on `toolCallId`, `toolName`, or `sequenceIndex`) answers from the result. Tool call ids default to `mock-tool-`; set `id` explicitly to match a later `toolCallId` entry. The fixture file is either a bare array of entries or the `{ "fixtures": [...] }` wrapper shown above.
The same script works in code via `@fabric-harness/sdk/testing`:
```ts
import { MockModelProvider, parseMockScript } from '@fabric-harness/sdk/testing';
const script = parseMockScript(fixtureJson); // validates, throws with the entry index
const provider = new MockModelProvider({ script });
```
## Single-file runs auto-mock
`fh run` executes any standalone `.ts`/`.js` file that exports `defineAgent()` — no `.fabricharness/` workspace, no `config.ts`, no install step beyond a resolvable `@fabric-harness/sdk`:
```ts title="agent.ts"
import { defineAgent, schema } from '@fabric-harness/sdk';
export default defineAgent({
name: 'hello',
input: schema.object({ message: schema.string() }),
output: schema.string(),
run: async ({ init, input }) => {
const fabric = await init();
const session = await fabric.session();
return session.prompt(input.message);
},
});
```
```sh
fh run ./agent.ts --message "hi"
# [fabric-harness] no model credentials resolved; using the mock model (set FABRIC_MODEL or pass --model for a real provider).
# [fabric-harness] session hello-4edbadf5-… completed (memory store — not persisted).
# Mock response: hi
```
When no model credentials resolve (`--model`, `FABRIC_MODEL`, or provider keys), the run falls back to the mock model and says so on stderr, so the command round-trips offline. Pass `--model` (or set `FABRIC_MODEL`) with a configured provider to run the same file against a real model. See [`fh run`](/docs/cli/run) for multi-export files, payload flags, and error behavior.
## MockModelProvider in code
`@fabric-harness/sdk/testing` exports the same provider the CLI uses:
```ts
import { MockModelProvider } from '@fabric-harness/sdk/testing';
const provider = new MockModelProvider();
const response = await provider.generate({
messages: [{ role: 'user', content: 'ping' }],
});
// response.message.content === 'Mock response: ping'
```
The mock is deterministic: it echoes the latest user message as `Mock response: ` and summarizes tool results after tool turns. To exercise a tool loop, script tool calls in the prompt with a `fabric-tool-calls` fenced block:
````ts
const response = await provider.generate({
messages: [{
role: 'user',
content: 'run the tool\n```fabric-tool-calls\n[{"tool":"lookup_order","input":{"id":"A-1"}}]\n```',
}],
});
// response.toolCalls === [{ id: 'mock-tool-1', name: 'lookup_order', input: { id: 'A-1' } }]
````
> The `/testing` entry point is shaped for test ergonomics and is not covered by the runtime SemVer commitment. Production code should not import from it.
## Stub sessions for unit tests
`StubFabricAgent` and `StubFabricSession` are the in-process implementations behind `init()`. Construct them directly with the mock model to unit-test agent logic without a workspace, server, or credentials:
```ts
import { describe, expect, it } from 'vitest';
import { StubFabricAgent } from '@fabric-harness/sdk/testing';
describe('support agent', () => {
it('answers through the session contract', async () => {
const agent = new StubFabricAgent({ model: 'mock/test-model' });
const session = await agent.session('demo');
const reply = await session.prompt('hello from the test');
expect(reply).toBe('Mock response: hello from the test');
});
});
```
For workspace-level runs, `runAgent({ agent, payload, mock: true })` from `@fabric-harness/node` covers the same ground through the full CLI pipeline — see [Testing locally](/docs/building/testing).
## Mock sandbox handles
Remote sandbox adapters in `@fabric-harness/connectors` are structural: they adapt a provider SDK handle to the Fabric `SandboxEnv` contract. That makes the provider handle mockable — back it with the SDK's in-memory `EmptySandboxEnv` and the real adapter runs end-to-end offline:
```ts
import { EmptySandboxEnv } from '@fabric-harness/sdk';
const inner = new EmptySandboxEnv('/home/daytona');
await inner.writeFile('/home/daytona/welcome.txt', 'hello from the mock daytona sandbox\n');
// expose inner's readFile/writeFile/exec/… through the provider's handle shape
```
The canonical pattern lives in [`examples/remote-coding-agent`](/docs/reference/source-access) (`connectors/mock-remote.ts` adapts `EmptySandboxEnv` to the `RemoteSandboxApi` contract), and each provider example ships its own structural mock with the live path gated behind an opt-in flag:
| Example | Mock handle | Live opt-in |
| --- | --- | --- |
| [`with-daytona`](/docs/reference/source-access) | `connectors/mock-daytona.ts` | `FABRIC_DAYTONA_LIVE=1` + `DAYTONA_API_KEY` |
| [`with-e2b`](/docs/reference/source-access) | `connectors/mock-e2b.ts` | `FABRIC_E2B_LIVE=1` + `E2B_API_KEY` |
| [`with-modal`](/docs/reference/source-access) | `connectors/mock-modal.ts` | `FABRIC_MODAL_LIVE=1` + Modal credentials |
| [`with-kubernetes`](/docs/reference/source-access) | `connectors/mock-kubernetes.ts` | `FABRIC_K8S_LIVE=1` + kubeconfig |
| [`with-vercel-sandbox`](/docs/reference/source-access) | `connectors/mock-vercel.ts` | `FABRIC_VERCEL_LIVE=1` + `VERCEL_TOKEN` |
| [`with-local-shell`](/docs/reference/source-access) | local sandbox directly | not needed — runs on the host |
Each example's default `pnpm run run` executes against the mock with no credentials; the live path fails fast with a clear error when the flag is set without its credentials. See [Sandbox connectors](/docs/building/sandbox-connectors) for the adapter contracts.
## Evals with `fh test`
Eval suites are TypeScript modules named `**/*.eval.ts`. Run the agent with `mock: true` inside the suite runner and the whole suite is deterministic and offline:
```ts title="evals/hello.eval.ts"
import { containsTextScorer, defineEvalSuite } from '@fabric-harness/evals';
import { runAgent } from '@fabric-harness/node';
export default defineEvalSuite({
name: 'hello-quality',
cases: [
{ id: 'named-user', input: { name: 'Ada' }, expected: 'Ada' },
],
runner: async ({ case: evalCase }) => {
const run = await runAgent({ agent: 'hello', payload: evalCase.input, mock: true });
return run.result;
},
scorers: [containsTextScorer()],
passThreshold: 1,
});
```
```sh
fh test
# Eval Results
#
# hello-quality PASS 100% 39ms
# PASS named-user contains_text=1.00
#
# Overall: PASS
```
`@fabric-harness/evals` ships deterministic scorers — `exactMatchScorer`, `containsTextScorer`, `regexMatchScorer`, `jsonShapeMatchScorer` — plus `llmAsJudgeScorer` for model-graded checks (the one scorer that needs a real provider). `fh test` exits non-zero when any suite fails, so the same command gates CI. See [Evaluations](/docs/building/evals) and [`fh test`](/docs/cli/test).
## When to add credentials
Everything above stays green with no keys. Add a provider key in `.env.local` when you are validating real model behavior, then drop `--mock`:
```sh
echo 'OPENAI_API_KEY=sk-...' > .env.local
fh doctor --live --model openai/gpt-5.5
fh run hello --name Preetham
```
See [Model providers](/docs/building/model-providers) for supported providers and credential resolution, and [Live tests](/docs/reference/live-tests) for the opt-in environment variable matrix used by the repo's own live suites.
---
# Testing Locally
Canonical: https://harness.techfabric.com/docs/building/testing
The mock model, the doctor, the dev server and the assertion helpers you use to test an agent without spending a token.
The framework is designed so that "test the agent" doesn't mean "spin up a full LLM." Use the mock provider for fast, deterministic checks and reach for live models only when behavior depends on the model. For the complete no-credential story — single-file runs, stub sessions, mock sandboxes, and evals — see [Test without credentials](/docs/building/test-without-credentials).
## Mock model
```sh
fh run ask --question "hi" --mock
```
`--mock` injects the deterministic mock provider and still exercises input/output validation. Combine
it with snapshot testing or schema-only assertions in your test suite.
## Vitest example
```ts
import { describe, expect, it } from 'vitest';
import { runAgent } from '@fabric-harness/node';
describe('ask agent', () => {
it('returns a string', async () => {
const { result } = await runAgent({
agent: 'ask',
payload: { question: 'hello' },
mock: true,
});
expect(typeof result).toBe('string');
});
});
```
## Doctor
```sh
fh doctor --tools # binary checks
fh doctor --live --model openai/gpt-5.5 # one real provider round-trip
```
## Dev server
`fh dev --mock` starts the same routes the deployed Node target uses without provider credentials. Invoke a finite job at `/jobs/:name`:
```sh
fh dev --mock --port 4000
curl -X POST -H 'Content-Type: application/json' \
-d '{"question":"What is Temporal?"}' \
http://localhost:4000/jobs/ask
```
## Live integration tests
Live tests are opt-in and skipped by default. Use them when validating real provider credentials and hosted resources:
```sh
pnpm --filter @fabric-harness/connectors test # Daytona / E2B / Modal live suites skip unless enabled
pnpm --filter @fabric-harness/azure test # Azure OpenAI / Foundry / ARM live suites skip unless enabled
pnpm --filter @fabric-harness/databricks test # Databricks live suites skip unless enabled
```
See [Live tests](/docs/reference/live-tests) for the full environment variable matrix.
Harness child-process tests start from a sanitized environment. Provider keys, cloud credentials,
database URLs, and agent markers are removed unless a test explicitly supplies an override. This
keeps offline suites repeatable on developer machines that happen to be logged into cloud services.
HTTP integration tests bind port `0` and use the operating system's assigned port, avoiding
parallel-suite collisions.
Maintainers should also run `pnpm check:unused`. The Knip configuration treats package entry points
as public API, checks internal exports and dependency declarations, excludes generated
`.fabricharness/build` output, and is included in `pnpm lint`.
### Maintainer workspace concurrency
The root `pnpm test` intentionally runs package suites with `--workspace-concurrency=1`. A measured
concurrency-2 run on 2026-07-30 caused the Node persistent-prompt admission test and the Databricks
capability-registry import test to exceed their bounded timeouts under shared CPU pressure. Keep the
workspace layer serial until those integration suites have isolated worker/resource budgets; Vitest
still parallelizes safely within each package. CI avoids the larger duplicate cost by running
typecheck, documentation, example, parity, release, coverage, and lint gates only in the Node 22
matrix lane.
## Recipes from `examples/`
The repo's `examples/` directory is the canonical reference for how to test each capability:
- [`examples/hello-world`](/docs/reference/source-access) — basic metadata agents and real model invocation.
- [`examples/with-tools`](/docs/reference/source-access) — built-in tools.
- [`examples/with-skill`](/docs/reference/source-access) — skill loading and typed results.
- [`examples/with-task`](/docs/reference/source-access) — durable task lifecycle and artifacts.
- [`examples/with-approval`](/docs/reference/source-access) — approval-gated commands.
- [`examples/with-docker`](/docs/reference/source-access) — Docker sandbox basics.
- [`examples/with-temporal`](/docs/reference/source-access) — Temporal worker integration.
- [`examples/with-config`](/docs/reference/source-access) — central config including SQLite session storage.
- [`examples/with-postgres-store`](/docs/reference/source-access) — Postgres session/artifact storage.
- [`examples/data-analyst`](/docs/reference/source-access) — Docker-backed CSV analysis with artifacts.
- [`examples/issue-triage-ci`](/docs/reference/source-access) — controlled CI pilot for read-only GitHub issue triage.
## What to assert in tests
For metadata agents, the most useful assertions are:
1. **Schema shape.** Output validates against the declared output schema.
2. **Tool/command scope.** No unexpected commands ran.
3. **Artifacts.** Expected artifacts were published with the right content type.
4. **Metrics.** Token / call counts stay within bounds for a given task.
5. **Idempotence.** Re-running the same prompt produces compatible output (when using the mock model or fixed seed).
---
# Tools
Canonical: https://harness.techfabric.com/docs/building/tools
The model-callable functions that ship with TechFabric Harness, and how a tool call is bounded before it reaches your system.
Tools are functions the model can call during a session. TechFabric Harness ships a built-in toolbelt that mirrors the proven coding-agent pattern.
## Built-in tools
| Tool | Purpose |
| --- | --- |
| `read` | Read a file from the sandbox. |
| `write` | Write a file to the sandbox. |
| `edit` | Apply a localized edit (find/replace) to a file. |
| `bash` | Execute a shell command in the sandbox. |
| `grep` | Search file contents. |
| `glob` | Match files by pattern. |
| `task` | Spawn a child task. |
All built-ins are **capability-aware**: `write` and `edit` honor any filesystem write scope declared on the session, `bash` honors the configured `commands` allowlist.
The model-facing tools have stable output and concurrency limits. See
[Agent behavior](/docs/reference/agent-behavior#built-in-tools) for paging, truncation, timeout,
exact-edit, and same-file serialization semantics.
## Constructing the toolset
```ts
import { createBuiltinTools } from '@fabric-harness/sdk';
const sandbox = await session.sandbox;
const tools = createBuiltinTools(sandbox);
await session.prompt('Find and fix the failing test', { tools });
```
By default, `session.prompt()` uses the built-in tools. You typically only override this when you want to *limit* which tools are available or *augment* with custom ones.
## Custom tools
> Both `defineTool` and `createBuiltinTools` are also exported from `@fabric-harness/sdk` — swap the import path when you want the minimal entrypoint.
```ts
import { defineTool, schema } from '@fabric-harness/sdk';
const fetchIssue = defineTool({
name: 'fetch_issue',
description: 'Fetch a GitHub issue by number.',
input: schema.object({ number: schema.number() }),
output: schema.object({ title: schema.string(), body: schema.string() }),
run: async ({ data }) => {
// data.number is validated against the input schema
return { title, body };
},
});
await session.prompt('Triage issue #42', { tools: [...tools, fetchIssue] });
```
---
# Voice
Canonical: https://harness.techfabric.com/docs/building/voice
Bidirectional audio for phone calls, browser and kiosk. Realtime mode and pipeline mode behind one VoiceSession contract.
`VoiceSession` is fabric-harness's surface for voice-capable LLMs. It streams audio bytes in both directions, surfaces text transcripts, and round-trips tool calls — same governance posture as text agents (cost telemetry, audit, approvals all apply).
fabric-harness ships **two voice modes** behind the same `VoiceSession` contract:
- **Realtime mode** — one model owns audio in + out + tools. OpenAI Realtime today; Anthropic / Gemini Live when GA.
- **Pipeline mode** — `STT → LLM → TTS`. Compose Deepgram / Cartesia STT with any LLM and ElevenLabs / Cartesia TTS. Swap providers without changing the caller code.
See [Voice Providers](/docs/building/voice-providers) for the provider matrix, pricing, and the picking heuristic. Telephony bridges (Twilio Media Streams, Vonage, Plivo) live in user-space; scaffold one with `fh add`.
## The contract
```ts
interface VoiceSession {
sendAudio(frame: Uint8Array): Promise;
sendText(text: string): Promise;
submitToolResult(input: { id: string; output: unknown }): Promise;
cancelResponse(): Promise;
events(): AsyncIterable;
close(): Promise;
}
type VoiceEvent =
| { type: 'session_open' }
| { type: 'audio_delta'; audio: Uint8Array }
| { type: 'text_delta'; delta: string; role: 'assistant' | 'user' }
| { type: 'transcript'; text: string; role: 'assistant' | 'user' }
| { type: 'tool_call'; id: string; name: string; input: unknown }
| { type: 'response_done'; usage?: ModelUsage }
| { type: 'error'; message: string };
```
`audio_delta` carries raw PCM 16-bit (or `g711_ulaw` for telephony). `tool_call` flags an LLM tool invocation — your code runs the tool and calls `submitToolResult({ id, output })` to feed the result back.
## OpenAI Realtime
```ts
import { OpenAIRealtimeVoiceProvider } from '@fabric-harness/sdk';
const provider = new OpenAIRealtimeVoiceProvider({
apiKey: process.env.OPENAI_API_KEY!,
model: 'gpt-realtime',
});
const voice = await provider.connect({
instructions: 'You are a friendly intake agent. One question at a time.',
voice: 'alloy',
audioFormat: 'pcm16', // or 'g711_ulaw' for Twilio
tools: [submitFieldTool],
turnDetection: 'server_vad', // OpenAI handles barge-in detection
});
// Pump audio in (mic / phone)
mic.on('data', (frame) => voice.sendAudio(frame));
// Pump audio out (speaker / phone)
for await (const event of voice.events()) {
if (event.type === 'audio_delta') speaker.write(event.audio);
if (event.type === 'tool_call') {
const output = await runTool(event.name, event.input);
await voice.submitToolResult({ id: event.id, output });
}
if (event.type === 'response_done') break;
}
```
The provider uses the platform `WebSocket` (Node 22+ + browsers) — no `ws` dependency on the consumer side.
## Pipeline mode (ElevenLabs / Cartesia / Deepgram)
When you want a non-OpenAI vendor combo — voice cloning, multilingual TTS, on-prem STT, or just lower per-minute cost — use `PipelineVoiceProvider`. It composes a `SttProvider`, a text `ModelProvider`, and a `TtsProvider` into the same `VoiceSession` contract. Swap any of the three without touching caller code.
```ts
import {
PipelineVoiceProvider,
DeepgramSttProvider,
ElevenLabsTtsProvider,
AnthropicModelProvider,
} from '@fabric-harness/sdk';
const provider = new PipelineVoiceProvider({
stt: new DeepgramSttProvider({ apiKey: process.env.DEEPGRAM_API_KEY!, model: 'nova-3' }),
tts: new ElevenLabsTtsProvider({
apiKey: process.env.ELEVENLABS_API_KEY!,
model: 'eleven_turbo_v2_5',
defaultVoice: '21m00Tcm4TlvDq8ikWAM', // 'Rachel'
}),
llm: new AnthropicModelProvider({ apiKey: process.env.ANTHROPIC_API_KEY! }),
model: 'claude-haiku-4-5-20251001',
});
const voice = await provider.connect({
instructions: 'Friendly intake agent. One question at a time.',
tools: [submitFieldTool],
});
// Same loop as realtime mode — events(), sendAudio(), submitToolResult().
```
Conversation flow:
1. Caller pumps audio into `voice.sendAudio()`.
2. Deepgram emits final transcripts → pipeline forwards as the next user turn.
3. LLM generates the assistant reply (with tool calls if relevant).
4. Assistant text streams to ElevenLabs → audio bytes arrive as `audio_delta` events.
5. `response_done` fires with rolled-up usage (`inputTokens`, `outputTokens`, `sttSeconds`, `ttsCharacters`, `costUsd`).
**Barge-in:** the STT VAD emits `speech_started` when the user interrupts; the pipeline aborts the in-flight LLM + TTS streams and returns to listening. Your UI should drain its audio buffer when `audio_delta` events stop arriving.
**Single-vendor variant** (no ElevenLabs key needed):
```ts
import { CartesiaSttProvider, CartesiaTtsProvider } from '@fabric-harness/sdk';
new PipelineVoiceProvider({
stt: new CartesiaSttProvider({ apiKey: process.env.CARTESIA_API_KEY! }),
tts: new CartesiaTtsProvider({ apiKey: process.env.CARTESIA_API_KEY! }),
llm: someLlmProvider,
model: 'gpt-4.1-mini',
});
```
For the full matrix (when to pick which mode, latency benchmarks, cost comparisons), see [Voice Providers](/docs/building/voice-providers).
## Tool calls
Tool execution is the *caller's* responsibility, not the voice session's. When `tool_call` lands:
1. Look up the tool by `name` in your registry.
2. Run it through whatever governance gates apply — approval policies, cost caps, rate limits — using the v1-era primitives unchanged.
3. Call `voice.submitToolResult({ id, output })`.
This keeps the audio path lean and avoids reimplementing the loop machinery on the voice side.
## Cost
Realtime audio is billed per audio token at materially higher rates than text. fabric-harness records `audioInputTokens` / `audioOutputTokens` on `usage` and rolls them into `costUsd` via the static price table:
```
$ fh metrics call-3a8b...
Tokens: input=1200 output=400 total=1600
Audio: input=12345 output=6789
Cost: $0.234567
```
Override the rates with `registerModelPrices` for negotiated contracts:
```ts
import { registerModelPrices } from '@fabric-harness/sdk';
registerModelPrices([{
provider: 'openai',
model: 'gpt-realtime',
inputPerMTok: 4,
outputPerMTok: 16,
audioInputPerMTok: 80,
audioOutputPerMTok: 160,
effectiveAt: '2026-05-08',
notes: 'Enterprise contract',
}]);
```
## Telephony bridges
Phone calls? Don't add a Twilio dep to fabric-harness. Use `fh add` to scaffold a project-local bridge:
```sh
fh add https://www.twilio.com/docs/voice/twiml/stream --category voice-telephony | claude
fh add https://developer.vonage.com/voice/voice-api/code-snippets --category voice-telephony | cursor-agent
```
This emits the canonical telephony spec (`packages/sdk/connector-spec/voice-telephony.md`) with a header pointing at the provider's docs. The coding agent reads the docs, follows the spec, and produces a single file at `./connectors/-bridge.ts`. Twilio uses μ-law 8kHz natively — set `audioFormat: 'g711_ulaw'` and skip the resample for the lowest-latency path.
## Browser-direct voice via the Node server
`fh dev` and generated Node builds expose a `WS /sessions/:id/voice` upgrade handler. The server
creates the OpenAI Realtime connection on behalf of the browser, so provider API keys stay
server-side. Tool calls relay through the agent's existing tool registry, so approval policies, cost
caps, and rate limiters apply to voice tools just like text tools.
```ts
import { connectFabricVoice } from '@fabric-harness/sdk';
const handle = connectFabricVoice({
url: `wss://app.example.com/sessions/${sessionId}/voice`,
authToken,
tenantId: 'acme',
voice: 'alloy',
instructions: 'Friendly intake agent. One question at a time.',
onEvent: (event) => {
if (event.type === 'audio') speaker.write(event.audio);
if (event.type === 'tool_call') {
// Optional: handle tool calls client-side. Default is server-side relay.
}
if (event.type === 'cost_limit') {
console.warn('Hit cost ceiling', event);
handle.close();
}
},
onError: console.error,
});
mic.on('frame', (pcm16) => handle.sendAudio(pcm16));
```
Server requirements:
- `OPENAI_API_KEY` env var (the bridge fails 1011 / `error` if missing).
- `ws` peer dep installed (`pnpm add ws`).
- Optional: `FABRIC_HARNESS_API_TOKEN`, `extractAuthToken`, `X-Fabric-Tenant` — same pipeline as the chat WS.
Query params customize each connection: `?model=gpt-realtime&voice=alloy&audioFormat=g711_ulaw&instructions=...`.
### Capture pipeline
1. `navigator.mediaDevices.getUserMedia({ audio: true })`.
2. Pipe through an `AudioWorklet` that resamples to PCM 16-bit 24kHz mono (or μ-law 8kHz for `audioFormat: 'g711_ulaw'`).
3. Forward each frame via `handle.sendAudio(buffer)`.
### Cost governance
`VoiceConnectOptions.costBudget` (and the WS bridge's `costLimit` option) wires voice into the v1.4 cost-budget machinery. On every `response.done`, the tracker observes the call cost and emits a `cost_limit` event when a ceiling is crossed. With `onExceed: 'approve'`, your `requestCostLimitApproval` is called before the session continues. Pair with `tenantCostLimit()` for per-tenant per-period ceilings.
## See also
- [Cost telemetry](/docs/building/model-providers#per-call-cost-telemetry)
- [Connector catalog](/docs/building/connector-catalog)
- [Session memory](/docs/building/session-memory) — useful for storing collected fields across calls
---
# Voice Providers
Canonical: https://harness.techfabric.com/docs/building/voice-providers
Choose and compose realtime or pipeline voice providers with OpenAI, ElevenLabs, Cartesia, and Deepgram.
TechFabric Harness supports two voice modes: **realtime**, where one model owns audio input, audio output, and tools; and **pipeline**, where you choose separate STT, LLM, and TTS providers.
## Choose a voice architecture
- Choose **OpenAI Realtime** when you want a single WebSocket connection with model-managed turn detection and tool calling.
- Choose a **Deepgram + ElevenLabs pipeline** when independent STT, LLM, and TTS selection matters.
- Choose a **Cartesia pipeline** when you want STT and TTS from one pipeline provider while retaining your preferred LLM.
- Serve either mode through `WS /sessions/:id/voice` to keep provider credentials on the server.
## Mode comparison
| Aspect | Realtime mode | Pipeline mode |
|---|---|---|
| Architecture | One WS, model handles audio in + out + tools | Three streams: STT → LLM → TTS |
| Voice flexibility | Provider's voices only | Any TTS vendor — voice clones, emotion, accents |
| Language support | Provider-bound | Determined by the selected STT and TTS providers |
| Tool calling | Native to the model | Through the underlying LLM (Anthropic/OpenAI/Gemini) |
| Barge-in | Server VAD | STT VAD + caller cancels TTS |
| Useful for | Phone agents, intake bots, simpler audio orchestration | Multilingual, branded voice, vendor flexibility, provider-specific governance |
## Provider matrix
| Provider | Role | Fabric integration | Consider when |
|---|---|---|---|
| OpenAI Realtime | Realtime end-to-end | `OpenAIRealtimeVoiceProvider` | You want one connection for audio, turn detection, responses, and tools. |
| ElevenLabs | TTS | `ElevenLabsTtsProvider` | Voice selection and TTS controls are central requirements. |
| Cartesia | TTS + STT | `CartesiaTtsProvider`, `CartesiaSttProvider` | You want one pipeline vendor for both speech directions. |
| Deepgram | STT | `DeepgramSttProvider` | You need a dedicated streaming transcription provider with endpointing controls. |
Provider models, languages, and prices change independently of TechFabric Harness. Confirm the current provider offering, then use `registerModelPrices` to apply your public or negotiated rate card to cost telemetry.
## Realtime mode (OpenAI)
```ts
import { OpenAIRealtimeVoiceProvider } from '@fabric-harness/sdk';
const provider = new OpenAIRealtimeVoiceProvider({
apiKey: process.env.OPENAI_API_KEY!,
model: 'gpt-realtime',
});
const voice = await provider.connect({
instructions: 'You are a friendly intake agent.',
voice: 'alloy',
audioFormat: 'pcm16', // or 'g711_ulaw' for Twilio.
tools: [submitFieldTool],
turnDetection: 'server_vad',
});
```
The model owns the audio loop end-to-end. Tool calls relay through the existing `tool_call` / `submitToolResult` contract — same as pipeline mode.
## Pipeline mode (BYO STT + LLM + TTS)
```ts
import {
PipelineVoiceProvider,
DeepgramSttProvider,
ElevenLabsTtsProvider,
AnthropicModelProvider,
} from '@fabric-harness/sdk';
const provider = new PipelineVoiceProvider({
stt: new DeepgramSttProvider({
apiKey: process.env.DEEPGRAM_API_KEY!,
model: 'nova-3',
}),
tts: new ElevenLabsTtsProvider({
apiKey: process.env.ELEVENLABS_API_KEY!,
defaultVoice: '21m00Tcm4TlvDq8ikWAM', // 'Rachel'
model: 'eleven_turbo_v2_5',
}),
llm: new AnthropicModelProvider({ apiKey: process.env.ANTHROPIC_API_KEY! }),
model: 'claude-haiku-4-5-20251001',
});
const voice = await provider.connect({
instructions: 'You are a friendly intake agent. One question at a time.',
audioFormat: 'pcm16',
tools: [submitFieldTool],
});
mic.on('frame', (pcm) => voice.sendAudio(pcm));
for await (const event of voice.events()) {
if (event.type === 'audio_delta') speaker.write(event.audio);
if (event.type === 'tool_call') {
const output = await runTool(event.name, event.input);
await voice.submitToolResult({ id: event.id, output });
}
if (event.type === 'response_done') {
// event.usage contains rolled-up tokens, sttSeconds, ttsCharacters, costUsd.
}
}
```
The contract is **identical** to realtime mode — `VoiceSession` events, tool calls, cost telemetry, `costBudget`, `cost_limit` events. Swap providers without rewriting your loop.
### Single-vendor variant (Cartesia)
```ts
import {
PipelineVoiceProvider,
CartesiaSttProvider,
CartesiaTtsProvider,
} from '@fabric-harness/sdk';
const provider = new PipelineVoiceProvider({
stt: new CartesiaSttProvider({ apiKey: process.env.CARTESIA_API_KEY! }),
tts: new CartesiaTtsProvider({ apiKey: process.env.CARTESIA_API_KEY! }),
llm: someLlmProvider,
model: 'claude-haiku-4-5-20251001',
});
```
## Barge-in
Pipeline mode wires barge-in through the STT VAD. When the user starts speaking while the agent is mid-utterance, the STT emits `speech_started`; the pipeline aborts the in-flight LLM call, cancels the TTS stream, and returns to listening. Your UI is responsible for stopping playback when `audio_delta` events stop arriving.
Realtime mode handles this server-side via `turnDetection: 'server_vad'`.
## Cost telemetry
Both modes feed the same telemetry surface. `response_done.usage` includes:
```ts
{
inputTokens: number; // LLM input
outputTokens: number; // LLM output
audioInputTokens?: number; // realtime mode only
audioOutputTokens?: number;// realtime mode only
sttSeconds?: number; // pipeline mode (Deepgram/Cartesia)
ttsCharacters?: number; // pipeline mode (ElevenLabs/Cartesia)
costUsd?: number; // rolled up via static price table
}
```
Wire `costBudget` to enforce per-call / per-session / per-tenant ceilings — voice participates in the same v1.4 cost-budget machinery as text agents.
```ts
import { CostBudgetTracker } from '@fabric-harness/sdk';
const budget = new CostBudgetTracker({ perCallUsd: 0.50, perSessionUsd: 5 });
await provider.connect({ costBudget: budget });
```
## Evaluate a voice stack
Test candidate providers with representative audio before choosing a production stack. Measure:
- time to first transcript and first synthesized audio;
- transcription quality for your languages, accents, vocabulary, and audio channel;
- interruption behavior under real network conditions;
- voice consistency and pronunciation for your domain;
- provider region, retention, residency, and audit controls;
- end-to-end cost using your own traffic distribution and rate card.
Because realtime and pipeline mode share the `VoiceSession` contract, you can run the same application loop against each candidate and compare session events and usage telemetry.
## When to bring your own provider
Implement `TtsProvider`, `SttProvider`, or `VoiceProvider` directly and pass it into `PipelineVoiceProvider`. Reasons to build your own:
- On-prem TTS (Riva, Coqui) for compliance.
- Whisper-via-vLLM for cheap multilingual STT.
- Translate-on-the-wire layers (e.g. STT in Spanish → MT → English LLM → TTS in Spanish).
The interfaces (`TtsProvider`, `SttProvider`) are intentionally narrow — `synthesize(text) → AsyncIterable` and `open() → SttSession` are the only required methods.
## See also
- [Voice](/docs/building/voice) — the `VoiceSession` contract and OpenAI Realtime usage.
- [Cost telemetry](/docs/building/model-providers#per-call-cost-telemetry) — pricing and rate-card overrides.
- [Connector catalog](/docs/building/connector-catalog) — telephony bridges (Twilio, Vonage) for phone audio.
---
# CLI Overview
Canonical: https://harness.techfabric.com/docs/cli
One binary that runs, builds, deploys, inspects and replays a workspace, so an agent has the same shape everywhere.
The TechFabric Harness CLI is a single binary that drives the entire framework: discovering agents, running them locally, building deployment artifacts, starting the dev server, inspecting persisted sessions, and managing approvals, tasks, artifacts, and builds.
## Invocation
The CLI is published as `fabric-harness` with `fh` as an exact alias:
```sh
fabric-harness --help
fh --help
```
For local framework development, run the built monorepo CLI directly:
```sh
node packages/cli/dist/bin/fabric-harness.js --help
pnpm fh --help
```
## Synopsis
```
fabric-harness --help
fabric-harness capabilities --json
fabric-harness run [options]
fabric-harness agents [--json]
fabric-harness describe [--json]
fabric-harness build [--target node|temporal-worker|docker|foundry-hosted-agent|cloudflare] [options]
fabric-harness dev [--target node|cloudflare|temporal-worker] [--mock] [--console] [options]
fabric-harness fiber [--url ] [--plain]
fabric-harness fiber [--url ] [--plain] [--job --input | --agent --id --message ]
fabric-harness docs
fabric-harness doctor [--target node|temporal-worker|databricks-app|databricks-serving|buzz] [--model provider/model] [--getting-started] [--tools] [--live] [--json]
fabric-harness init [directory] [--dir ] [--model provider/model] [--template ] [--store memory|file|sqlite|postgres|redis]
fabric-harness new [--force]
fabric-harness sessions
fabric-harness builds
fabric-harness inspect
fabric-harness logs [--events]
fabric-harness checkpoints
fabric-harness artifacts [--json]
fabric-harness artifact get [--out ]
fabric-harness metrics [--json]
fabric-harness tasks [--json]
fabric-harness task [--json]
fabric-harness cancel-task [--actor ] [--reason ]
fabric-harness compact [--keep ] [--summary ]
fabric-harness replay
fabric-harness approvals [--pending] [--state]
fabric-harness approve [--actor ] [--reason ]
fabric-harness reject [--actor ] [--reason ]
fabric-harness verify-attestation
fabric-harness verify-provenance
fabric-harness test [path] [options]
fabric-harness temporal-worker [--task-queue ] [--address ] [--env ]
fabric-harness add [connector] [--print]
```
## Command groups
| Group | Commands | Page |
| --- | --- | --- |
| Run agents | `run` | [run](/docs/cli/run) |
| Discover | `agents`, `describe` | [agents](/docs/cli/agents) |
| Build | `build`, `builds`, `verify-attestation`, `verify-provenance` | [build](/docs/cli/build), [builds](/docs/cli/builds) |
| Dev server | `dev` | [dev](/docs/cli/dev) |
| Testing | `test` | [test](/docs/cli/test) |
| Diagnostics | `doctor` | [doctor](/docs/cli/doctor) |
| Integration compatibility | `capabilities --json` | [compatibility contract](/docs/cli/compatibility) |
| Scaffolding | `init`, `new job`, `new agent` | [TechFabric Harness in 5 minutes](/docs/getting-started/five-minutes) |
| Interactive use | Fiber (`fiber`, `console`) | [Fiber terminal console](/docs/cli/console) |
| Documentation | `docs list`, `docs read`, `docs search` | [docs](/docs/cli/docs) |
| Sessions | `sessions`, `inspect`, `logs`, `replay`, `metrics`, `compact` | [sessions](/docs/cli/sessions) |
| Tasks | `tasks`, `task`, `cancel-task` | [tasks](/docs/cli/tasks) |
| Approvals | `approvals`, `approve`, `reject` | [approvals](/docs/cli/approvals) |
| Artifacts & checkpoints | `artifacts`, `artifact get`, `checkpoints` | [artifacts](/docs/cli/artifacts) |
| Temporal | `temporal-worker` | [temporal-worker](/docs/cli/temporal-worker) |
| Recipes | `add` | [add](/docs/cli/add) |
`fh init` keeps generated core dependencies on the release line shipped with the CLI. The current
CLI writes `^6.2.0` ranges for `@fabric-harness/sdk` and, when the selected template needs the Node
runtime, `@fabric-harness/node`. Managed `fh add` recipes use the same core dependency floors and
reject incompatible installed ranges before writing files.
## Global options
| Flag | Purpose |
| --- | --- |
| `-h`, `--help` | Print help text. |
| `--env ` | Load `.env`-style variables before `run`/`dev`/`temporal-worker`. Repeatable; shell env wins. |
## Configuration defaults
`.fabricharness/config.ts` may set:
- `run.target`, `run.model`, `run.idPrefix`,
- `temporal.address`, `temporal.taskQueue`,
- `agent.model`.
CLI flags always win over config. See [Configuration](/docs/getting-started/configuration).
---
# fh add
Canonical: https://harness.techfabric.com/docs/cli/add
Browse the ecosystem recipes, scaffold one into your workspace, or print it first to see what it will write.
```
fabric-harness add [kind] [name] [options]
```
`fh add` has two recipe modes:
| Mode | Command shape | Result |
| --- | --- | --- |
| Managed recipe | `fh add ` or `fh add ` | Resolves dependencies, writes implementation/environment/test files, and prints a verification command. |
| Connector guide | `fh add --print` | Prints Markdown for project-specific provider wiring. |
Managed recipes install first-party packages from the release line current when this CLI was
published. The recipe registry and generated project templates are checked together in CI; for
example, the current CLI emits SDK/Node `^6.2.0` ranges and Databricks `^7.1.1` ranges, while
connectors, evals, Temporal, and Cloudflare remain on their independently versioned lines. An existing
incompatible first-party major fails before files are written.
The catalog labels every entry with its mode. Some connectors also have package helpers in `@fabric-harness/connectors`, `@fabric-harness/channels`, `@fabric-harness/azure`, or `@fabric-harness/databricks`.
## Examples
### List available recipes
```sh
fh add
fh add --json
```
### Scaffold directly
```sh
fh add channel slack
fh add slack
fh add channel teams
fh add database postgres
fh add sandbox e2b --dir ./agent-service
fh add modal
fh add policy safe-defaults
```
Scaffold kinds are `channel`, `database`, `databricks`, `sandbox`, `tooling`, `model-provider`, `skill`, and `policy`.
Existing files are preserved unless `--force` is supplied. When `package.json` exists, Fabric detects
pnpm, npm, Yarn, or Bun and installs missing declared dependencies using recipe-owned compatible
version ranges. An incompatible existing range stops before files are written. Pass `--no-install`
to update the manifest without running the package manager.
Every recipe has a unique one-word alias. The explicit kind/name form remains useful in automation
and makes ownership clear. Successful installation prints the package-manager-specific Vitest command
for the generated contract test.
Every generated TypeScript or Markdown file has a marker such as:
```ts
// fabric-harness-recipe: channel/slack@1
```
The marker lets [`fh update`](/docs/cli/update) distinguish an unchanged generated file from user
customizations. Use `--dry-run` before applying a recipe in an existing project.
```sh
fh add channel slack --dry-run
fh add channel slack --dry-run --json
```
### Print a connector recipe
```sh
fh add daytona --print
fh add e2b --print
fh add github-mcp | codex
fh add discord --install-deps --print
```
For sandbox connectors, the output demonstrates the `RemoteSandboxApi` / `SandboxEnv` adapter pattern. For MCP, KB, and data connectors, it demonstrates the appropriate public primitive: `connectMcpServer`, `FilesystemSource`, scoped `Command`, or `ToolDef`.
## Categories
```sh
fh add https://provider.example/docs --category sandbox --print
fh add https://provider.example/docs --category channel --print
fh add https://provider.example/docs --category mcp --print
fh add https://provider.example/docs --category kb --print
fh add https://provider.example/docs --category data --print
```
| Category | Output shape |
|---|---|
| `channel` | Verified `Channel`, stable conversation key, durable dispatch, governed outbound tools |
| `sandbox` | `SandboxEnv` / `SandboxFactory` |
| `mcp` | `connectMcpServer()` wrapper |
| `kb` | `FilesystemSource` |
| `data` | `ToolDef`, `Command`, or MCP wrapper |
| `database` | Bounded data tools; unified persistence bundles are documented for every maintained backend |
| `tooling` | Telemetry, evaluation, or grading integration |
## Options
| Flag | Description |
| --- | --- |
| `--json` | Print the catalog, or the direct-recipe change plan when kind/name are supplied. |
| `--dir ` | Target directory for direct scaffold recipes. |
| `--force` | Replace existing scaffold files. |
| `--dry-run` | Print files, dependency resolutions, skips, and conflicts without writing. |
| `--no-install` | Update dependency sections without running the detected package manager. |
| `--install-deps` | Install dependencies declared by a connector guide. |
| `--print` | Print a connector guide even when stdout is a terminal. |
| `--category ` | Category for a provider documentation URL. |
| `--pr` | Open a draft connector PR for the guide workflow. |
## Package helpers
Prefer package helpers when available:
- `@fabric-harness/connectors`: Daytona, E2B, Modal, Vercel, Kubernetes, generic remote sandbox, S3, Azure Blob.
- `@fabric-harness/channels`: 18 maintained adapters spanning chat, developer tools, support, commerce, billing, knowledge, and email.
- `@fabric-harness/databases`: Postgres, MySQL, MongoDB, Redis, and SQLite governed data tools.
- `@fabric-harness/azure`: Azure OpenAI, Key Vault, Blob artifacts, Foundry Agent Service, Azure ARM tools.
- `@fabric-harness/databricks`: SQL, Jobs, notebooks, Unity Catalog, Agent Services, MLflow, workspace source.
## See also
- [Connector catalog](/docs/building/connector-catalog)
- [fh update](/docs/cli/update)
- [Ecosystem catalog](/docs/ecosystem)
- [Sandbox connectors](/docs/building/sandbox-connectors)
- [Capability matrix](/docs/reference/capability-matrix) — which connectors are first-class today.
---
# fh agents and fh describe
Canonical: https://harness.techfabric.com/docs/cli/agents
List the finite jobs and persistent agents in a workspace, and inspect any one of them without opening its source.
## `fh agents`
```
fabric-harness agents [--json]
```
List every finite job under `.fabricharness/jobs/` and persistent agent under
`.fabricharness/agents/`. The default output is human-readable; `--json` includes each definition's
`kind` (`job` or `agent`), source path, model, target, description, and triggers.
```sh
fh agents
fh agents --json
```
## `fh describe`
```
fabric-harness describe AGENT_NAME [--json]
```
Show metadata for a single definition. Finite jobs expose their declared model, target default,
triggers, examples, and input/output schemas. Persistent agents expose their route name and
`kind: "agent"` without running the per-instance initializer.
```sh
fh describe ask
fh describe ask --json
```
For finite agents declared with `defineAgent({...})`, `describe` shows:
- name and description,
- input schema, rendered as JSON Schema,
- output schema,
- declared model,
- triggers (`webhook`, etc.),
- run target default,
- examples (if `examples: [...]` was provided).
Plain default-exported functions are rejected. Finite definitions use `defineAgent({...})`;
persistent definitions use `createAgent(...)`.
## Why this exists
`describe` is the contract for tooling. CI, dev servers, deploy pipelines, and operator tools can
use `--json` to discover a definition without parsing TypeScript.
---
# fh approvals and fh approve
Canonical: https://harness.techfabric.com/docs/cli/approvals
Resolve a pending human approval from the terminal, so a paused agent can carry on without anyone opening a console.
Agents that gate destructive actions can call `session.approval.request({ reason, risk, timeoutMs })` to pause until a human responds. The CLI is the simplest way to resolve those requests during development and CI.
## `fh approvals`
```
fabric-harness approvals [session-id] [--pending] [--state]
[--url ] [--token-env ] [--tenant ]
```
List approval requests for a session. `--pending` filters to unresolved requests; `--state` includes the request's current state machine (timeouts, history of actions).
Against a deployed App, omit the session id to discover approvals across every session visible to
the authenticated tenant:
```sh
fh approvals \
--url https://your-app.databricksapps.com/api \
--token-env DATABRICKS_APP_TOKEN
```
The remote path uses `GET /approvals`, which requires `approval:read` and is tenant-filtered. It
does not require `admin:read`.
## `fh approve`
```
fabric-harness approve [--actor ] [--reason ]
[--url ] [--token-env ] [--tenant ]
```
Mark an approval as approved. The associated session resumes as soon as the running agent (or worker) observes the resolution.
```sh
fh approve ask-1f4f... appr-7 --actor preetham --reason "Cleared by SRE"
# Deployed App: the authenticated principal becomes the recorded voter.
fh approve ask-1f4f... appr-7 \
--url https://your-app.databricksapps.com/api \
--token-env DATABRICKS_APP_TOKEN \
--reason "Cleared by SRE"
```
## `fh reject`
```
fabric-harness reject [--actor ] [--reason ]
[--url ] [--token-env ] [--tenant ]
```
Mark an approval as denied. The agent typically aborts the gated action and returns an actionable failure.
## See also
- [Building agents → Approvals](/docs/building/approvals)
- [Policies and approvals](/docs/reference/policies-approvals)
---
# Artifacts and Checkpoints
Canonical: https://harness.techfabric.com/docs/cli/artifacts
List, fetch and inspect the artifacts and checkpoints a session produced, long after the run itself has finished.
**Artifacts** are files (Markdown, JSON, CSV, images, ...) that an agent publishes during a session via `session.artifact(name, content, options?)`. **Checkpoints** are explicit named save points.
## `fh artifacts`
```
fabric-harness artifacts [--json]
```
List artifacts for a session. The default output shows id, name, content type, byte size, and creation time.
## `fh artifact get`
```
fabric-harness artifact get [--out ]
```
Fetch a single artifact. If `--out` is omitted, the artifact prints to stdout (use redirection for binary content). If both an id and a name match, the id wins.
```sh
fh artifact get ask-1f4f... report.md --out ./reports/report.md
```
## `fh checkpoints`
```
fabric-harness checkpoints
```
List checkpoints for a session. Each entry includes label, created time, and (when available) a sandbox snapshot reference.
## See also
- [Building agents → Artifacts](/docs/building/artifacts)
- [Artifacts and observability](/docs/reference/build-manifest)
---
# fh build
Canonical: https://harness.techfabric.com/docs/cli/build
Compile a workspace and emit a deployment manifest, which is the artifact every deploy target consumes.
```
fabric-harness build [--target ] [options]
```
Compiles `.fabricharness/jobs/` and `.fabricharness/agents/`, then emits a deployment artifact under
`.fabricharness/build//` with a schema-v2 `manifest.json`.
Build manifests use the current time during normal local development. Set the standard
`SOURCE_DATE_EPOCH` variable to whole Unix seconds for reproducible artifacts. In GitHub Actions,
TechFabric Harness automatically uses the `GITHUB_SHA` commit timestamp when that commit is available,
so rebuilding the same source commit produces the same manifest and directory digest.
## Targets
| Target | Output |
| --- | --- |
| `node` (default) | Shared v2 Node HTTP server with finite `/jobs/:name` and persistent `/agents/:name/:id` routes. |
| `temporal-worker` | A worker entrypoint that registers Fabric workflows + activities against a Temporal task queue. |
| `docker` | `Dockerfile` + Node bundle ready for `docker build`. |
| `cloudflare` | Worker entrypoint, Durable Object session store, Sandbox container binding, `wrangler.jsonc`. |
| `foundry-hosted-agent` | `Dockerfile`, `azure.yaml`, `infra/main.bicep`, `foundry-agent.yaml`, server bundle. |
| `databricks-app` | Databricks App bundle, `app.yaml`, Lakebase-aware server, and Declarative Automation Bundle config. |
| `databricks-serving` | MLflow pyfunc proxy and Model Serving deployment assets. |
The remaining targets (`aks`, `aca`, `render`) are listed with their capability status in
[Deploy targets](/docs/ecosystem/targets).
## Validation
Every build statically validates the workspace before emitting anything: skill
and role references, imports the chosen target's runtime cannot provide, and
files in agent directories that export no definition. See
[`fh doctor --workspace`](/docs/cli/doctor#workspace-validation) for the full
check list.
An error-severity finding fails the build:
```txt
Workspace validation failed for target cloudflare:
error .fabricharness/jobs/report.ts:1 [target-incompatible-import]
"node:fs/promises" is not available on the cloudflare target.
Move this work behind a tool that runs in the sandbox, or build for a target whose runtime provides it.
```
That artifact would previously have built cleanly and failed at runtime, after a
deploy. Validation runs before any output is written, so a rejected build leaves
no half-built artifact for a deploy step to pick up.
The same workspace still builds for a target whose runtime provides the import —
findings are target-specific, not global. Use `--validate warn` to record
findings as build warnings instead, or `--validate off` to skip the check.
## Options
| Flag | Description |
| --- | --- |
| `--out ` | Override the output directory (default `.fabricharness/build/`). |
| `--env ` | Load `.env`-style variables before building (e.g. `--env .env.production`). Auto-loads `.env`/`.env.local`; shell env wins. |
| `--no-clean` | Skip cleaning the output directory before emit. |
| `--validate ` | Static workspace validation before emit. Defaults to `error`. |
| `--sbom` | Emit a CycloneDX SBOM via Syft when available. |
| `--sbom-required` | Fail if Syft is missing. |
| `--provenance` | Emit `provenance.json` for the build artifact. |
| `--attestation` | Emit `attestation.intoto.jsonl` with a manifest digest subject. |
| `--sign-provenance` | Sign `provenance.json` via `cosign sign-blob`. Implies `--provenance`. |
| `--signing-key ` | cosign key path or env reference (default `env://COSIGN_PRIVATE_KEY`). |
| `--docker-build` | Run `docker build` after emitting `--target docker`. |
| `--docker-push` | Run `docker push` after `--docker-build`. |
| `--docker-tag ` | Tag for the built/pushed image. |
| `--image-sbom` | Emit an image SBOM via Syft. |
| `--image-sbom-required` | Fail if image SBOM cannot be produced. |
## Examples
### Node server artifact
```sh
fh build --target node
node .fabricharness/build/node/dist/server.mjs
```
### Docker image with SBOM
```sh
fh build --target docker --docker-build --docker-tag myorg/agents:latest --sbom --image-sbom
```
### Cloudflare scaffold
```sh
fh build --target cloudflare
cd .fabricharness/build/cloudflare
npm install @cloudflare/sandbox @fabric-harness/cloudflare @fabric-harness/sdk
npx wrangler dev
```
### Foundry Hosted Agent scaffold
```sh
fh build --target foundry-hosted-agent
cd .fabricharness/build/foundry-hosted-agent
azd up
```
### Signed provenance + attestation
```sh
export COSIGN_PRIVATE_KEY=$(cat cosign.key)
fh build --target node --provenance --sign-provenance --attestation
```
## What's in `manifest.json`
- The TechFabric Harness version and target,
- separate `jobs` and persistent `agents` collections with schemas, models, and triggers,
- declared sandbox backend(s),
- session-store backend, if configured,
- digest of the artifact,
- optional provenance/attestation metadata.
`createdAt` is the wall-clock build time by default, the `SOURCE_DATE_EPOCH` time when explicitly
configured, or the source commit time in GitHub Actions. It is therefore stable for protected
exact-commit release rebuilds without hiding differences in any artifact file.
The CLI exposes manifests through `fh builds`. A Node server can read workspace-local artifacts at
`GET /builds/:target/manifest`; Cloudflare exposes its embedded manifest at `GET /manifest`.
See also: [Build and run artifacts](/docs/deployment/build-artifacts), [Build manifest](/docs/reference/build-manifest), [`fh builds`](/docs/cli/builds), [`fh verify-attestation`](/docs/cli/builds#verify).
---
# Builds and Verification
Canonical: https://harness.techfabric.com/docs/cli/builds
List the build manifests you have emitted and verify the provenance and attestations attached to each one.
## `fh builds`
```
fabric-harness builds
```
List every build manifest emitted under `.fabricharness/build/`. Output includes target, output dir, agent count, manifest digest, and timestamps.
## `fh verify-attestation`
```
fabric-harness verify-attestation
```
Verify the in-toto attestation produced by `fh build --attestation`. The argument can be a build directory (the CLI finds `attestation.intoto.jsonl`) or a direct path to the attestation file.
```sh
fh verify-attestation .fabricharness/build/node
fh verify-attestation .fabricharness/build/node/attestation.intoto.jsonl
```
## `fh verify-provenance`
```
fabric-harness verify-provenance
```
Verify the SLSA-style provenance produced by `fh build --provenance`. Optionally signed with cosign via `--sign-provenance`.
```sh
fh verify-provenance .fabricharness/build/node
```
If a `provenance.sig` is present, the CLI invokes `cosign verify-blob` with the configured public key. Otherwise it validates structure and digests only.
## See also
- [`fh build`](/docs/cli/build) — emit artifacts and provenance.
- [Build manifest](/docs/reference/build-manifest) — manifest schema.
- [Security hardening](/docs/reference/security-hardening) — when to require provenance.
---
# Compatibility Contract
Canonical: https://harness.techfabric.com/docs/cli/compatibility
Negotiate TechFabric Harness CLI features safely from Desktop, CI, and other automation.
TechFabric Harness exposes a machine-readable contract so an integrating application can verify behavior before it runs a command. Use capability negotiation together with a supported version range. A version check alone cannot prove that a build includes the commands, targets, and protocol versions your application needs.
```bash
fh capabilities --json
```
The command writes JSON to stdout and does not inspect a workspace or read credentials. Abridged
output (the installed version and arrays vary by release):
```json
{
"schemaVersion": 1,
"product": "fabric-harness",
"cliVersion": "",
"protocolVersion": 1,
"buildManifestVersion": 2,
"commands": ["init", "run", "build", "deploy", "doctor", "capabilities"],
"buildTargets": ["node", "temporal-worker", "docker", "cloudflare", "databricks-app", "databricks-serving"],
"runtimes": ["inline", "temporal"],
"features": ["cli.preview-deploy", "buzz.doctor", "databricks.app-build", "databricks.app-deploy", "databricks.ai-gateway"]
}
```
The arrays can gain values in a compatible release. Consumers should require the values they use and ignore values they do not recognize. A changed `schemaVersion`, `protocolVersion`, or `buildManifestVersion` requires an explicit compatibility decision.
```mermaid
flowchart LR
A[Desktop or automation] --> B[Resolve fh executable]
B --> C[Check supported semver range]
C --> D[Run fh capabilities --json]
D --> E{Required protocol and features present?}
E -->|Yes| F[Run init, build, deploy, or run]
E -->|No| G[Stop with an upgrade or compatibility error]
```
## Consumer example
```ts
import { execFileSync } from 'node:child_process';
const capabilities = JSON.parse(
execFileSync('fh', ['capabilities', '--json'], { encoding: 'utf8' }),
);
const required = [
'cli.preview-deploy',
'buzz.doctor',
'databricks.app-build',
'databricks.app-deploy',
'databricks.ai-gateway',
];
if (capabilities.protocolVersion !== 1) {
throw new Error(`Unsupported Harness protocol ${capabilities.protocolVersion}`);
}
const missing = required.filter((feature) => !capabilities.features.includes(feature));
if (missing.length > 0) {
throw new Error(`Harness is missing: ${missing.join(', ')}`);
}
```
## Release testing
Fabric Desktop tests both its minimum certified Harness release and the current npm `latest`. Harness contributors can test an unpublished CLI against the adjacent Desktop checkout:
```bash
pnpm test:desktop-compat
```
The command builds and packs `@fabric-harness/cli`, installs that tarball in isolation, then drives Desktop's real Databricks bridge through scaffold, mock run, build validation, and deploy preview for every bundled template. It also verifies the capability contract and scans generated files for leaked credentials.
Keep application templates on the compatible ranges emitted by the installed CLI. Fabric packages
are independently versioned, so matching majors are not a compatibility signal. CI builds the
executable CLI, scaffolds every platform template, and compares each generated range with the
corresponding workspace package so a release cannot leave a fresh project on a stale or invented
version. See [release and documentation status](/docs/reference/release-status) for the current
published versions and runtime requirements.
---
# Fiber terminal console
Canonical: https://harness.techfabric.com/docs/cli/console
Operate jobs, persistent agents, streams, tool calls, and approvals from a keyboard-first terminal UI.
**Fiber** is TechFabric Harness's optional terminal UI. Open it with `fh fiber`. Fiber is a client of the public Fabric HTTP protocol: it does not start a second
runtime or bypass server policy. Authentication, tenant isolation, durable admission, offsets, tool
events, and approval resolution behave the same as other clients.
For the shortest local loop, start the Node dev server and Fiber together:
```sh
FABRIC_HARNESS_API_TOKEN=local-token fh dev --mock --console
```
`--console` is intentionally limited to `fh dev --target node`. Cloudflare and Temporal development
processes keep their existing external lifecycle and can be observed by running `fh fiber --url ...`
separately.
Start the server in one terminal:
```sh
FABRIC_HARNESS_API_TOKEN=local-token fh dev --mock
```
Then open the interactive console:
```sh
FABRIC_HARNESS_API_TOKEN=local-token fh fiber
```
Fiber reads `/admin/agents` and presents an explicit selected row, connection state, activity pane,
and input line. Use the arrow keys and Enter to choose a finite job or persistent agent. Persistent
agents use `--id ` when supplied and `default` otherwise. Messages are admitted
asynchronously. You can keep typing while a turn is active: Fiber shows pending prompts and admits
them in order through the server's durable FIFO path. It follows each returned offset and renders
assistant messages, tool calls, tool results, approval events, errors, and terminal settlement.
Model and tool output is stripped of terminal control sequences before it is rendered. Color is
supplemental—selection and run state are also labeled in text.
Use Escape to return to target selection when the queue is idle. Ctrl-C aborts the active client
request, clears prompts that have not been admitted, and exits. Fiber never prints bearer tokens.
## One-shot jobs
```sh
fh fiber --job report --input '{"topic":"weekly usage"}'
```
Input must be JSON. The job uses the authenticated `/jobs/:name` route and prints the complete result
envelope.
## One-shot persistent messages
```sh
fh fiber \
--agent support \
--id customer-42 \
--session billing \
--message "Where is invoice 1007?"
```
Use `--url` for a deployed server, `--tenant` for explicit operator tenant selection, and
`--token-env` when the bearer token is stored under a different environment variable:
```sh
PROD_AGENT_TOKEN=... fh fiber \
--url https://agents.example.com \
--token-env PROD_AGENT_TOKEN \
--tenant acme \
--agent support --id customer-42 --message "Retry the export"
```
Tokens are read from the environment rather than command arguments, so they do not appear in shell
history or process listings.
## Plain and non-interactive terminals
Use `fh fiber --plain` for the line-oriented prompt. CI and redirected stdin are not TTYs,
so Fiber fails with actionable guidance in those environments. One-shot `--job` and
`--agent --message` commands remain non-interactive and do not load Fiber.
## Interactive commands
| Command | Effect |
| --- | --- |
| `:approvals ` | List approval requests for a session. |
| `:approve [reason]` | Approve through the server's RBAC route. |
| `:reject [reason]` | Reject through the same route. |
| `:quit` | Close the terminal client. |
The identity used to resolve an approval comes from the authenticated server principal. Text typed
after the approval id is a reason, not an actor override.
## Failure behavior
- Connection and HTTP failures remain visible in the status and activity panes; they never become
successful output.
- Additional prompts remain visible in Fiber's queue and are admitted in order; a failed prompt does
not prevent the next queued prompt from running.
- Closing Fiber aborts active polling and discards prompts that were not yet admitted. Work already
durably admitted remains governed by the server and can be inspected by run/submission id.
- ANSI, OSC, and control bytes in model/tool output are removed before terminal rendering.
- Missing or invalid authentication is reported by the server and Fiber preserves the server's
public, redacted error message.
- Exiting Fiber launched by `fh dev --console` closes the watcher and dev server cleanly.
---
# fh deploy
Canonical: https://harness.techfabric.com/docs/cli/deploy
Build and deploy a TechFabric Harness workspace to Databricks, Node, containers, Kubernetes, Cloudflare, Foundry, Temporal, or Render.
```sh
fh deploy --target [--preview] [--profile ] [--env ] [--env-name ] [--with-app]
```
`fh deploy` selects a target driver, builds the workspace for that target, runs its prerequisite
checks, and invokes the platform deployment command. Use `--preview` first in automation or a new
account; it prints the driver actions without changing the target platform.
## Targets
| Target | Deployment path |
| --- | --- |
| `databricks-app` | Build and deploy a Databricks App through its Declarative Automation Bundle |
| `databricks-serving` | Build the MLflow proxy and deploy a Databricks Model Serving endpoint |
| `node` | Run the built Node server locally |
| `docker` | Build and run the generated container |
| `cloudflare` | Deploy the generated Worker |
| `aks`, `aca`, `aci` | Deploy to Azure Kubernetes Service, Container Apps, or Container Instances |
| `foundry-hosted-agent` | Deploy the generated Microsoft Foundry Hosted Agent assets |
| `k8s` | Apply the generic Kubernetes artifact |
| `temporal-worker` | Deploy the generated Temporal worker |
| `render` | Deploy the generated Render service |
## Options
| Flag | Behavior |
| --- | --- |
| `--target ` | Required deployment target |
| `--preview` | Print intended commands without executing them |
| `--profile ` | Databricks CLI profile used by App/Serving deployment |
| `--env ` | Load an explicit `.env`-style file before config and preflight |
| `--env-name ` | Load `.env.` and matching workspace environment overrides |
| `--with-app` | After `databricks-serving` succeeds, also build and deploy `databricks-app` |
Shell variables win over env-file values. Secrets are passed to provider commands through the
environment; they are not written to build manifests.
## Databricks Apps
```sh
fh doctor --target databricks-app
fh deploy --target databricks-app --preview --profile production
fh deploy --target databricks-app --profile production --env .env.production
```
The App driver expects the Databricks CLI, authentication, and a workspace bundle generated by the
`databricks-app` build. When the App declares capabilities in
`FABRIC_DATABRICKS_APP_CAPABILITIES`, doctor fails closed if required Lakebase, Genie, OBO, or cost
bindings are absent.
## Databricks Serving, optionally paired with an App
```sh
fh doctor --target databricks-serving
fh deploy --target databricks-serving --preview
fh deploy --target databricks-serving --with-app
```
Serving deployment runs a network-free doctor preflight before any provider mutation. Missing
Python/MLflow, Databricks authentication, or package output stops the command with an actionable
error. `--with-app` is accepted only with `databricks-serving`; the App deployment starts only after
Serving succeeds.
Existing endpoint deployments are safe to retry while Databricks is converging an earlier change.
The driver waits for endpoint readiness before replacing the served model, waits again before
updating AI Gateway inference-table configuration, and waits for that gateway update to settle
before reporting success. New endpoints follow the same readiness boundary between creation and
the gateway update. A readiness or update failure stops the deploy and does not start `--with-app`.
## Failure behavior
- A missing or unknown target fails before a driver runs and prints close target suggestions.
- An invalid flag or using `--with-app` on another target fails without deploying.
- Databricks preflight failures stop before build/deploy provider calls.
- A target command failure is returned as a failed deploy; the CLI does not claim rollback of
resources already accepted by the provider.
See [Databricks deployment](/docs/deployment/databricks), [`fh build`](/docs/cli/build), and
[`fh doctor`](/docs/cli/doctor).
---
# fh dev
Canonical: https://harness.techfabric.com/docs/cli/dev
Start a local HTTP and SSE server for a workspace, so you can drive an agent from a browser or curl while you build it.
```
fabric-harness dev [--target node|cloudflare|temporal-worker] [options]
```
`fh dev` boots the selected target's development server and watches both `.fabricharness/jobs/` and
`.fabricharness/agents/`. The Node target uses the same v2 server as Node-derived builds. Cloudflare
uses Wrangler for finite jobs and Durable Object-backed persistent agents.
## Routes
- `GET /health`
- `GET /ready`
- `POST /jobs/:name` — invoke a finite job with its JSON input.
- `POST /agents/:name/:id` — durably admit a persistent message and return `202`.
- `GET /agents/:name/:id/stream?offset=...` — tail persistent conversation records.
- `GET /builds/:target/manifest` — read a build manifest still under the workspace.
Cloudflare also exposes `GET /manifest`; persistent instances use Durable Objects for FIFO
submissions, leases, conversation streams, abort, and deletion.
## Options
| Flag | Description |
| --- | --- |
| `--target ` | Default `node`. |
| `--env ` | Load `.env`-style variables. Repeatable; shell env wins. |
| `--mock` | Use the deterministic mock model provider for every request. |
| `--console` | Open the Fiber terminal UI after the Node target starts; unsupported for Cloudflare and Temporal targets. |
| `--host ` | Listen address (default `127.0.0.1`). |
| `--port ` | Listen port (default `3000`; the CLI selects the next available port when it is busy). |
| `--auth-token-env ` | Require `Authorization: Bearer $$VAR` for invocations. |
| `--max-body-bytes ` | Reject request bodies larger than `n`. |
| `--rate-limit-window-ms ` | Rate-limit window in ms. |
| `--rate-limit-max ` | Rate-limit max requests per window. |
## Example
```sh
fh dev --mock --port 4000 --auth-token-env FABRIC_DEV_TOKEN
```
Or keep the server and operator view in one process:
```sh
FABRIC_DEV_TOKEN=local-token fh dev --mock --console --auth-token-env FABRIC_DEV_TOKEN
```
```sh
curl -X POST \
-H "Authorization: Bearer $FABRIC_DEV_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"question":"What is Temporal?"}' \
http://localhost:4000/jobs/ask
```
## When to use which
- **`fh run`** — one-shot invocations, scripts, CI smoke tests.
- **`fh dev`** — local development with hot reload, finite job webhook testing, and durable
persistent-agent streams.
See [HTTP Server](/docs/reference/http-server) for the complete Node route table.
## Transport-free runs
`fh run` loads the selected public definition directly and does not start an HTTP listener. Finite
agents continue to accept JSON payloads. Persistent agents accept a message and durable instance
identity controls:
```bash
fh run support --id customer-42 --new \
--data '{"account":"acme"}' --prompt "Triage this request"
# Continue only the incarnation printed by the first command.
fh run support --id customer-42 --uid inst_... --prompt "Continue"
```
Initial data is schema-validated before instance identity or conversation records are committed.
`--new` fails if the instance already exists, `--uid` fails if the instance is missing or has been
replaced, and the command prints the resolved UID to stderr. SIGINT/SIGTERM cancels the active
finite or persistent run and interaction-scoped dynamic sandboxes are reclaimed when it settles.
Projects that already use Vite 8 can instead add `fabricHarness()` from
`@fabric-harness/vite`; see [Vite integration](/docs/reference/vite-integration). This is optional and
uses the same discovery/build registry as `fh dev` and `fh build`.
---
# docs
Canonical: https://harness.techfabric.com/docs/cli/docs
List, read, and search TechFabric Harness documentation as Markdown from the CLI.
The docs commands consume the published `llms.txt`, `llms-full.txt`, and per-page Markdown routes.
They work in a source checkout or any project with the CLI installed.
## List routes
```sh
fh docs list
fh docs list --query databricks
fh docs list --json
```
Each entry includes its stable slug, title, and description. Duplicate priority/index entries are
removed.
## Read a page
```sh
fh docs read databricks/quickstart
fh docs read /docs/operating/auth
fh docs read https://harness.techfabric.com/docs/building/http-applications
```
Output is the page's Markdown source, suitable for a terminal pager or an LLM context file:
```sh
fh docs read databricks/integrations | less
fh docs read reference/api > /tmp/fabric-api.md
```
## Search the corpus
```sh
fh docs search unity catalog lineage
fh docs search durable approval --limit 5
fh docs search lakebase credential --json
```
Search ranks title and description matches before full-corpus occurrences and prints a nearby text
snippet. Use `--base-url` to test a preview or self-hosted documentation worker with the same routes.
---
# fh doctor
Canonical: https://harness.techfabric.com/docs/cli/doctor
Diagnose a workspace before you blame the agent, covering tool wiring, configuration and live model connectivity.
```
fabric-harness doctor [--target node|temporal-worker|databricks-app|databricks-serving|buzz] [--model provider/model] [--getting-started] [--tools] [--workspace] [--live] [--json]
```
`fh doctor` validates that your workspace is set up correctly and (optionally) that a real model can be invoked.
## Options
| Flag | Description |
| --- | --- |
| `--target ` | Validate the chosen run, deployment, or first-party integration target. |
| `--model ` | Probe a specific model (combine with `--live`). |
| `--getting-started` | Check Node version, ESM package setup, Fabric dependencies, npm scripts, and `.fabricharness/agents` discovery. |
| `--tools` | Enumerate built-in tools and required binaries (`docker`, `gh`, etc.). |
| `--workspace` | Statically validate the workspace: skill and role references, target-incompatible imports, and files in agent directories that export no definition. |
| `--live` | Make a bounded live model request, or run read-only relay and durable-state certification for `--target buzz`. |
| `--json` | Emit a machine-readable report. |
## Examples
### Local readiness
```sh
fh doctor --getting-started --tools
```
### Live model check
```sh
cp .env.example .env.local
# edit .env.local and set OPENAI_API_KEY=...
fh doctor --live --model openai/gpt-5.5
```
### Workspace validation
```sh
fh doctor --workspace
fh doctor --workspace --target cloudflare
```
Reads source without executing it, so it needs no credentials, no network, and
does not import your agent modules. It reports:
| Code | Severity | Meaning |
| --- | --- | --- |
| `skill-not-found` | error | A `session.skill('name')` call whose name no skill defines — neither a `.fabricharness/skills/` directory, a build-time `SKILL.md` import, nor a `defineSkill()` in the same file. |
| `role-not-found` | error | An `init({ role })` naming a role no file defines. |
| `target-incompatible-import` | error | A Node built-in the chosen target's runtime does not provide. Only reported when `--target` names such a runtime. |
| `no-default-export` | warning | A file in an agent directory exporting no default definition. |
Errors exit non-zero, so this composes into CI. Warnings do not.
Without `--target`, target-specific checks are skipped — a workspace that only
ever runs on Node should not be warned about its `node:fs` import.
Two checks are deliberately conservative, because a false positive on correct
code is worse than a missed finding: a skill imported from a package cannot be
resolved from source, so the skill check is suppressed for that file; and the
role check is scoped to `init({ ... })` options, since `role:` is also the LLM
message discriminator.
### Temporal target
```sh
fh doctor --target temporal-worker
```
The doctor checks Temporal connectivity using config/env defaults — make sure `FABRIC_TEMPORAL_ADDRESS` and your task queue are correct.
### Databricks App bindings
```sh
fh doctor --target databricks-app --json
fh doctor --target databricks-app --live --json
```
Every Databricks report includes a workspace-readiness section. It uses four intentionally narrow
classifications:
| Classification | Meaning |
| --- | --- |
| `contract-supported` | The installed Harness code has an offline-tested adapter. It says nothing about this workspace's grants or enrollment. |
| `live-certified-here` | The installed package, declared cloud, and region exactly match retained public certification evidence. |
| `preview-enabled` | A bounded read-only live probe reached that preview API in this workspace. |
| `not-established` | The command cannot prove the claim. Treat it as unknown, not as disabled. |
Offline mode reports the configured host, inferred cloud/region, installed package version, exact
public certification relationship, and contract support for Apps, Unity Catalog/OBO, SQL, Genie,
Model Serving, AI Gateway, Lakebase, managed MCP, Supervisor Agents, and managed memory. `--live`
adds bounded read-only identity and service discovery probes. A successful service probe proves
reachability under the probing principal; it does not prove every end user's Unity Catalog grant.
The command never prints tokens or connection strings.
Set `FABRIC_DATABRICKS_APP_CAPABILITIES` to the comma-separated capabilities the App claims:
`lakebase`, `genie`, `obo`, and/or `system-tables-cost`. Doctor fails closed when a claim lacks its
required endpoint, database, Genie space, OBO opt-in, warehouse, or cost-scope binding. It never
prints credential values. The report also includes a redacted effective inventory:
- identity kind and credential source;
- enabled model-callable tools with effects and service names;
- static or input-bound governed resources;
- approval coverage;
- persistence, scheduler identity, lease, and catch-up modes; and
- unsupported or dynamically resolved capabilities.
Persistent-agent initializers are interaction-scoped, so doctor reports their tool inventory as
dynamic instead of executing an initializer with a fabricated identity.
### Buzz bridge
```sh
fh doctor --target buzz --json
fh doctor --target buzz --live
```
The offline preflight validates Node 22/native WebSocket support, the relay and forwarding URLs,
redacted Nostr-key and HMAC-secret shape, the server-owned community/channel allowlist, durable
PostgreSQL or Lakebase configuration, and the current v2 generated recipe. It specifically rejects
the old process-memory tail and recipes that omit decision-receipt or dead-letter wiring.
`--live` is a read-only certification profile. It performs a bounded NIP-11 probe, requires NIP-01
and NIP-42, signs NIP-98 queries with the configured adapter identity, verifies the returned NIP-01
event integrity, and requires that identity in both the authoritative relay roster and every
configured channel member list. It then connects to PostgreSQL or Databricks Lakebase with a
single-connection pool and reads only content-free operational evidence: relay cursor lag,
cursor-update age, unresolved dead-letter depth, and prepared/published decision-card counts.
The default limits are 300 seconds of relay lag, 300 seconds since the cursor update, and zero
unresolved dead letters. Set `BUZZ_MAX_RELAY_LAG_SECONDS`, `BUZZ_MAX_CURSOR_AGE_SECONDS`, and
`BUZZ_MAX_DEAD_LETTERS` to explicit non-negative integers when the deployment SLO differs. The JSON
report is safe for CI evidence: it never includes event content, private keys, forwarding secrets,
database URLs, or database passwords. Relay and channel membership prove transport admission only;
they do not grant Fabric action authority.
## What it checks
- Workspace root resolution.
- Agents discovered.
- Node/package/ESM/dependency setup (when `--getting-started`).
- Built-in tool schemas and binaries on PATH (when `--tools`).
- Model provider configuration.
- Optional live model round-trip when `--live`.
- Optional Temporal connectivity for `--target temporal-worker`.
- Databricks CLI, authentication, and claimed App bindings for Databricks deployment targets.
- Buzz bridge configuration, v2 durability wiring, and optional read-only relay/Lakebase certification.
---
# fh run
Canonical: https://harness.techfabric.com/docs/cli/run
Execute a workspace agent, or any standalone agent file, against a local Node runtime or a Temporal worker.
```
fabric-harness run [options]
```
Runs the named agent. The CLI loads the agent module, validates the input payload, calls `run({ init, input, payload })`, and validates the output.
`` is either a workspace agent name (resolved from `.fabricharness/jobs/`) or a path to an agent file (see [Single-file mode](#single-file-mode)).
## Single-file mode
When `` resolves to an existing `.ts`, `.mts`, `.js`, or `.mjs` file, `fh run` executes that file directly — no `.fabricharness/` workspace, no `config.ts`, no credentials required:
```sh
fh run ./agent.ts --message "hello"
```
- The file is transpiled on the fly. Package imports (including `@fabric-harness/sdk`) resolve from the nearest `node_modules` to the file, falling back to the CLI's own install.
- Every exported `defineAgent()`/`createAgent()` value is discovered. With exactly one export it runs directly; with several, pass `--agent ` to select one (the error lists the exported names). Persistent `createAgent()` agents still require a workspace.
- When no model credentials resolve (`--model`, `FABRIC_MODEL`, or provider keys), the run falls back to the mock model and says so on stderr, so the command round-trips offline. `--mock` keeps its explicit meaning.
- The reply (and any structured output) prints on stdout; the session id prints on stderr as `[fabric-harness] session completed (memory store — not persisted).` — single-file runs use an in-memory store, so there is nothing to resume.
- An explicit path that does not exist (`./missing.ts`) is a hard error; a bare name still falls back to workspace agent resolution.
## Options
| Flag | Description |
| --- | --- |
| `--agent ` | Single-file mode: select one export when the file exports several agents. |
| `--id ` | Session identifier. **Reusing an existing id resumes that persisted session** — new turns append to its history. If omitted, a session id is generated using `${idPrefix}-${uuid}`. |
| `--resume ` | Explicit alias for `--id `: continue the persisted session ``. |
| `--fork ` | Fork persisted session `` into a new session: copies its history into a fresh id (pass `--id` to name the fork), then continues the copy. Requires a persisted store. |
| `--target ` | Where the agent runs. Defaults to `node`. `temporal-worker` switches `--runtime` to `temporal`. |
| `--runtime ` | Equivalent lower-level flag. Usually set via `--target`. |
| `--model ` | Override the agent's default model, e.g. `openai/gpt-5.5`. |
| `--mock` | Use the deterministic mock model provider — no credentials needed. |
| `--mock-script ` | Scripted mock responses from a JSON fixture file (aimock-style `{ match, response }` entries). Implies `--mock`; see [Test Without Credentials](/docs/building/test-without-credentials#scripted-mock-responses). |
| `--cwd ` | Default sandbox/session working directory for this run. Relative paths stay scoped inside the sandbox workspace. |
| `--uid ` | Persistent agents: require an existing incarnation with this id. |
| `--new` | Persistent agents: require creation of a new instance. |
| `--data ` | Persistent agents: immutable initial data for a new instance. |
| `--tenant ` | Tenant identifier for this run. |
| `--prompt ` | Message text for persistent agents; also the prompt for `--runtime temporal` mode. |
| `--payload ''` | Pass an inline JSON payload. |
| `--payload-file ` | Read JSON payload from a file. |
| `--stdin` | Read JSON payload from stdin. |
| `--set key=value` | Set a payload field. May be repeated. |
| `--` | Shortcut for setting a payload field, e.g. `--question "..."`. |
| `=` | Positional shortcut for setting a payload field, e.g. `question="..."`. |
| `--env ` | Load `.env`-style variables. Repeatable; shell env wins. |
## Resuming and forking sessions
Every successful workspace run prints its session id on stderr with a resume hint:
```text
[fabric-harness] session sess-1 completed. Resume: fh run hello --resume sess-1
```
With a durable store (`store: { backend: 'file' }`, the default), passing the same id again — via `--id` or the explicit `--resume` alias — continues the session: the model sees the prior history and new entries append to it. `--fork ` instead copies a persisted session into a new id and continues the copy, leaving the original untouched:
```sh
fh run hello --id sess-1 --message "first"
fh run hello --resume sess-1 --message "second" # continues sess-1
fh run hello --fork sess-1 --id sess-1-copy --message "third" # branches a copy
```
With `store: { backend: 'memory' }` nothing is persisted, and the run says so explicitly (`session completed (memory store — not persisted).`) — `fh sessions` staying empty after a memory-store run is expected, not data loss. Resume and fork require a persisted store.
## Payload precedence
When more than one source is given, fields merge in this order (later wins):
```
--payload → --payload-file → --stdin → --set / -- / key=value
```
## Examples
### Single file, no workspace
```sh
fh run ./agent.ts --message "hello"
fh run ./multi.ts --agent beta --message "hello"
```
### Inline JSON
```sh
fh run ask --payload '{"question":"What is Temporal?"}'
```
### Field shortcut
```sh
fh run ask --question "What is Temporal?"
fh run ask question="What is Temporal?"
fh run ask --set question="What is Temporal?"
```
### Working directory
```sh
fh run code --cwd /workspace/project --prompt "Run tests and summarize failures"
fh run code --cwd packages/core --payload '{"prompt":"Inspect this package"}'
```
`--cwd` is a sandbox cwd, not a host directory switch. Use it to make file/shell tools default to a repository subdirectory.
### From a file or stdin
```sh
fh run ask --payload-file input.json
echo '{"question":"hi"}' | fh run ask --stdin
```
### Real model
Put provider keys once in a repo/workspace `.env.local`; TechFabric Harness auto-loads it and shell env still wins.
```sh
cp .env.example .env.local
# edit .env.local and set OPENAI_API_KEY=...
fh run ask --model openai/gpt-5.5 --question "What is Temporal?"
```
Use explicit `--env ` only for test/CI overrides.
### Temporal worker target
```sh
# In a separate terminal:
fh temporal-worker
# Then:
fh run ask --target temporal-worker --id ask-001 --prompt "What is Temporal?"
```
## What happens during `run`
1. Resolve workspace root by walking up to find `.fabricharness/`. (Single-file mode skips this and the next step — the file is loaded directly with SDK defaults.)
2. Load `.fabricharness/config.ts` and merge with env and CLI flags.
3. Resolve agent path, target, runtime, model, id, and sandbox cwd.
4. Apply env model provider (e.g. set provider credentials).
5. Validate input payload against the agent's input schema (metadata agents only).
6. Call the agent's `run({ init, input, payload })`.
7. Validate output against the output schema (metadata agents only).
8. Persist session and print the session id on stderr (with a resume hint, or a `not persisted` notice on the memory store).
## See also
- [`fh agents` / `fh describe`](/docs/cli/agents) — discover what's runnable.
- [Sessions](/docs/cli/sessions) — inspect what just happened.
- [Configuration](/docs/getting-started/configuration) — defaults and precedence.
---
# Sessions, Inspect, Logs, Replay, Metrics, Compact
Canonical: https://harness.techfabric.com/docs/cli/sessions
Inspect what actually happened during a run, with logs, replay, metrics and compaction over a session already finished.
Every `fh run` and every dev-server invocation persists a session under `.fabricharness/sessions/` (or your configured store). The CLI ships several commands for reading those sessions. With `store: { backend: 'memory' }` nothing is persisted — the run says so on stderr and `fh sessions` stays empty; that is expected, not data loss. To continue a persisted session, re-run with the same `--id` (or `--resume `); see [`fh run`](/docs/cli/run#resuming-and-forking-sessions).
## `fh sessions`
```
fabric-harness sessions
```
Lists persisted sessions. Output includes session id, agent name, last update time, and entry counts.
## `fh inspect`
```
fabric-harness inspect
```
Shows the full structured session: prompts, assistant messages, tool calls, tool results, shell commands, task starts/ends, approvals, compactions, checkpoints, and artifacts.
## `fh logs`
```
fabric-harness logs