Test Without Credentials

Everything on this page runs offline with zero API keys. Reach for a real model only when the behavior under test depends on the model itself.

Mock model from the CLI

--mock injects the deterministic mock provider into fh run and fh dev. Discovery, input/output schema validation, the model loop, and HTTP routing all still execute:

fh run hello --name Preetham --mock
# Mock response: Say hello to Preetham.

fh dev --mock --port 3000
curl http://localhost:3000/jobs/hello \
  -H 'content-type: application/json' \
  -d '{"name":"Preetham"}'
# {"result":"Mock response: Say hello to Preetham.","runId":"…"}

A mock/-prefixed configured model resolves the same way without any flag — fh init --model mock/test-model writes model: "mock/test-model" into .fabricharness/config.ts, and plain fh run hello (or the generated project's npm run run) executes against the mock. The production guard still applies: mock models throw when FABRIC_ENV/NODE_ENV is production unless FABRIC_ALLOW_MOCK_MODEL=1 is set.

Scripted mock responses

For deterministic multi-turn scripts — including tool loops — pass an aimock-style fixture file to fh run --mock-script <path> (implies --mock, no credentials, wins over every other model selection):

{
  "fixtures": [
    {
      "match": { "userMessage": "order", "hasToolResult": false },
      "response": {
        "toolCalls": [{ "name": "lookup_order", "arguments": { "id": "A-1" }, "id": "call-1" }]
      }
    },
    {
      "match": { "toolCallId": "call-1" },
      "response": { "content": "Order A-1 ships tomorrow." }
    }
  ]
}

fh run support --mock-script ./fixtures/ask.json --question "where is my order?"

Each entry is { match, response }. The first entry whose match is satisfied wins; requests that match nothing fall back to the echo behavior. Match fields (all specified fields must match):

Responses are { "content": "..." } and/or { "toolCalls": [{ "name", "arguments", "id?" }] }. Scripted tool calls execute for real — the mock emits the call, the loop runs your tool, and a later entry (matched on toolCallId, toolName, or sequenceIndex) answers from the result. Tool call ids default to mock-tool-<n>; set id explicitly to match a later toolCallId entry. The fixture file is either a bare array of entries or the { "fixtures": [...] } wrapper shown above.

The same script works in code via @fabric-harness/sdk/testing:

import { MockModelProvider, parseMockScript } from '@fabric-harness/sdk/testing';

const script = parseMockScript(fixtureJson); // validates, throws with the entry index
const provider = new MockModelProvider({ script });

Single-file runs auto-mock

fh run executes any standalone .ts/.js file that exports defineAgent() — no .fabricharness/ workspace, no config.ts, no install step beyond a resolvable @fabric-harness/sdk:

import { defineAgent, schema } from '@fabric-harness/sdk';

export default defineAgent({
  name: 'hello',
  input: schema.object({ message: schema.string() }),
  output: schema.string(),
  run: async ({ init, input }) => {
    const fabric = await init();
    const session = await fabric.session();
    return session.prompt(input.message);
  },
});

fh run ./agent.ts --message "hi"
# [fabric-harness] no model credentials resolved; using the mock model (set FABRIC_MODEL or pass --model for a real provider).
# [fabric-harness] session hello-4edbadf5-… completed (memory store — not persisted).
# Mock response: hi

When no model credentials resolve (--model, FABRIC_MODEL, or provider keys), the run falls back to the mock model and says so on stderr, so the command round-trips offline. Pass --model (or set FABRIC_MODEL) with a configured provider to run the same file against a real model. See fh run for multi-export files, payload flags, and error behavior.

MockModelProvider in code

@fabric-harness/sdk/testing exports the same provider the CLI uses:

import { MockModelProvider } from '@fabric-harness/sdk/testing';

const provider = new MockModelProvider();
const response = await provider.generate({
  messages: [{ role: 'user', content: 'ping' }],
});
// response.message.content === 'Mock response: ping'

The mock is deterministic: it echoes the latest user message as Mock response: <text> and summarizes tool results after tool turns. To exercise a tool loop, script tool calls in the prompt with a fabric-tool-calls fenced block:

const response = await provider.generate({
  messages: [{
    role: 'user',
    content: 'run the tool\n```fabric-tool-calls\n[{"tool":"lookup_order","input":{"id":"A-1"}}]\n```',
  }],
});
// response.toolCalls === [{ id: 'mock-tool-1', name: 'lookup_order', input: { id: 'A-1' } }]

The /testing entry point is shaped for test ergonomics and is not covered by the runtime SemVer commitment. Production code should not import from it.

Stub sessions for unit tests

StubFabricAgent and StubFabricSession are the in-process implementations behind init(). Construct them directly with the mock model to unit-test agent logic without a workspace, server, or credentials:

import { describe, expect, it } from 'vitest';
import { StubFabricAgent } from '@fabric-harness/sdk/testing';

describe('support agent', () => {
  it('answers through the session contract', async () => {
    const agent = new StubFabricAgent({ model: 'mock/test-model' });
    const session = await agent.session('demo');
    const reply = await session.prompt('hello from the test');
    expect(reply).toBe('Mock response: hello from the test');
  });
});

For workspace-level runs, runAgent({ agent, payload, mock: true }) from @fabric-harness/node covers the same ground through the full CLI pipeline — see Testing locally.

Mock sandbox handles

Remote sandbox adapters in @fabric-harness/connectors are structural: they adapt a provider SDK handle to the Fabric SandboxEnv contract. That makes the provider handle mockable — back it with the SDK's in-memory EmptySandboxEnv and the real adapter runs end-to-end offline:

import { EmptySandboxEnv } from '@fabric-harness/sdk';

const inner = new EmptySandboxEnv('/home/daytona');
await inner.writeFile('/home/daytona/welcome.txt', 'hello from the mock daytona sandbox\n');
// expose inner's readFile/writeFile/exec/… through the provider's handle shape

The canonical pattern lives in examples/remote-coding-agent (connectors/mock-remote.ts adapts EmptySandboxEnv to the RemoteSandboxApi contract), and each provider example ships its own structural mock with the live path gated behind an opt-in flag:

Each example's default pnpm run run executes against the mock with no credentials; the live path fails fast with a clear error when the flag is set without its credentials. See Sandbox connectors for the adapter contracts.

Evals with fh test

Eval suites are TypeScript modules named **/*.eval.ts. Run the agent with mock: true inside the suite runner and the whole suite is deterministic and offline:

import { containsTextScorer, defineEvalSuite } from '@fabric-harness/evals';
import { runAgent } from '@fabric-harness/node';

export default defineEvalSuite({
  name: 'hello-quality',
  cases: [
    { id: 'named-user', input: { name: 'Ada' }, expected: 'Ada' },
  ],
  runner: async ({ case: evalCase }) => {
    const run = await runAgent({ agent: 'hello', payload: evalCase.input, mock: true });
    return run.result;
  },
  scorers: [containsTextScorer()],
  passThreshold: 1,
});

fh test
# Eval Results
#
# hello-quality  PASS  100%  39ms
#   PASS  named-user  contains_text=1.00
#
# Overall: PASS

@fabric-harness/evals ships deterministic scorers — exactMatchScorer, containsTextScorer, regexMatchScorer, jsonShapeMatchScorer — plus llmAsJudgeScorer for model-graded checks (the one scorer that needs a real provider). fh test exits non-zero when any suite fails, so the same command gates CI. See Evaluations and fh test.

When to add credentials

Everything above stays green with no keys. Add a provider key in .env.local when you are validating real model behavior, then drop --mock:

echo 'OPENAI_API_KEY=sk-...' > .env.local
fh doctor --live --model openai/gpt-5.5
fh run hello --name Preetham

See Model providers for supported providers and credential resolution, and Live tests for the opt-in environment variable matrix used by the repo's own live suites.