dawn-fde-toolkit

Appearance

Testing ​

Voice-agent testing for the Avoca engagement. Hamming is the canonical testing platform; we do not fragment into Vapi Test Suites or other tools. Authoring happens entirely in the Avoca dashboard's Simulation Testing UI, not at hamming.ai directly. Hamming is the run engine and result store; you don't navigate there.

The model in one minute ​

ComponentRole
HammingDrives the caller side of a phone call: dials, plays a persona, listens. Stores test cases, guardrails, run history.
VapiRuns the agent side: real LLM, real prompt, real assistant. Receives the PSTN call. Invokes tools via avoca-next's dispatcher.
avoca-next dispatcherThe HTTP endpoint each Vapi tool POSTs to. Detects whether the call is a Hamming test (via caller-ID or test-phone URL marker). For test calls, looks up tool-call mocks in Supabase and returns the mock response instead of executing the real tool.
Avoca dashboard (v2)The authoring surface. Simulation Testing UI creates test cases (in Hamming) AND configures tool-call mocks (in Avoca's Supabase) in one dialog.

The agent's reasoning path is real. Only the leaf tool calls are substituted with mocks. This means tests exercise the real assistant + prompt + dispatcher, while not polluting production downstream systems.

Two testing surfaces ​

Same Hamming storage, different acceptance criteria.

SurfacePurposeHamming shapeAcceptanceCadence
Evaluation / A/B at scalePick between prompt or tool-call variantssamplingCount > 1, one case, N runs per variantThreshold delta or rankManual, when comparing variants
RegressionGate changes; previously-passing scenarios must keep passingtestCases + guardrails + scheduled-runsEvery named case passes (binary)Automatic, post-change or scheduled

A regression case names a single deterministic outcome; an eval case names a behavior under uncertainty and grades it statistically. Both go through Avoca's Simulation Testing UI; they differ in sampling count and acceptance criteria, not authoring surface.

For the design rationale, see insight_two_voice_testing_surfaces.

Prerequisites at a glance ​

Before any team can run Hamming tests, three things have to be in place:

  1. A test phone provisioned for the voice assistant. Done via Provision Test Phone Number admin action (actions.ts:1164).
  2. For Blueprint (MULTI_AGENT) mode: the test phone synced to Vapi. Creates a persistent Vapi squad copying the assistant's sub-agents. See sync-test-phone.ts.
  3. A Hamming agent registered against that test phone. Populates hamming_agent_id on the phone_numbers row. Without this, the Simulation Testing agent dropdown is empty even for teams with the feature flag.

See How-To: Register a Hamming Agent for the full prereq walkthrough.

Sub-pages ​

  • Tool Call Mocks, architecture explainer. State ownership, four-tier resolution, where mocks live (Avoca, not Hamming), terminology mapping.
  • How-To: Register a Hamming Agent, the one-time prereq each new team needs.
  • How-To: Test Transfer Decision, worked example with a paired contrast (in-hours-transfer-fires vs after-hours-take-message).
  • How-To: EAS Replica, stub. The replica enterprise + shops sandbox we'll spin up as a sandbox for prompt-variant testing. Opens after hamming-test-setup ships.

Status ​

  • First regression test (transfer decision pair): in progress. See .indusk/planning/hamming-test-setup/.
  • EAS replica enterprise (sandbox for prompt iteration): queued. Opens after hamming-test-setup. Stub at how-to-eas-replica.
  • A/B at scale (Surface 1) eval harness: queued. Depends on the replica being available.
  • Enterprise test fanout (catch Blueprint override drift): research in progress. See .indusk/planning/enterprise-test-fanout/. The "base edit doesn't propagate to per-shop overrides" failure mode that single-team tests cannot catch; addresses the gap.