How to evaluate a B2B voice agent: call quality, interruptions, verified outcomes, handover, consent review and the full cost of a pilot.
A voice demo is not a benchmark for a sales workflow. Connection quality, caller behaviour, tools and handover all affect the experience. This guide sets out a practical test method rather than claiming provider rankings, production call measurements or guaranteed conversion results. Choose acceptance criteria with the people who will operate and review the service.
Define the voice workflow
A hypothetical first workflow is an inbound enquiry call: explain that the caller is speaking with an AI assistant, collect agreed details and offer an approved booking or staff handover. Specify which questions it may answer and which commitments it cannot make. Outbound use needs a separate review of the audience, permissions and applicable calling rules.
Four measures to define before testing
Response delay: measure from the caller finishing a turn to audible agent speech, including long pauses caused by tools. Interruption handling: check whether it stops, captures the correction and continues appropriately. Information quality: verify names, numbers and answers against the source. Task outcome: confirm the booking, record update or handover in the receiving system. Agree thresholds for the particular workflow and report failed calls as well as successful ones.
How to compare hosted platforms
Give shortlisted platforms the same test scenarios, source information and tool actions. Test quiet and noisy calls, relevant accents, changed details and requests for a person. Record configuration, region and sample size, and compare distributions rather than one attractive average. Check account permissions, recordings, export options, support and total charges before choosing. A provider's demonstration or published claim is not independent evidence of performance on your calls.
How to assess a custom stack
A custom design can combine speech recognition, a model, speech output and telephony, or use a realtime speech model. It also adds responsibility for transport, interruptions, tool execution and support. For example, LiveKit's turn-handling documentation describes configurable turn detection and interruption behaviour. Those controls need testing in the actual setup; a custom stack is not inherently faster, cheaper or more private.
Calculate cost per verified outcome
For a hypothetical inbound pilot, include telephony, platform and model charges, numbers, recordings, integrations, review and maintenance. Measure the cost of a correctly completed task, including failed and repeated calls. Compare with the existing service at a comparable quality level. Do not value an unconfirmed meeting as revenue, count every call as a completed enquiry or assume that released staff capacity becomes a cash saving.
Failure cases to include
Test unfamiliar names, relevant accents, background noise and callers who correct a date or amount. Require confirmation of important details before a booking or record change. Check unavailable calendars, tool timeouts, callers who become frustrated and failed transfers. Provide an accurate fallback, such as an agreed callback request, rather than claiming that a human is available when no transfer has succeeded.
Disclosure, permission and data handling
Have qualified advisers review the calling plan, including jurisdictions, consent, recording, opt-outs and retention. The FCC's AI-voice ruling confirms that AI-generated voices fall within relevant TCPA artificial-voice restrictions; it is not a blanket permission for B2B calls. As a design principle, identify the assistant clearly and offer a human route. The exact legal requirements depend on the call and applicable rules.
Choose the architecture on requirements
Compare hosted and custom approaches against required integrations, data handling, reliability, support and operational capacity. Verify contractual features and technical controls rather than assuming a volume threshold dictates the answer. Start with client-owned accounts and a bounded pilot; agree who owns numbers, access, source content and the continuing support process.
What evidence should a pilot produce?
Keep a versioned scenario set, reviewed call outcomes and evidence of tool results. Document missed details, incorrect answers, delays, opt-outs and handover failures. Record the limits of the sample and what remains untested. Expand only after the business approves the findings and operating responsibilities, without treating a forecast or small demo as a production benchmark.