How do you test an AI receptionist before it goes live?
A call that sounds perfect and books nothing is the failure you are hunting, and the only place it shows up is the action log. Here are the nine test calls to run before a client build goes live, what to check after each one, and the registration clock that can add two weeks whatever your testing says.
Before an AI receptionist answers a real call for a real client, run a fixed set of test calls against a phone number nobody has advertised, and read the transcript and the action log after every one. Not how the call sounded. What it actually did. A call that sounded perfect and booked nothing is the failure you are hunting, and the only place it shows up is the log.
That is the short version. The rest is which calls to make, why the recording lies to you, and the one dependency that can hold a launch for two weeks no matter how clean your testing is.
Why listening to the call is not testing
OpenAI publishes an evaluation guide for its full-duplex voice stack, and the most useful line in it for an agency owner is this: "A natural-sounding confirmation does not prove the correct backend actions were completed" (OpenAI cookbook). The agent can say "great, you're booked for Thursday at 2" in a warm, convincing voice and book nothing.
So the guide splits evaluation into two axes, and you should steal both. One is audio and interaction quality: did it answer fast enough, did it talk over the caller, how long did it go silent. The other is reasoning and task quality: did it complete the task, did it call the right tools with the right arguments, did it hand off when it should have. Task completion is scored pass or fail. Tool accuracy compares the tools the agent executed against the ones it should have, and missing or extra calls drag the score down.
The guide stages testing in three steps it calls crawl, walk and run. Crawl is synthetic audio and one controlled request. Walk is real recorded audio, still single-turn, which is where accents, background noise and bad phone lines show up. Run is a simulated caller holding a multi-turn conversation, interruptions included. You are not going to build an eval harness for a four-truck HVAC company. You need the shape of one: clean easy request first, then realistic audio, then a messy conversation.
One more line is worth taping to your monitor. "Voice models are nondeterministic, so representative scenarios should be run multiple times." One clean booking call is not a pass. It is one sample.
The nine calls to make
HighLevel's support documentation for building voice agents lists the scenarios it tells you to test, and the list is good because it is short enough to actually run (HighLevel docs). Nine calls:
- A common caller request. The thing most callers want.
- A question the knowledge base should answer. Hours, address, whether they take a particular insurance.
- A question the agent should not answer. Price a job it cannot price, give medical or legal advice, commit to an arrival window.
- An incomplete or unclear request. Mumble. Trail off. Say "yeah I need someone to come out, uh, for the thing."
- Each configured action, one at a time. Booking, SMS, contact update, workflow trigger.
- A transfer or escalation request. "Can I just talk to a person."
- A period of silence. Say nothing for twenty seconds and see what it does.
- An interruption while the agent is speaking. Cut it off mid-sentence.
- The closing sequence. Let the call end the way it is meant to end.
Run each twice, for the nondeterminism reason above. Eighteen calls at three minutes each is about an hour of work. That is the whole gate.
Two of the nine fail most often in the field. Number three, the refusal, is where a generic prompt gets a client in trouble by quoting a price it invented. Number six, the transfer, is where the configuration is usually right and the routing is usually wrong. HighLevel is blunt about a trap there: "Call Transfer does not run during a Web Call test." A browser test will happily show you a transfer that never touched the phone network. Test transfers on a real phone call or do not claim you tested them.
Test on a number nobody has advertised
ServiceTitan tells its customers to go live by forwarding one tracking number first, and specifically to use a number from the reserve pool that has not been advertised yet, or to get a fresh one (ServiceTitan help center). Only after that number behaves do you touch the rest of the call routing.
Do the same. At the carrier level a test call is indistinguishable from a real one, and you do not want to find out the agent was answering the main line during the hour you spent breaking it on purpose.
ServiceTitan also says the quiet part out loud, twice on the same page: "Any test calls you place with the voice agent will be included in your bill." Same here. The 100 voice minutes on the $49 plan are a sales allowance for your demos, not a testing budget for a client's production build. Voice on a client build is metered per minute at the wholesale rate agreed on your pricing call, and the top tier on the platform page lists $0.35 a minute as the guide. Eighteen calls at three minutes is under an hour. Budget roughly twenty dollars of wholesale cost into every launch.
Read the log, not your memory of the call
After each test call, open the call record and check four things: transcript, recording, summary, action results. The action results are the ones people skip and the ones that matter. Did the booking land on the calendar. Did the SMS send. Did the contact field update. The agent's confirmation is not evidence. The calendar entry is evidence.
When you change something, change one thing and retest. HighLevel's docs say the same, and the reason is obvious the first time you adjust the prompt, the interruption sensitivity and the response speed in one pass and cannot tell which one fixed it or which one broke the transfer.
The texting side runs on somebody else's clock
Here is the dependency that catches people who did everything else right. If the build sends texts from a regular ten-digit US number, that number needs A2P 10DLC registration: a brand, then a campaign. Brand approval through The Campaign Registry is usually quick, but a brand kicked to manual review takes seven business days or more. The campaign is the slow one. Twilio's documentation currently carries a banner saying campaign reviews are taking 10 to 15 days, and the formal text on the same page says the manual vetting process "can take between two to three weeks to complete" (Twilio docs).
No amount of testing shortens that. You start registration during intake, not during testing, which is part of why the EIN sits at the top of the intake list. Wait until the voice side passes to think about messaging and you have added two weeks to a delivery you already promised.
Twilio's registration quickstart is worth one read before your first campaign. It says most application failures happen at that step, and why: one-word campaign descriptions, sample messages with no opt-out language, and opt-in processes nobody can verify on a real website (Twilio quickstart).
What testing does not catch
Four honest ones.
Testing proves the agent handles the calls you thought of. It says nothing about the calls you did not. The real test is the first two weeks of live transcripts: read them, find the failure you never imagined, add it to the list. OpenAI's guide recommends that loop for its own harness, and it works at the scale of one roofing company.
You cannot test your way out of bad intake. If the client gave you the wrong price band or forgot a service line, the agent will deliver that wrong answer flawlessly and every test will pass. Intake quality sits upstream of testing quality, which is the whole argument of what you need from a client before the build starts.
Latency is not yours to fix. Twilio's engineering blog on evaluating voice agents notes that most people believe a normal human conversational response lands somewhere between 300 and 1,200 milliseconds, which is the bar an agent gets judged against whether anyone says so or not (Twilio). You can hear a sluggish agent in testing. You cannot repair it from the agency side. What you can do is hear it before the client does and say something first.
And there is no certified pass. Nobody in this category can hand you a guaranteed-correct AI receptionist. What a test round buys is that the obvious failures are gone before a paying customer finds them.
Run the last round with the client listening
The final test call should not be yours. Get the owner on the phone, have them call the agent, and let them ask the three questions they are worried about. They always have three.
This makes go-live their decision instead of your announcement, and it surfaces the business rule nobody wrote down. Half the time the owner hears a perfectly correct answer and says "oh, we don't actually do that anymore." Five-minute fix before launch. Angry phone call after it.
Then flip the routing the boring way: overflow and after-hours first, main line later or never. We covered forwarding versus porting in what happens after your client signs, and a good test round does not change that answer.
FAQ
How many test calls before go-live? Eighteen is a reasonable floor: the nine scenarios above, run twice. Coverage matters more than the count. If you have not tested a refusal, a transfer and an interruption, you have not tested.
Do test calls cost money? Yes. Voice on a client build is metered per minute at the wholesale rate agreed on your 30-minute pricing call. The 100 voice minutes on the $49 platform plan are for your demos, not for a client's production build. Budget test minutes into the delivery.
Can I test everything in a browser instead of calling? No. Browser testing is fine for prompt behavior and knowledge base answers, and it is faster to iterate with. Transfers and real telephony conditions need an actual phone call. HighLevel's documentation states plainly that call transfer does not run during a web call test, and the caution applies anywhere.
What if the client wants to test it themselves for a week? Let them, on the unadvertised number, with a deadline. A week of owner testing on a parked number is cheap and builds confidence. What you cannot allow is open-ended testing with no go-live date. That is usually a client going cold.
Does testing cover the text-back side? Partly. You can confirm the message fires and reads correctly, but delivery at scale depends on A2P 10DLC registration, and campaign review is running one to three weeks. Start it during intake.
Do this on your next build
Write the nine scenarios into a one-page checklist with a column for the transcript check and a column for the action check, and do not mark a build ready until every row has both. Reuse it on the next client in the same trade and it gets faster each time.
If you want to hear what the finished conversation should sound like, the live demo on our homepage is a real agent you can interrupt and push off-script. Do that to it, then do it to your client's build before anyone else gets the chance. An agency workspace is $49 a month.