Valitron.ai | Rami Akily

I learned to interview before I built an interviewer

High-volume hiring breaks at the first round. Recruiters can't give hundreds of applicants a real conversation, so most get filtered on keywords and are never heard. Valitron gives every applicant a voice interview of up to 40 minutes with Vali, our AI interviewer, then scores it and ranks the shortlist.

Before I planned a single part of the architecture, I read Lou Adler's Hire with Your Head cover to cover. I had to become an expert interviewer myself before I could build a product that interviews better than most humans. Adler's performance-based hiring became the backbone: interview against what the job actually has to deliver, probe for evidence, and score ten factors instead of gut feel.

We launched in six weeks with a team of three. Three problems have carried most of the engineering weight since.

Turn-taking

A voice interview lives or dies on who speaks when. Talk over a candidate and you lose their answer; wait too long and it feels like a bad phone line. I split the problem in two. The browser decides when the candidate may speak. OpenAI's semantic_vad model decides when they have finished.

One turn, drawn like a logic analyser. I stretched the time axis so the two 200 ms windows are readable; the code is simplified from the interview client.

How we got to 400 ms

Jun 2025I tuned voice detection, turned off mic auto-gain and delayed Vali's first greeting.
Jan 2026Vali kept interrupting itself: its own voice leaked back through the candidate's mic. Echo cancellation and longer silences fixed it, at the cost of speed.
Jan 2026A latency pass: required silence cut from 1,800 to 800 ms, the noise model's buffer halved.~2.5 s faster
Jan 2026I moved end-of-turn detection from silence timers to a semantic model that judges whether a thought is finished.
Feb 2026Muting now follows measured audio levels, not data-channel events, which arrived out of step with what the candidate heard.
Mar 2026Handoff guards cut from 1,000 and 700 ms to 200 ms each.1,700 → 400 ms

No barge-in

An interview isn't a chat. The transcript is what gets scored, so I chose one voice at a time, which keeps it clean and in order.

Trade-off It feels slightly less natural when a candidate wants to cut in.

Semantic end-of-turn over timers

Candidates pause to think. A silence timer punishes exactly the thoughtful answers we want to hear in full.

Trade-off Latency is less predictable, and it depends on the provider's model.

Evals that steer the Examiner

The Examiner is a separate reasoning model that reads the whole transcript and scores Adler's ten factors. Its first version sounded confident and was quietly wrong. So I stopped editing the prompt, built an eval suite, and made every prompt change since answer to it.

Real interview	Production v1	My cold read	v2.4
Interview A	78	78	78
Interview B, 30 minutes	n/a	73	70
Interview C	64	73	72
Interview D, graduate	63	67	68
Interview E	53	68	65
Interview F	27	Incomplete	Incomplete

I scored every real interview by hand, factor by factor, before the model saw it. Against my reads, mean absolute error fell from about 9.4 points in production v1 to about 1.2 in v2.4.

Assert order and policy, not exact scores

LLM judges drift a few points from run to run, so a test that expects exactly 72% fails on noise. The 22 golden cases check score bands, rankings, a five-run spread of 8 points or less, and banned wording, such as blaming candidates for topics nobody asked about.

Test the request that ships

The harness calls the same request builder as production. A prompt regression can't pass the eval and still reach a candidate.

28 characters flipped a verdict

Rewording two phrases about the same behaviour moved a candidate from Limited to Good Match in 3 of 3 runs. So v2.4 needs two weak factors, not one, before it rules someone out.

Trade-off I deliberately depart from Adler here, and I've documented why.

Don't punish the pipeline

The model flagged "inconsistencies" that were really speech-to-text spelling the company name three different ways. Proper-noun variants are now treated as transcription noise, not dishonesty.

Know when to stop prompting

Two rounds of instructions couldn't restore the model's eye for boilerplate answers. A stronger model did, unprompted.

What it still gets wrong: delivery sways the score more than substance. A coached rewrite of the same interview scored 89 against the original's 70. I track that as an open problem rather than hide it.

An in-flight black box

When an interview ends early, the candidate is gone and the recruiter wants to know why. Was it their Wi-Fi, a laptop going to sleep, a closed tab, or us? I scoped a flight recorder for every interview and directed a team member to build it. I later hardened it myself.

One interview, 18:20 until it stopped

Sample data

00:061phase: liveVali and the candidate are connected
12:102event-page-hiddenThe tab went to the background
13:523event-page-visibleBack after hiddenForSec: 102
18:204event-webrtc-connection-droppedinterviewStopped: true
18:21abandonment beaconphase: live, elapsedSec: 1101

What the drop event says, most likely first

Network connectivity was lost mid-interview
ICE keepalive packets timed out
A firewall or NAT dropped the UDP flow
OpenAI's server closed the peer connection
The tab was backgrounded and the OS throttled it

Heartbeats never missed a beat and the tab was visible when the call dropped, so it wasn't the laptop sleeping. That points to the network. The Examiner marks it as a technical failure, not a candidate who quit.

One session replayed from its events. The event names and cause lists are real; the session is illustrative.

Every event carries its hypotheses

Each of about 33 browser events ships with a ranked list of likely causes, a suggested fix and the code location. Whoever reads the logs starts from a diagnosis, not a stack trace.

Last seen alive

A heartbeat every 60 seconds, how long the page was hidden, and the interview's phase when it stopped. Together they separate a dropped network from a closed laptop lid.

A dropped call isn't the candidate's fault

Connection drops flow into the Examiner. A transcript that ends abruptly is marked as a technical failure instead of being scored as a candidate who quit.

Lost logs are the worst logs

Serverless functions froze before their logs were exported, so saved interviews lost their final lines. I flush after each response now, and a sanitiser keeps each session snapshot under a 256 KB cap.

Trade-off An extra flush on every response.

Other calls I made

The model has no clock

Asking Vali to "wrap up if time permits" produced a six-minute interview. The platform now sends the wrap-up at 37 minutes, and a test fails if that phrase ever returns to the prompt.

A hard stop at 40 minutes

Set from context-window maths, about 590 tokens a minute. It's a quality limit first and a connection guard second.

Unknown is not zero

A factor we couldn't assess now returns nothing instead of a zero. Before that, a candidate whose mic failed outscored a fluent but empty one, 60% to 40%.

Recruiter questions are data, not instructions

Custom questions are wrapped as data to block prompt injection, and loaded on the server so a candidate can never grade themselves.

Numbers

83%: Less time spent screening candidates2025
6 weeks: From concept to launch, with a team of three2025
~400 ms: To hand the turn back to the candidate, down from ~1,700 msMar 2026
1.2 pts: Mean score error against my own hand scoring, down from 9.4Aug 2026
3 pilots: Visa, Boston Consulting Group and PepsiCo2026