Architecting a Real-Time Voice AI Technical Interviewer with WebSockets & Monaco Editor

Traditional technical screening tests force candidates through rigid LeetCode forms with zero human interaction. We engineered AI Technical Interviewer to simulate a real senior engineering interview: the AI speaks with natural vocal cadence, observes candidate keystrokes in real time, offers gentle proactive guidance when candidates pause, and generates objective evaluation rubrics.

1. The Vision: Conversational Code Evaluations

A genuine technical interview is a dialogue, not a silent test. The platform tracks candidate problem-solving reasoning, algorithmic trade-offs, code modularity, and error recovery across 9 programming ecosystems including JavaScript, TypeScript, Python, Java, C++, and SQL.

2. Sub-300ms Voice Turn-Taking with Zero-TTS

To prevent awkward conversation pauses, we integrated Shunya Labs Zero-TTS neural voice synthesis. Audio buffers stream directly over binary WebSocket frames, enabling the AI to begin speaking within 280ms of candidate voice cessation.

3. Streaming Monaco Editor Deltas over WebSockets

Rather than periodically polling full text snapshots, we hook into the Monaco Editor's onDidChangeModelContent listener to stream lightweight diff patches. A debounce timer detects candidate hesitation (>35s with empty buffers) and triggers contextual spoken encouragement.

4. Deterministic Finite State Machine Architecture

To prevent LLM hallucination and out-of-order interview progression, the session state is managed by a strict Finite State Machine (FSM):

type InterviewState = 
  | 'GREETING'
  | 'QUESTION_INTRODUCTION'
  | 'ACTIVE_CODING_AND_OBSERVING'
  | 'HINT_PROMPT'
  | 'CODE_EXECUTION_AND_TESTS'
  | 'POST_INTERVIEW_DEBRIEF'
  | 'SCORECARD_GENERATION';

function transitionState(current: InterviewState, action: string): InterviewState {
  switch (current) {
    case 'ACTIVE_CODING_AND_OBSERVING':
      if (action === 'RUN_TESTS') return 'CODE_EXECUTION_AND_TESTS';
      if (action === 'STALL_TIMEOUT') return 'HINT_PROMPT';
      return current;
    default:
      return current;
  }
}

5. Multi-Model Fallback Chains Across OpenRouter

To eliminate rate-limit aborts during live interviews, we implemented an automated failover chain using OpenRouter: Claude 3.5 Sonnet → GPT-4o → Gemini 2.5 Pro → DeepSeek R1. If a primary provider returns a 429 or 503 status, the request transparently reroutes in under 120ms.

6. Automated Hiring Scorecards & Rubrics

Upon completion, the candidate's code is compiled inside an isolated sandbox. The system produces a structured radar scorecard evaluating Code Correctness, Time Complexity, Clean Code Architecture, and Verbal Communication.