Immersive AI Roleplay with Avatars
Avatar-based roleplays for the hardest conversations educators face, with on-the-fly lip-sync, dynamic facial expressions, multilingual speech, and AI voice evaluation.
Project Overview
A teacher training program for new educators in their first two years on the job. The instructional coaching team was burning out doing 1:1 roleplay sessions every cohort. They wanted a self-serve practice tool that handled the same scenarios coaches were running in 1:1s — parent conferences, classroom redirection, peer disagreement — without requiring a live coach in the loop.
The brief was a self-serve practice tool for the program's new-teacher cohort. It had to handle the scenarios coaches were running in 1:1s — parent conferences, redirection conversations, peer disagreement — without requiring a live coach in the loop.
A Roleplay Coach avatar introduces the persona and the goal. A second avatar plays the partner. From there, learners speak with their own voice, and the avatar speaks back with on-the-fly lip-sync and shifting facial expressions. The AI evaluates each spoken turn on tone, pacing, and word choice, and surfaces a personalized debrief with a one-click retry.
The avatar switches languages mid-conversation when the scenario calls for it. The voice rubric was co-authored with the coaching team so the AI scores what the coaches actually score. Coaches now use 1:1 time to review transcripts the learner has already practiced, not to act out scenarios from scratch.
What's Inside
8 roleplay scenarios
Across parent meetings, classroom redirection, peer disagreement, and admin debriefs.
2 consistent avatars
A Roleplay Coach for intro and debrief, and a roleplay partner for the live scenario.
Voice input with live scoring
Streaming Whisper transcription feeds the same evaluation as typed text.
Voice evaluation rubric
Tone, pacing, and word choice, scored against a coach-authored rubric.
Multilingual avatar
Switches languages mid-conversation; voice, lip-sync, and rubric all follow.
Personalized debrief
Strengths, focus areas, and a one-click retry button.
What the Client Provided
- Recorded 1:1 coaching sessions with consent (the source for the scenario library)
- The coaching team's “hardest conversations” list (Google Doc)
- Persona briefs for each of the 8 scenarios
- Existing tone-and-pacing rubric from the coaching team, in both languages
- Bios and avatar reference photos for the Coach and Partner avatars
Design and Build Process
Scenario Authoring
Worked with the coaching team to pull the scenarios educators actually dread. Each got a goal, a persona, an escalation map, and a clear definition of done.
Avatar Pipeline
Built the avatar pipeline on HeyGen with custom expression triggers and a streaming visemes layer for sub-frame lip-sync.
Voice Eval Rubric
Co-authored the rubric with coaches: what counts as a clear tone, a measured pace, and language that lands. Tuned the AI to the rubric, not a generic sentiment model.
Multilingual Layer
Added language-switching mid-conversation. The rubric was co-authored in both languages so scoring stays consistent across the switch.
Pilot and Retry Loop
Piloted with two cohorts of new teachers. Reworked the debrief screen until 80 percent of learners hit the “Try Again” button without prompting.
Tools and Stack
Deliverables
- Live roleplay tool (web app)
- 8 scenario packages (persona, goal, escalation map, rubric)
- Voice evaluation rubric (Google Doc, in both languages)
- Avatar expression and animation library
- Coach-facing transcript viewer for 1:1 review
- Embedded directly into the training program's website, with SCORM 2004 + xAPI packages for Cornerstone, Canvas LMS, and LearnDash
- Train-the-coach guide and a recorded walkthrough
Other Relevant Work

From LearnWorlds to Teachable, with AI Interactive Lessons
A national CTI training course migrated from LearnWorlds to Teachable, rebuilt with refreshed video and a flagship AI interactive lesson learners can chat with.
Cinematic AI Avatar Lesson Video
A polished AI avatar lesson video with two hosts, sound design, designed slides, micro-animations, and hand-crafted B-roll.

AI Rapid Fire — Sales Objection Drills with an AI Coach
A rapid fire AI tutor that drills sales reps on customer objections with type, click, and voice input, with sub-second feedback.





