VR training with AI-driven coaching
How XRDA3 combined immersive VR and a real-time LLM coach to replace 40 hours of classroom safety training with 6 hours of personalised VR — across 14 manufacturing plants and 3,400 workers.
XR + AI
The challenge
Reduce on-the-job injury rate while standardising safety training across 14 plants — without taking workers off the line for full-day classroom sessions.
Classroom-based safety training had three structural problems. Knowledge retention was poor — testing two weeks after each session showed only 38% recall on critical procedures. Trainer availability was inconsistent across geographies. And the cost was real: each plant lost roughly 8,000 productive hours annually to mandatory training rotations.
The client had piloted off-the-shelf VR safety modules from two vendors. Both shipped impressive demos and produced disappointing outcomes. Workers completed scenarios but couldn't recall key decisions a week later. The vendor playbooks didn't customise scenarios to each plant's actual machinery, so the training felt abstract.
The approach
Combine immersive VR scenarios — modelled on each plant's actual equipment — with a real-time AI coach that observes the trainee, asks adaptive questions, and tailors subsequent scenarios to expose individual weaknesses.
The breakthrough was treating the LLM not as a chat interface, but as a co-trainer running alongside the VR session. As the trainee navigates a scenario, the coach watches their gaze, hand position, action sequence, and elapsed time — and surfaces voice prompts that feel like a senior worker standing next to them.
If the trainee skips a step, the coach asks "what should you have checked before opening that valve?" If they hesitate, the coach offers a focused hint. After every scenario, a 90-second debrief covers what they did well and what scenarios they should run next — chosen by the model from a library of 84 plant-specific situations.
What we built
- Unity-based VR runtime on Meta Quest 3 with OpenXR, deployed via MDM to 280 headsets across 14 plants
- Plant-specific scenario library of 84 procedures — modelled from photogrammetry of real equipment, validated by plant safety officers
- Real-time LLM coach using a fine-tuned small model with safety-domain prompting, running in cloud with sub-400ms response time
- Trainer dashboard showing every worker's progression, scenario completion, weak areas, and aggregate plant-level safety insights
- Telemetry pipeline capturing every gaze, action, and decision — feeding back into the AI model's adaptive scenario selection
- Offline mode for rural plants with intermittent connectivity — coach continues working with cached models, syncs telemetry when online
Tech stack
Snowflake · dbt · Looker · embedded React dashboardOutcomes
Honest reflections
This was a hard programme. The first scenario library shipped with audio that didn't match Indian and GCC dialects — we re-recorded the entire prompt set in three months. Photogrammetry of plant equipment was slower than expected because shutdown windows were limited. And the LLM coach hallucinated procedure details twice in the first three months — which is why we added retrieval-grounding to the safety knowledge base before scaling beyond two pilot plants.
The lesson we keep coming back to: in regulated industries, "AI coach" means "AI plus a human-validated knowledge base." Without the second part, you're just shipping a confident-sounding hallucination engine. With it, you've replaced an experienced trainer with something better.
Capabilities used
Have a workforce-training problem worth solving?
If your training is too long, too generic, or too forgettable — we'd like to hear about it.