AMIE is tested by Google for clinical video consultations.

Google‘s medical AI system, AMIE (Video), engaged in video consultations with professional patient actors, receiving clinical evaluator ratings comparable to those of primary care physicians across several core metrics. Fifteen trained actors simulated conditions across various medical areas such as cardiopulmonary, abdominal, HEENT, neurological or psychiatric, and musculoskeletal presentations. Google notes that further studies involving real patients are necessary before drawing conclusions about clinical applications. AMIE utilizes an asynchronous multi-agent architecture to allocate tasks like dialogue, clinical reasoning, and perception among different agents. According to Google, a single agent cannot effectively manage natural conversational speeds along with detailed reasoning and continuous audio-visual processing. The system comprises three agents: the talker agent manages spoken interactions while maintaining conversational flow with input from the planner agent. The planner agent updates differential diagnoses, management plans, and identifies missing information throughout the consultation. Meanwhile, the perception agent continuously analyzes video and audio streams, detecting non-verbal cues and physical findings to contextualize them within the clinical conversation. Addressing latency is key, as deep clinical reasoning can take time, impacting rapport if pauses are prolonged. Google’s architecture allows for swift patient-facing responses by decoupling dialogue from the background processes of reasoning and perception. Automated evaluations indicate that each agent enhances clinical measures such as history-taking, clinical reasoning, and treatment recommendations, alongside patient-centered communication and response times. In a multi-arm randomized study, AMIE’s real–time video consultations were compared with a text–only version and assessments by ten board-certified primary care physicians using the same video setup. An independent panel of 20 experienced primary care physicians reviewed all consultations using established clinical criteria. The study encompassed five body systems, each scenario adhering to a standardized consultation format with trained actors. Evaluators found AMIE comparable to the physicians’ group in terms of history–taking thoroughness, diagnostic accuracy, management appropriateness, and communication quality. AMIE (Video) performed as well as or better than AMIE (Text) in these areas. The video system was rated higher for eliciting physical signs and guiding actors through virtual examination maneuvers than either physicians or the text-only AMIE version. Case-specific perception and examination scores showed consistency with these findings. Patient actors expressed a preference for the synchronous video interface over text chat, rating it easier to use and more effective for communicating health concerns. AMIE was favorably rated for empathy, rapport, and confidence in care compared to both study alternatives.

Google developed an automated evaluation suite to advance its video system even before conducting human studies. This framework is informed by a taxonomy of telehealth competencies from medical literature, focusing on visual cues, auditory signals, and physical examination techniques. Single-turn assessments were designed to test specific perception and reasoning tasks. For instance, Google highlighted examples such as anatomical laterality and identifying signs of respiratory distress. In contrast, multi-turn simulated audio consultations evaluated the system’s conversational abilities over extended interactions. In some of these multi-turn simulations, visual input was incorporated through text descriptions. For example, in a Parkinson‘s scenario, an AI patient simulator might describe a patient holding up a paper to the camera with cramped, tiny handwriting. This setup allowed Google to test dialogue behavior with visual data, although it didn’t simulate an end-to-end live video feed. The automated suite enabled quick modifications to the system design and identified capability gaps before transitioning to actor-based studies. The following OSCE evaluation employed a synchronous video consultation interface, though patient scenarios were still pre-planned. The division between these methods should inform procurement and governance discussions. Automated assessments can evaluate predefined perceptual tasks on a large scale, while simulated video consultations can gauge interaction quality under controlled conditions. However, neither method effectively measures performance with patients whose symptoms, behaviors, connectivity, environment, and medical history fall outside of prepared cases. Current production evidence is limited to text-based work, and Google acknowledges several limitations in the AMIE research. Professional actors cannot fully replicate the variability found in real patient interactions, and scenarios were also designed to exclude cases where audio-visual perception might provide more diagnostic value. Targeted automated evaluations revealed occasional perception and reasoning errors, and Google also noted sporadic technical issues that could disrupt conversational naturalness. Project Astra remains a prototype, with system-level technical concerns extending beyond just this medical application. Google indicates that research with real patients will be the next step. The company has initiated related efforts in clinical settings using the text-based AMIE. A feasibility study with Beth Israel Deaconess Medical Center yielded initial evidence on safety and utility in clinical practice. Moreover, a nationwide randomized study with Included Health is currently evaluating AI within real-world virtual care. Google‘s study provides controlled evidence on video consultation behavior, physical examination guidance, and clinician scoring. However, it has yet to demonstrate that AMIE can safely diagnose or manage actual patients in a production setting.