Author: admin

  • DocsAlot MCP Connector

    Maintain your help-center by asking Claude, Cursor or Codex

    Discussion
    |
    Link

  • Introducing Quine: An AI research system designed for the complexity of biology

    Introducing Quine: An AI research system designed for the complexity of biology

    Diagram of Project Quine’s closed-loop approach to biological research. A scientist asks questions that are passed to Quine for in silico computation. Quine combines a world model of biology, spanning genomics, proteins, chemistry, cell state, and bioimaging, with a harness for orchestration and reasoning, supported by knowledge and tools. Quine generates proposals that are tested through real-world wet-lab experiments. Experimental measurements return to the scientist and inform the next question, creating an iterative loop between computational modeling and real-world experimentation.

    At a glance

    • Quine (opens in new tab) is a research effort to create a multimodal world model of biology and an interactive harness connecting models, scientific tools, literature, and researchers. 
    • In collaboration with researchers at the Broad Institute of Harvard and MIT, we have used this system to prioritize compounds predicted to drive therapeutic tumor-state shifts and validated several top-ranked candidates across multiple wet-lab assays. 
    • The Quine Fellows program (opens in new tab) will give a cohort of scientists access to the system and an opportunity to accelerate their own research and provide scientific feedback.  
    • Quine is experimental research technology intended only for research, not clinical or medical use, and its outputs may be incomplete or inaccurate and require review by qualified researchers and appropriate scientific and experimental validation. As the technology matures, we expect to expand access through products like Microsoft Discovery (opens in new tab).

    For more than two decades, Microsoft Research has worked at the intersection of computation and biology. Our research has spanned immunology, virology, genomics, biomedical imaging, cell biology, and protein engineering. That work has produced foundational methods, new science, and technology that reached the clinic, from rare and infectious disease diagnosis to cancer biomarker detection.

    Across that work, one lesson has become increasingly clear: biology does not divide itself into the neat boundaries our models and tools often do. Genes influence proteins; proteins interact within cells; cells organize into tissues; and experiments continually reshape what scientists know and what they choose to ask next. Making progress on the hardest biological questions therefore requires more than increasingly capable models of individual datasets or tasks. It requires systems that can connect knowledge across scale and modalities, reason about experiments and evidence, and participate in the iterative process through which science advances. 

    Today, Microsoft Research is introducing Quine (opens in new tab), a research effort designed to work across those boundaries, reflecting our long-term vision for a discovery system that evolves through scientific use. Quine brings together a world model of biology with a harness that connects scientific tools, literature, the wet lab, and the researchers using them.

    The limits of experimentation

    Even as experimental techniques have improved and wet-lab throughput has increased, biology remains fundamentally constrained by time and complexity. Nature cannot be rushed, nor can it be derived from first principles. Experiments are slow, iteration cycles are long, and many of the most important questions involve interactions, combinatorial design spaces, and downstream effects that are simply too large to explore experimentally alone.

    At the same time, advances in large-scale machine learning, particularly the emergence of general-purpose foundation models and reasoning models that can iteratively work through problems, suggest a new possibility. These systems are beginning to demonstrate capabilities beyond pattern recognition: integrating information across domains, reasoning over abstractions, and supporting iterative problem-solving. Just as importantly, many of the techniques developed for human language have proven remarkably adaptable to aspects of biology, enabling models to learn representations of biological systems across diverse data types and scales.

    This raised a provocative question for us: what would it take to build a world model for biology? By world model, we mean a system that can represent the state of a biological system, predict how that state will evolve in response to interventions, and reason over the consequences of those interventions multiple steps into the future. Such a system would not replace experimentation. Rather, it would allow us to use computation to explore, propose, rank, and prioritize potential paths forward before committing scarce laboratory resources. Our north star is a future in which scientists, models, and experiments operate in a continuously accelerating loop.  

    What we’ve built

    Quine is our first step toward that vision. It includes both a world model of biology and a harness that connects the model to orchestration and reasoning models, scientific tools, the literature, and the teams of scientists using them.  

    Figure 1: Quine’s two frontier components — a world model of biology and an interactive harness — sit inside a loop that begins and ends with the scientist. A question becomes a set of proposals, proposals become designs worth testing, and the experiment returns measurements that sharpen both the scientist’s next question and the model itself. 

    Core to this effort is the world model, which learns shared representations across biological modalities and scales, including sequence, structure, function, cellular state, and imaging data. By training across these modalities jointly, Quine can use evidence from one modality to inform predictions in another, capturing relationships that would otherwise remain siloed or lost when separate single-domain specialist models are orchestrated by the system. This multimodal design reflects the reality of biology: understanding any one level requires context from the others. Rather than diluting performance, we find that learning across these connected representations strengthens it, enabling the model to generalize more effectively and to support a broader range of biological reasoning tasks.

     Figure 2: Biology is multiscale and multimodal. (A) Biological evidence stacks from molecules to cells to tissues to patients, with scientific language running underneath all of it. (B) The modalities Project Quine trains on — genomics, proteins, chemistry, RNA and cell state, bioimaging — each observe a different span of that range, and they overlap. Reading down is the causal path from genotype to phenotype; reading across is physical scale. Because the model learns these representations jointly rather than orchestrating separate specialists, evidence at one level can inform predictions at another. 

    By integrating this model in a unified system that facilitates computational experimentation and learns from iteration, researchers can generate predictions across modalities and reason about the consequences of biological interventions before they are tested in the lab. Critically, the world model doesn’t need to be perfect—indeed, it will never perfectly model biology—it simply needs to usefully inform experimental design. 

    What this looks like in cancer biology

    A concrete example comes from our work on pancreatic ductal adenocarcinoma (PDAC), the most common form of pancreatic cancer and one of the most challenging to treat. In collaboration with researchers at the Broad Institute of MIT and Harvard, we have spent years developing and applying patient-derived ex vivo models to investigate a longstanding hypothesis: that tumor behavior and drug response depend not only on genetics, but also on transcriptional cell state. In PDAC, tumor cells can occupy different cellular states associated with how they respond to treatment.  

    With Quine, we have now put that hypothesis into practice, exploring whether non-genetic features of cancer, particularly cellular state, can serve as actionable therapeutic targets. We used Quine to predict and prioritize thousands of compounds based on their potential to shift tumor cells between therapeutically relevant states. In wet-lab studies focused on the transition from classical to basal cell states, Quine’s highest-ranked compounds produced the largest intended shifts across experimental assays. The entire process—from rapidly narrowing the compound search space to prioritizing a handful of promising candidates to be validated in the lab—took just one weekend, potentially saving months of experimental work and significant research costs. Notably, some of the strongest effects came from compounds with unexpected mechanisms of action, offering early evidence that AI can uncover new opportunities for drug repurposing and discovery. 

    While the reverse state transition proved more difficult (from basal to classical cell states), Quine predicted that available compounds would have this weaker effect. More notably, the experiments revealed something we did not fully anticipate: Quine predicted that several compounds would consistently move cells toward a distinct third phenotype—an observation borne out in the lab—suggesting that the pancreatic cancer cell-state landscape is richer than a simple classical-basal axis. The experiments not only tested the model’s hypotheses but also generated new ones. 

    Figure 3: Quine-guided exploration of pancreatic cancer cell states. In pancreatic ductaladenocarcinoma (PDAC) cell lines, top-ranked compounds produced the largest classical-to-basal transcriptional shifts as validated by wet lab experiments. Weaker reverse shifts and unexpected movement toward an additional phenotype were also consistent with Quine’s predictions.

    Our continued work here will use newly integrated RNA datasets and tasks to represent this richer landscape, strengthen state-transition predictions, and provide calibrated confidence estimates that help scientists prioritize the most promising hypotheses for wet-lab testing. And this is exactly the feedback loop Quine was built to support. Each round of experimentation improves our understanding of biology, and each new insight can become training signal for the next generation of the system.  

    Learning through discovery – an invitation

    We created Quine to improve how we do science ourselves. As the system evolved, we embedded it in our ongoing scientific programs. Every research question, experimental result, and unexpected finding became an opportunity to refine the models, improve the workflows, and better understand where AI could meaningfully accelerate our research. Over time, the system became central to how we approached discovery across multiple domains, including in cancer biology, protein engineering, genomics, and bioimaging.

    We’d like to continue building on that foundation by introducing Quine at this early stage. Quine grew out of our own research, but we don’t want its continued development to happen in isolation. Many of the questions that will determine where it goes next are not ones we can answer alone; they require the expertise, creativity, and perspectives of the broader scientific community. For this reason, we’re opening the Quine Fellows program to put the system directly into the hands of scientists working at the frontier of biology and medicine.

    Building responsibly will always come first. We believe progress in AI and biology must go hand in hand with safety, security, and responsible stewardship. We are taking a deliberate, phased approach to Quine’s development and access, with initial availability limited to the Quine Fellows program and select research collaborations. Ongoing internal review and built-in safeguards will help us maintain appropriate oversight as the system evolves. As the technology matures, we expect to expand access through products like Microsoft Discovery (opens in new tab).

    It’s tempting to frame progress in AI for biology around benchmarks and leaderboards. Those matter, but the real test of Quine is what happens when it encounters science as it’s actually practiced. Can it be useful when evidence is incomplete? When facing questions genuinely never asked before? Can it help make the path from questions to discovery shorter?

    Ultimately that’s the standard we care about. The ambition is for Quine to recede into the background of scientific practice, while the science moves faster in the foreground. The protagonists are not the model or the platform. They are the scientists, the experiments, and the discoveries that follow. 

  • Pegasus 1.6 by TwelveLabs

    Transforms egocentric video into robot training data

    Discussion
    |
    Link

  • Foglio

    Simple, private docs with smart AI templates, zero bloat.

    Discussion
    |
    Link

  • JOGAK

    Make everyday vlogs with less editing

    Discussion
    |
    Link

  • One year in: How Microsoft Research Asia – Singapore is advancing research, partnership and talent for real-world impact

    One year in: How Microsoft Research Asia – Singapore is advancing research, partnership and talent for real-world impact

    Illustrated Singapore skyline at dusk featuring Marina Bay Sands and Gardens by the Bay, with colorful fireworks and confetti overhead. Blue and orange graphic waves frame the scene, with a Merlion outline in the lower left and open space on the left for text.

    On July 24, 2025, Microsoft Research Asia – Singapore (MSRA – Singapore) opened its doors as Microsoft’s first research lab in Southeast Asia. The launch built on more than two decades of collaboration between Microsoft Research Asia and Singapore’s universities, research institutes, and government agencies. It marked a shift to a closer, more integrated relationship aimed at supporting Singapore’s innovation ecosystem and advancing the development and beneficial use of AI.

    “When we established Microsoft Research Asia — Singapore one year ago, our vision was not simply to open another research lab. It was to build a bridge connecting Microsoft’s frontier AI research with Singapore’s world-class universities, forward-thinking government agencies, and dynamic industry ecosystem. One year in, what gives me the greatest confidence is not just the research we have produced, but the depth of partnership we have built. That is the foundation on which lasting impact is built.”

    — Dr. Lidong Zhou, corporate vice president at Microsoft and managing director of Microsoft Research Asia

    Established in 1998, MSRA has a presence in Beijing, Shanghai, Vancouver, and Tokyo, along with Singapore. The Singapore lab strengthens the collaboration with local partners and extends the scientific and societal impact of Microsoft’s global research community. 

    Over the past year, the Singapore lab has grown its team, deepened partnerships, and explored how frontier AI research can create real-world value. Guided by its mission to advance frontier AI for humanity, its work spans four pillars: next-generation AI models and agentic systems, domain-specific AI for real-world impact, AI-native research practices, and ecosystem and talent development.

    From frontier research to real-world impact – and back

    Research has its greatest impact when it moves beyond the laboratory into practice: in health and medicine, enterprises, and the daily operations of organizations solving real problems. MSRA – Singapore aims to do just that through a two-way exchange: partners bring domain expertise, real-world data, and operational challenges, while the lab contributes frontier research and engineering capabilities. Together, they translate fundamental advances into practical solutions, while surfacing new questions that feed back into research in a kind of research-to-impact flywheel. Advances in AI foundations enable more capable models and agentic systems that can reason, plan, use tools, and collaborate with people. These capabilities create new opportunities for AI-native innovation. In turn, real-world applications reveal limitations and generate new questions, feeding insights back into foundational research and sustaining a continuous cycle of discovery, application, and learning.

    Healthcare offers an early example of the lab’s research-to-impact model. Research on multimodal healthcare AI and self-evolving diagnostic agents is deployed and extended through partnerships across Singapore’s healthcare ecosystem. The lab is collaborating on multimodal and agentic AI approaches to support clinical decision-making, unlock insights from complex medical data, and advance research in disease diagnosis, risk assessment, and personalized care, with the longer-term goal of enabling meaningful real-world use.

    Beyond healthcare, the lab is working with organizations across financial services, education, and technology to explore how frontier AI can address sector-specific needs and operational challenges.

    “Over the past year, the lab has become an integral part of both Microsoft’s local ecosystem and Singapore’s broader innovation community. Its progress demonstrates the value of establishing a sustained research presence in Singapore.”

    — Wee Luen Chia, managing director, Microsoft Singapore

    Singapore has made AI a national priority through its National AI Strategy 2.0 (opens in new tab), which focuses on deploying industry-transforming AI, advancing frontier research in AI foundations, and developing responsible, socially beneficial applications. These priorities span areas including healthcare, education, finance, and Smart Nation (opens in new tab) services. MSRA – Singapore’s research portfolio aligns with these priorities, with a particular focus on building AI systems that are reliable and trustworthy across Southeast Asian contexts. 

    The Singapore Economic Development Board (EDB) has been a foundational partner. It supported the lab’s establishment and continues to support joint PhD training through the Industrial Postgraduate Programme (IPP). The relationship has since grown into strategic collaboration. In February 2026, EDB co-hosted an Executive Roundtable on AI in Logistics and Transportation, bringing together more than ten regional industry leaders to explore sectoral challenges.

    “Singapore is building deep research capabilities to push the boundaries of AI and other frontier technologies. Microsoft Research Asia – Singapore is an important partner in this effort. Since its launch in 2025, the lab has brought Microsoft’s global research expertise closer to our ecosystem, partnering academia, industry, and government to advance frontier AI research and translate it into real-world applications in areas such as healthcare. MSRA is also developing the next generation of AI research talent who can create meaningful impact for our economy and society.”

    — Low Jin Yong, senior vice president and co-head, Digital Industry Singapore

    Government engagement extends beyond EDB. In July 2025, IMDA (opens in new tab) co-hosted Microsoft Research Asia Singapore Day (opens in new tab) — Advancing Industrial AI, a full-day event spanning healthcare AI, interactive world simulators, AI agents, and societal AI. MSRA – Singapore has also engaged with the Ministry of Trade and Industry and the Ministry of Health. 

    In addition to its deep ties with government agencies, MSRA has built strong connections with Singapore’s academic community, supporting more than 75 research projects with Singapore institutions since 2004. The opening of the Singapore lab has brought new engagements, with nine new projects in its first year with National University of Singapore (NUS) and Nanyang Technological University, Singapore (NTU) across healthcare AI, robotics, AI systems, and multi-agent systems. 

    A landmark example is the five-year Framework Research Agreement (opens in new tab) with NUS, announced in 2025, which provides a platform for joint AI research, talent development, and academic exchange. It defines shared research priorities in healthcare, societal AI, spatial intelligence, and data-intensive computing alongside an annual NUS–MSRA Academic Symposium that brings researchers from both organizations together to shape the collaboration agenda. 

    “This partnership creates new opportunities for frontier research, talent development, and knowledge exchange, strengthening connections across academia, industry, and the broader innovation ecosystem.”

    — Professor Liu Bin, deputy president (Research and Technology), National University of Singapore

    The collaboration is already producing concrete research across several frontier areas. NUS Assistant Professor Mike Shou and MSRA’s Xinxing Xu are developing unified multimodal AI for healthcare. NUS assistant professor Yang You and MSRA distinguished scientist Furu Wei are studying systems foundations for diffusion-based language models. NUS assistant professor Jialin Li is working with MSRA on rigorous distributed agreement in multi-agent systems and verifiable machine learning systems.

    “Whether during my visit to the Systems Research Group in Beijing or through our ongoing collaborations, I have been impressed by how deeply MSRA researchers engage with difficult problems and how openly they exchange ideas. For young faculty and students, it is a rare environment to explore ambitious ideas.“

    — Jialin Li, Sung Kah Kay Assistant Professor, School of Computing, National University of Singapore, MSRA StarTrack Scholar

    The NUS–Microsoft Research Asia AI Summer School has reached more than 300 students since its launch in 2025, attracting participants from Singapore, Southeast Asia, and beyond. Through three days of lectures, hands-on workshops, and small-group discussions, the program creates a forum for researchers and students to engage with emerging AI questions and scholars shaping the field. The 2025 program featured a keynote by Doug Burger, technical fellow and corporate vice president of Microsoft, on the AI revolution. Ahead of the 2026 Summer School, Lidong Zhou delivered a lecture on system intelligence, helping frame the conversations that followed. 

    “I have seen the NUS–MSRA partnership grow through sustained collaboration in AI research and talent development. What makes this partnership especially meaningful is the opportunity to bring together researchers and students from both communities to learn from one another, pursue new ideas, and create lasting impact together.”

    — Tulika Mitra, dean, NUS School of Computing

    Beyond NUS, Microsoft Research Asia — Singapore has built connections across Singapore’s research community, including with NTU, Singapore Management University (SMU) (opens in new tab), Agency for Science, Technology and Research (A*STAR), and other institutions. Research arising from these partnerships has already been published at leading venues, including the International Conference on Learning Representations (opens in new tab) (ICLR) and the Conference on Language Modeling (opens in new tab) (COLM). 

    From early exposure to research leadership: Cultivating Singapore’s next generation of AI talent

    For Microsoft Research Asia, talent development is not a standalone program but a long-term investment in Singapore’s research and innovation capacity, supporting a comprehensive talent pipeline spanning internships, joint PhD supervision, visiting scholar programs, and fellowship initiatives. Through these opportunities, students and early-career researchers gain access to world-class mentorship, a curiosity-driven culture, frontier research challenges, and a global network connecting academia, industry, and real-world applications.

    What makes this model distinctive is its continuity. Students may engage with Microsoft Research Asia through workshops or summer schools, gain hands-on experience through its flagship research initiative, Microsoft Research Asia Stars of Tomorrow Internship Program, or pursue doctoral research through joint PhD programs. Some participants later return as faculty collaborators, mentors, or research leaders. Together, these programs form a connected pathway that supports researchers at successive stages of development. 

    The IPP represents a sustained form of collaboration, enabling PhD students to conduct long-term research under joint mentorship from academia and industry. Qiming Huang, the first IPP student at MSRA — Singapore, is co-supervised by NUS professor Mike Shou and MSRA researcher Xinxing Xu. Through the program, Huang worked closely with researchers from both institutions, gaining exposure to academic rigor and industry-driven research. At CVPR 2026 (opens in new tab), one of the world’s leading conferences in AI and computer vision, Qiming Huang’s paper on RobotSeg – a foundation model for robot segmentation in image and video – was selected for an oral presentation.  

    “Working alongside researchers from both Microsoft Research Asia and NUS changed how I approach AI research. Beyond technical guidance, I learned to frame ambitious questions, collaborate across disciplines, and think about the real-world impact of research

    — Qiming Huang, IPP student from National University of Singapore

    NTU assistant professor Jianfei Yang, embodies the continuous collaborative opportunities enabled by Microsoft Research Asia. As a student, Yang’s team won the Beauty of Programming competition, an experience that inspired him to pursue a PhD and begin an academic career. Over the years, he watched many of his classmates and close friends grow into outstanding researchers through MSRA programs. Today, he collaborates with MSRA researchers in robotics and jointly supervises students through the IPP, creating for the next generation the same opportunities that inspired his own journey.

    “Although I never interned at Microsoft Research Asia myself, it always felt close. Having the opportunity to work alongside a research lab that inspired so many researchers of my generation, and now to co-mentor students together, is both meaningful and rewarding.”

    — Jianfei Yang, assistant professor, director of Multimodal embodied AI and Robotic Systems (MARS) Lab, Nanyang Technological University

    NTU associate professor Ziwei Liu, himself a former MSRA intern, recently saw one of his students become the only Asia-based recipient of the Microsoft Research global Fellowship award for 2025–2026. 

    “As a former intern, I have seen how an environment that brings together world-class researchers, open scientific exchange, and ambitious young talent can shape the next generation of researchers. It is especially meaningful to see my own students now benefit from these opportunities.”

    — Ziwei Liu, associate professor, College of Computing and Data Science, Nanyang Technological University

    The Stars of Tomorrow Internship Program has engaged more than 90 students from Singapore-based institutions over the years. For many students, these internships serve as a gateway to frontier research and help bridge the gap between academic study and real-world innovation.

    For exceptional doctoral students, the Microsoft Research Asia Fellowship provides another pathway for growth, recognition, mentorship, and access to Microsoft’s global research community. Since the program began, 13 students from Singapore-based universities have received the fellowship, many of whom have gone on to become professors, industry research leaders, and technology entrepreneurs around the world.

    The next phase: Scaling what works

    As Microsoft Research Asia — Singapore enters its second year, the focus is shifting from building foundations to scaling the strongest models of collaboration and demonstrating their impact more broadly.

    One year ago, the lab opened with a mission to advance frontier AI in close partnership with the ecosystem around it. That mission is already taking shape through joint research, talent development, and the first partnerships connecting lab work to real-world challenges.

    The year ahead is about scaling what works. By deepening these connections, Microsoft Research Asia — Singapore can bring more frontier research into practice and create meaningful value for Singapore, the wider region, industry, and society. 

  • Introducing Gemini 3.8 Live with Live Avatar

    Introducing Gemini 3.8 Live with Live Avatar

    Gemini 3.8 Live with Live Avatar brings real-time visual presence to Gemini’s conversational AI. By natively coupling our live dialogue capabilities with low-latency streaming video, Live Avatar enables a more natural and intuitive conversational experience for enterprises and their users.


    Shuo-yiin Chang

    Research Scientist

    CJ Zheng

    Software Engineer, on behalf of the Gemini Audio Team


    an image with the phrase

    Building on the momentum of last week’s Gemini 3.8 Live launch, today we are excited to introduce Gemini 3.8 Live with Live Avatar — bringing near real-time visual presence to our native live dialogue models. By pairing near real-time video generation with speech, the Live Avatar feature creates an experience that listens, sees, and speaks with a dynamic visual persona.

    With precise lip-syncing, natural expressions, and fluid turn-taking, Live Avatar enables enterprises to expand their virtual offerings more interactively. Whether providing engaging customer service or delivering interactive walkthroughs, it transforms digital exchanges into richer, more accessible experiences.

    Starting today, Gemini 3.8 Live with Live Avatar is available in Gemini Enterprise.

    See how Gemini 3.8 Live with Live Avatar supports a wide range of characters, each with a distinct look, voice, and expressive presence.

    More natural and multimodal conversations

    Conversation is inherently multimodal: we listen, look, speak, and use facial expressions to communicate. Live Avatar brings these capabilities to enterprise agents. By processing visual and audio inputs simultaneously, it generates enriching conversations for a more comprehensive experience.

    Watch how Gemini 3.8 Live with Live Avatar takes in what it sees and hears in near real time, responding with expressive audio and video for a more natural conversation.

    Asynchronous tool execution with continuous presence

    Beyond visual presence, the feature is backed by Gemini’s advanced reasoning. With asynchronous tool calling, Live Avatar can trigger tool calls and fetch data in the background while continuing active dialogue, handling complex tasks while ensuring an uninterrupted conversational flow.

    See how Gemini 3.8 Live with Live Avatar handles complex tasks like checking in a guest at a hotel. Calling tools in the background while the dialogue continues uninterrupted.

    Conversational experiences built for global scale

    Conversational presence should feel natural and not be limited by languages. Live Avatar features native multilingual speech-to-speech synchronization. The feature dynamically adapts its lip-sync and expressions and can seamlessly transition across 97 languages without degrading video fidelity or introducing visual drift.

    Watch how Gemini 3.8 Live Avatar switches between languages mid-conversation, with lip-sync and expressions adapting seamlessly across 97 languages.

    A Live Avatar to fit your brand needs

    Organizations often need distinct visual identities to fit their brand. In addition to a library of diverse, preset avatars, organizations can customize their Live Avatars. From a high-quality reference image, developers can generate a fully animated, responsive avatar while preserving reference likeness, brand styling, or character identity. Custom avatar creation is currently available only through enterprise allowlisting.

    Trust and transparency at its core

    We built Live Avatar with strict safeguards designed to respect identity, and keep AI-generated content transparent. All output generated by our AI products is watermarked with SynthID. This imperceptible watermark is woven directly into the audio and video output, helping to ensure AI-generated content remains detectable to help minimise misinformation and misattribution. To explore our comprehensive approach to safety and responsible deployment, read our model card.

    Get started with Gemini 3.8 Live with Live Avatar

    Gemini 3.8 Live with Live Avatar is available in Gemini Enterprise. Explore the API documentation to get started.

  • Offloaded inference for real-world physical AI robotics

    Offloaded inference for real-world physical AI robotics

     Two images showing the robot performing the same task of handing over a tape from one robot arm to another. The videos show that while the task succeeds when the inference latency is low, the handover fails with high inference latency as is the case with onboard robot GPUs.

    At a glance

    • Challenges a core assumption in robotics AI: Our research shows that running physical AI inference exclusively on onboard GPUs can limit robot performance, battery life, and scalability, and that offloading inference to edge or cloud GPUs can offer significant advantages.
    • Demonstrates measurable benefits of inference offloading: Across representative mobile manipulation workloads, offloading improved task success rates, enabled larger AI models, and helped robots respond more effectively in dynamic, real-world environments.
    • Extends robot operating time: Replacing power-hungry onboard AI compute with lightweight onboard hardware and remote inference can substantially improve battery life, enabling robots to operate longer between charges.
    • Introduces a new capability in the Physical AI Toolchain: Developers can now containerize, deploy, and orchestrate robotics AI workloads across robots, edge infrastructure, and the cloud using Kubernetes-based tooling for distributed inference.

    Readily-available physical AI, with robotics assisting users in manufacturing, home, and warehouses scenarios, holds immense potential to improve safety, productivity, and assistance across a wide range of tasks. In many ways, AI for the physical world represents a major frontier for AI . Physical AI must operate in open, unpredictable environments, interact with both other robots and people, and work with a diversity of embodiments. Realizing this vision requires advances along three dimensions: robot hardware, embodied AI models, and systems infrastructure for training and inference. While robot hardware and the AI models have advanced rapidly in recent years, we turn our focus on a relatively under-addressed aspect: inference infrastructure of physical AI. Enabling robots to effectively and safely operate in the physical world will require sophisticated systems to handle large volumes of distributed inference compute.

    Today, the prevailing approach to physical AI is to provision a GPU onboard the robot, e.g., by wiring a GPU to the robot. In this model, the robot’s inference will be confined to the onboard GPU, and provide the robot with the necessary chunks and sequence of actions for the execution of its tasks. While higher-level planning may be performed in the cloud, task execution typically remains tied to the robot itself. We challenge this assumption.  As physical AI models grow in size and sophistication, the constraints of onboard compute become increasingly apparent. GPUs consume significant power, reduce battery life, add cost and weight, and can limit the ability to run the latest generation of AI models. 

    To better understand the systems implications of physical AI, we conducted the first systematic study of robotics workloads. We focused on mobile robotic manipulation, with the canonical task such as “check for rubbish in the kitchen and put it in the trash.” Such a task involves planning the path to the kitchen, perceiving the environment to find rubbish, navigating to the rubbish, picking up the rubbish, and navigating back to the trash can for disposal. We evaluated representative models across three core capabilities: semantic mapping and planning, navigation, and manipulation, as summarized in Figure 2. 

    Mobile robotic manipulation involves the components of semantic mapping & planning, manipulation, and safe navigation. The table above lists the state-of-the-art models used for each of these components in our measurement study.
    Figure 2: Details of the models used for the different components of mobile manipulation.

    Offloading physical AI inference out of the robot improved its response time and accuracy, along with battery lifetime and cost. We evaluated the inference models across a range of onboard, edge, and cloud compute configurations. Details of the specific test hardware are available in our technical report. 

    Diagram showing that offloading Physical AI inference from robots improves performance, battery efficiency, and cost.

    Benefits in task performance: Our evaluation shows offloading inference can significantly improve robot performance across mapping, planning, navigation, and manipulation workloads. Some smaller GPUs could not accommodate the mobile manipulation stack. On GPUs with sufficient memory, mapping and planning slowed by up to 383% compared to an A100, thus limiting the robot’s abilities in dynamic spaces. Navigation showed a 30% drop in its timely detection of obstacles with lighter GPUs. While the VLA models did not dramatically slow down with smaller GPUs, the slowdown was still sufficient to drop their accuracies by 50%. In other words, onboard GPUs limited the performance of the robots while offloading their inference to an on-premise or cloud GPU boosts their operations, as shown in the videos below and quantified in the graphs. As physical AI models continue to grow in size and complexity, the benefits of offloading are likely to become even more pronounced.

    Figure 3a: The video shows the handover task with onboard GPUs.

    Figure 3b: The video shows the handover task when the inference is offloaded.

    Robots can support GPUs such as Jetson Nano and Orin AGX onboard. The graphs above show that the success rate of object handover and navigation are low with onboard GPUs, but increase substantially with beefier GPUs when offloaded such as Jetson Thor and A100.
    Figure 4: Success rates of robot arms handing over objects to each other when inference is performed with different GPUs (some onboard, and some offloaded). Offloading improves success rates.

    Benefits in battery lifetime: Beyond performance, onboard GPUs also significantly drained the battery life of the robot. We compared the increase in battery lifetime by replacing an onboard GPU with a Raspberry Pi-5 board and shipping all the data to the offloaded GPU. The larger onboard GPUs, such as Jetson Thor, drained robot batteries by up to 160% (or a few hours) for even the larger robots.

    Battery lifetime of the Stretch-3 robot can improve by over 100% when their GPU inference is offloaded out of the robot.
    Figure 5: Impact of offloading GPU inference on the battery life of the robots; the above numbers are for the Stretch-3 robot.

    The above results show that offloading GPU inference out of the robot is critical for functioning in the open world with large models and long battery lifetimes. Nonetheless, offloading inference out of the robot involves a complex tradeoff involving performance, network latency and bandwidth, and available GPU resources. We believe that our measurement study will inform the design of physical AI inference systems.

    Spotlight: Event Series

    Research Forum | abstract background with colorful hexagons

    Microsoft Research Forum

    Join us for a continuous exchange of ideas about research in the era of general AI. Watch the latest episodes on demand.

    We have built a toolset for easy inference offloading out of the robot and distributing inference between the edge GPU and cloud. Kubernetes is a natural platform to provide a uniform abstraction to distribute robotic AI between the robot’s compute, edge GPU, and overflowing to the cloud. The toolset allows automatic containerization and offloading of robotics workloads using declarative specifications, distributes physical AI containers with smart policies using Kubernetes, and integrates with robotic simulators, LeRobot, and ROS2 for easy development. The sequence of steps below shows how the toolset can be prompted with what to offload, and how it creates a separate container for GPU inference and offloads the same.

    Toolset to make offloading of physical AI inference easy by automatically generating a container for GPU inference and deploying it using Kubernetes to the machine with GPU.
    Figure 6: Steps in the offloading toolset with containerization and deployment.

    Microsoft has recently released the Physical AI Toolchain (opens in new tab) for operationalizing physical intelligence at scale. Physical AI Toolchain is an open-source, production-ready framework that integrates Microsoft Azure (opens in new tab) cloud services with NVIDIA’s (opens in new tab) physical AI stack, accelerating robotics and physical AI developers to automate and scale data curation, augmentation, and evaluation across perception, mobility, imitation learning, and reinforcement learning pipelines. We are announcing the addition of an industry-first capability for offloaded physical AI inference for robots as part of the Physical AI Toolchain. This release includes example projects for offloading inference of a SO-101 and a UR10e. The videos below show the offloading of the inference of Microsoft’s Rho model, targeted at dual-arm robots, to a Jetson Thor GPU, which controls the actions of the Mobile Aloha robot (opens in new tab).  

    Figure 7a: Offloading of GPU inference of Microsoft’s Rho model controlling the Mobile Aloha robot for interactions with the BusyBox to press the blue button.

    Figure 7b: Offloading of GPU inference of Microsoft’s Rho model controlling the Mobile Aloha robot for interactions with the BusyBox to turn the knob to position 4.

    Check out the inference offload feature, look into the source code, and let us know your feedback. We have already tested it with many real-world use cases, and look forward to hearing about your deployment experiences.

  • Advancing Private AI Compute with secure, server-side memory

    Advancing Private AI Compute with secure, server-side memory

    September 23, 2026 Responsibility & Safety

    Google Private AI Compute Team

    A technical update on our Private AI Compute architecture, which will enable persistent, cross-device AI memory with on-device privacy standards.

    AI is becoming more capable and intuitive — remembering what matters, understanding the world around you, and acting at your direction. Privacy and trust are core to making that possible, ensuring your data stays private and protected as AI systems evolve to provide more continuous assistance across your devices.

    Today, we are sharing how we will bring private, server-side memory to our Private AI Compute platform. This breakthrough resolves a longstanding dilemma in modern AI: how to give an assistant long-term continuity across devices while upholding the strict privacy standards typically limited to on-device processing.

    Bringing on-device privacy to cloud-scale memory

    With this new technical capability, a new persistent memory layer will be able to function like a secure digital vault in the cloud. Under this model, the information needed to assist you is sealed within dedicated, encrypted storage, while the cryptographic keys required to unlock it are held exclusively on your personal devices — ensuring your data is inaccessible to anyone else, even Google.

    The diagram below shows how this update to Private AI Compute will work. When an AI model needs to access information to assist you, an authenticated, end-to-end encrypted channel connects your device to a protected, isolated environment in the cloud. That space, or “secure enclave,” temporarily decrypts your data in isolated memory to handle the request, saves any new context, and immediately encrypts it, keeping your information private as if it never left your device.

    An architectural diagram illustrating a secure data flow from a user client, through secure enclaves for inference and server-side memory, to memory storage, featuring encrypted communication channels and key management (KEK, DEK, and Wrapped DEK).An architectural diagram illustrating secure data flow from a user client, through two secure enclaves (an inference pipeline and server-side memory), to memory storage, with accompanying details on encrypted communication channels and key management (KEK and DEK) below.

    By combining hardware-enforced secure enclaves, encrypted channels, and per-user databases shielded by device-derived encryption keys, this architecture ensures your data stays fully private and under your control.

    This evolution is necessary to meet the computing needs of the AI era. Local, on-device processing has historically been the gold standard for privacy — but frontier AI models often require far more computing power than any one device can provide. Bringing advanced AI to personal assistants means solving how to tap into the power of the cloud while ensuring personal data can remain as protected as if it never left your device.

    To that end, we previously introduced our Private AI Compute platform, allowing users to process complex tasks in hardware-isolated cloud enclaves. Until now, that technology — along with similar solutions across the industry — was strictly “stateless,” meaning it wiped all context the moment a task ended. Workarounds, like having AI save a list of personal facts and preferences, aren’t enough to support the rich, continuous experiences people expect from personal AI. Making that level of assistance possible means engineering a way for cloud-scale AI to securely retain context over time and across devices.

    Building trust, looking ahead

    Imagine pulling up assembly instructions on your laptop that you previously viewed through smart glasses, or resuming complex conversations between mobile and web. Private AI Compute is designed to make that kind of seamless assistance possible – keeping the pieces it needs to remember safely locked away. But the user’s trust in that system’s privacy is also important.

    Building that trust starts with transparency. That’s why, alongside our updated technical whitepaper, we’re publishing a tamper-proof public record of our server software. Devices running Private AI Compute will be able to verify that our software is authentic and unaltered before sending any personal data. In addition, we’re providing an update on our technical methods, including the results of an independent audit by a leading cybersecurity firm. By sharing these resources, we invite the broader privacy community to verify Private AI Compute’s protections.

    Adding private, persistent memory to Private AI Compute shows how deeply personal assistance can be private by design. We invite the community to review the updated Private AI Compute Technical Brief and our system architecture, security proofs, and verification protocols.

    Acknowledgements

    This research was co-developed by Google DeepMind, Platforms & Devices, Core and Cloud teams. We would also like to thank Four Flynn, Jay Yagnik, and David Kleidermacher for their executive sponsorship of this work.

    Piloting the world’s first double-blind AI evaluations

  • Gemini 3.8 text-to-speech says hello

    Gemini 3.8 text-to-speech says hello

    Gemini 3.8 Flash TTS and Gemini 3.8 Flash-Lite TTS are our most expressive audio generation models yet. Generate custom character voices and direct scene dialogue across Google AI Studio, Gemini API, Gemini Enterprise, Gemini Notebook, and Google Vids.


    Leland Rechis

    Group Product Manager

    Alan Cowen

    Director, Research Science, on Behalf of the Gemini Audio Team


    a text card image reading

    Today, we’re introducing two new text-to-speech models to the Gemini family, transforming voice generation from static presets into a dynamic creative studio. These models enable creators, developers, and enterprises to create richer, more expressive audio experiences, while enabling improved user experiences in products like Gemini Notebook and Google Vids.

    • Gemini 3.8 Flash TTS: Built for deep creative direction and character design. Create entirely new voices from scratch using natural language prompts to bring characters to life across gaming, immersive audiobooks, podcasts, and interactive media. Direct every performance line by line with granular control over acting cues, pacing, dialect shifts, and backchanneling.
    • Gemini 3.8 Flash-Lite TTS: Built for high-volume, cost-efficient scale. Optimized for high-volume dubbing, audio content creation, and expressive voice agents with fine-grained control over tone, pacing, and expressive nuance.

    These models complement our fast-growing Gemini Audio family, following 3.5 Live Translate, 3.5 Transcribe, 3.8 Live, and 3.8 Live Extended Thinking.

    Create and customize your own voices

    Scale up from 30 original voices to an infinite library. Whether you need an entirely original character voice or a consistent brand ambassador, our 3.8 Flash TTS model powers a full vocal studio. This enables you to create and use expressive, natural-sounding voices for every moment, while empowering developers and enterprises to easily build custom audio experiences.

    • Generative voice design: With Gemini 3.8 Flash TTS, create bespoke voices from scratch by customizing role, accent and voice characteristics across more than 100 languages and dialects using natural language prompting — whether you’re bringing a dramatic, fire-breathing dragon to life or crafting a charismatic narrator with a distinct regional cadence.

    • Expansive voice library: Access 2,000+ production-ready voices with broad language coverage — including regional varieties like Mexican Spanish, Quebec French, and Scots English.
    • Voice replication: Recreate consistent vocal profiles from just a 30-second audio sample of your voice or a voice you have the rights to use, backed by built-in consent verification, SynthID watermarking, and C2PA credentials to protect both developers and their vocal talent.
    • Save and scale: Save and manage the custom voices you designed to ensure consistent performance and minimal drift across ongoing projects.
    • Voice remixing: Coming soon, pick a voice from our voice library and fine-tune timbre, pitch, pace, and accent. Use prompts to dial in characteristics (e.g. “add subtle Southern US accent” or “soften the delivery”).

    Direct the performance, line by line

    Once you’ve selected your voices, both TTS models give you precise control over how each line is delivered.

    • Direct performance line by line: Write your own stage directions or let Gemini steer delivery with natural script cues — from a calm customer service agent to a whispered suspense scene.

    • Long-form generation: Maintain high voice quality, natural pacing, and character timbre across hours of continuous audio with minimal speaker drift — ideal for podcasts and audiobooks.
    • Native two-speaker scene staging: Direct multi-turn conversations seamlessly from a single script —whether for a podcast or dramatic storytelling—while keeping both voices distinctly separated with natural conversational turn-taking.
    • Scripted vocal bursts & backchanneling: Add realistic conversational texture using non verbal cues (like , , and active-listening interjections (like |mhm| or|yeah|) for precise comedic timing and reaction beats.

    Get expressive high-quality speech generation built for global scale

    Gemini 3.8 Flash TTS delivers leading voice customization capabilities, securing the #1 overall spot on Hume AI’s Voice Design Benchmark (71.4) and also leading in accent modeling (60.8).

    Gemini 3.8 Flash TTS and Gemini 3.8 Flash-Lite TTS enable truly expressive performances without sacrificing reliability, also securing the #1 and #2 spots respectively on Hume AI’s Overall Quality Index. The model shows major improvements on a wide range of use cases such as long-form content and dual-speaker screenplay control compared to Gemini 3.1 Flash TTS.

    In blind human preference evaluations on Voice Arena, Gemini 3.8 Flash and Flash-Lite TTS secure top positions amongst competitors in key global languages, including Japanese, Brazilian Portuguese, Vietnamese, Modern Standard Arabic (MSA), Mexican Spanish and Hindi. With support for over 100 languages, these models empower creators, developers, and enterprises to build high-quality, multilingual voice experiences worldwide.

    An evaluation showing text-to-speech quality benchmark Hume AI

    an evaluation chart showing text to speech voice design leaderboard Hume AI

    an evaluation chart showing text to speech leaderboard for Voice Arena

    Build with trust, consent, and transparency

    We built our voice creation and replication capabilities with strict safeguards to help protect voice talent, respect identity, and ensure content transparency. For voice replication our system leverages consent verification: users must provide a verbal consent recording from the voice owner that matches the reference speaker before a voice can be created.

    More broadly, every audio clip generated by our Gemini Audio models is watermarked with SynthID. This imperceptible watermark is woven directly into the audio output, ensuring AI-generated speech remains detectable to help prevent misinformation. For more details on our approach to safety and responsibility, review the model card.

    Try our new Google AI Studio audio playground

    Starting today, developers can experience these new speech generation capabilities in Google AI Studio. Built like a voice design workspace, you can prompt entirely new vocal identities from scratch or replicate your own voice 1 , then bring them directly into a dual-speaker screenplay editor to direct line-by-line delivery.

    Try voice replication in Google AI Studio.

    Deploy high-performance voice interfaces with ease

    By using the Gemini API, developer platforms such as Agora, LiveKit, Pipecat, Vercel enable developers to build and deploy high-performance speech generation experiences with ease.

    We’re partnering with companies like Figma, HeyGen, Linguana, Wondercraft, 99.co, and Ollang, who are integrating our latest TTS models to help accelerate global dubbing, localize media with nuanced regional accents, and power conversational voice agents at scale.

    a quote from Darius Cheung, CEO and Co-Founder of 99 Group

    a quote from Mason Adams, Developer Evangelist, Agora

    a quote from Jonathan Gur-Zeev, Director of Product, Figma Weave

    a quote from Bin Liu, VP of Engineering for Hygen

    a quote card from Luke Pane, Developer, katsuyo

    quote from Ritwik Baranwal, Associate Director AI/ML of kuku.

    a quote from Oded Shafran, Co-Founder & CTO of linguana

    Aziz Ulak, CTO & Co-founder, Ollang

    a quote from Phil Marshall, Founder and CEO of Spoken

    quote from Zina Rahman, Co-Founder and CEO, Transforms.AI

    quote from Mei Ki Yiu, CTO of Wondercraft

    Start using our latest Gemini Audio models:

    Gemini 3.8 Flash TTS is rolling out starting today:

    Gemini 3.8 Flash-Lite TTS is rolling out starting today:

    Get the latest news from Google in your inbox

    Sign up for our newsletters with product updates, event information, special offers, and more.

    Your information will be used in accordance with Google’s privacy policy. You may opt out at any time.