Why PMs Matter More in Frontier AI: Notes From Google Gemini's Head of Product on Evals, Safety, and Shipping

Doshi began by describing model development as something other than a fully predictable engineering process. In her words, "The model isn't predictable. It's kind of like Snape's potions class." Developers can put all the ingredients in, but they have to expect some magic, and that magic will be surprising.

For a PM, that means shaping the inputs and direction of a model while also recognizing where its unexpected behavior is useful, and turning that into something users value.

Asked what Google looks for when hiring PMs, Doshi said the team is looking for people who understand the "vibes" of its models or of other models. She said she wants PMs who have an opinion and who play with these models every day.

She said the ideal candidate stress-tests a model on things they are excited about. She also addressed a common view that Google lags behind the top two labs, saying she thinks Google does want to win the frontier conversation and should not be described as not participating in the frontier.

On AI safety, Doshi said the safety teams are working with intention from initial training through deployment, and that safety is embedded in that process. She emphasized the mitigations and structures placed around models. One example is deploying monitors that can watch a model's chain of thought and identify whether it is aligned or misaligned with its directed actions, and then intervene. She said this gives more confidence in releasing models and also helps the team understand where misalignment appears and improve models over time.

She also pointed to AI's role in hardening systems. Doshi said one thing the team is doing with Gemini 4 is investing heavily in cyber defense and AI for protection, which the host noted would mean significant spending on defending systems against increasingly capable AI.

Asked whether AI safety is a solvable problem and whether she is optimistic, Doshi said the answer is both: the team needs to keep thinking about it consistently. She said her optimism comes from whether people are being intentional and thoughtful about what the safety story looks like.

She said a number of people need to stay worried, keep having conversations, and keep interrogating how models are built. She described that as the step-by-step process the field has to follow.

The host noted that AI products are becoming very complicated. He cited criticism that Google makes it hard to find its many AI tools, saying many people do not know AI Studio exists. He also said interfaces for Codex and ChatGPT and Claude Code have grown into tabs, dropdowns, and dots.

He relayed a point from a PM at one of the labs, who described urgency to get frontier AI to users as fast as possible. The reasons given were partly to stay competitive and partly because the lab does not yet know exactly what people will use the tools for, so the plan is to ship, learn the use cases, and then improve the user experience.

Doshi said the team needs to keep experimenting with new paradigms and interaction approaches. She pointed to Flow, a product designed around video and music editing and generation, and NotebookLM as examples of different form factors. She said Google's labs teams are actively testing different forms and UX for different individuals.

She added that the right UX depends on the user. AI Mode in Search serves billions of people asking questions, so it needs an accessible interface for everyone. Antigravity, by contrast, targets power users who use slash commands and work across the terminal and the IDE, so it needs to support a wide range of actions.

Doshi also said a model's behavior is itself part of the experience. Her line, "The personality of the model is the UX of the model," was offered as part of that point.

The host said a common view is that even if model development stopped, models can already do far more than users are doing with them, which he called an overhang. He asked whether exposing these powers to users is a large part of the PM job.

Doshi said a fun part of the work is figuring out what is new and where the next frontier lies, and deciding which capabilities to invest in. She said this happens in close collaboration with internal teams such as Search, the Gemini app, YouTube, and Labs, and also with external customers to understand their needs and what would meaningfully change their experience. The goal is to build a flywheel of feedback.

Asked how someone new to evals can get better at writing them, Doshi said Google is still working out how to upskill its PMs on this, and that some PMs are stronger at it than others. She said a big step is to start with just five prompts rather than trying to build a massive benchmark.

She said even a mini evaluation of about 20 prompts can stress-test a set of capabilities and show signal between two models or between model A and model B. Starting simply, she said, can mean judging the outputs yourself. She noted that defining what good means is hard, especially for long, multi-step, multi-turn trajectories, and that collecting good examples gives clarity about what you are trying to achieve before you scale up.

The host introduced a story shared by Eli Collins, VP of product at Google DeepMind, about a shift in how models and products were evaluated for skin tone. Doshi said that during her six years in responsible AI, a large part of the work was examining skin tone representation in models and products.

She explained that the Fitzpatrick scale, with six skin tones, was the standard early on. Realistically, she said, six tones cannot capture the diversity of skin tones in the world. The scale had also been built for healthcare, which meant it skewed very light for a global population.

The team worked with Harvard professor Monk, whose scale of ten skin tones came from user interviews and years of research in anthropology and sociology. The team brought that scale into its evaluations.

The host introduced a claim that Doshi researched agent-like coordination years ago, before agent swarms became a common topic. Doshi said that at Stanford she worked with Professor Michael Bernstein on what he called flash teams.

The question was how to bring together crowdsourced workers and efficiently pull together the skills needed for a task. Different people have different strengths, so the team had to be assembled intentionally to make crowdsourcing at scale efficient. The work was done with people, using platforms such as mTurk.

She said the principle was that a team can be spun up and spun down while staying efficient for both the individuals and the person running it. Now, she said, agents make it possible to replicate that world, and the same principles can apply to setting up an effective system of sub-agents.

Doshi described a framework for how far ahead to build when using models. She said the important point is that models keep changing, so the capabilities available today will not be the ones available tomorrow or the next day. Builders therefore need to create features that are not always working today.

Her framework says that about 60% of what you build should work now, and the prototype should at least be functional. About 20% should be not perfect but good enough, where you can see how models might get there in the very near future. The final 20% is where you see glimmers of light and believe that if you could crack it, it would be an amazing product experience.

She contrasted this with the common advice to build about three months ahead of where models will be, saying that even as a model person she finds it hard to predict exactly where models will be in three months. Her proxy is to use product signals to see where those glimmers of light are, which she likened to a portfolio of product bets.

Doshi said some PMs on her team have leaned into internal tools and play with models often, and that they have become valuable to the rest of the team by building skills others can use. The team has started sharing skills in a repository for the PM team.

PMs who have become stronger at eyeballing data and reading longer traces and trajectories can run workshops for others, so the team collectively builds that confidence. She said the core of her approach is confidence-building: how to build everyone's confidence in the team's tools and their ability to use them effectively.

The host proposed one of the most important new habits: between a stimulus and a response, insert the question "can AI do this for me?" He said he has built that habit much more strongly over the past few months, which has made a large difference as he builds workflows and automations.

Doshi agreed that some tasks, such as writing a strategy document, may still need to be written by hand, but she said many tasks can be checked for replicability and for how much the models can help. She added that one successful example is often enough to change a mindset, and that confidence in what the model can do grows from there.

Doshi said her team is putting in a lot of hours, and she is asking how to create a sustainable work environment. She said burnout comes from two sources: one is feeling that there is churn and chaos, and the other is work that feels like grunt work or a waste of time.

She said working on something you are passionate about is exhausting but also brings more joy. She expressed hope that AI tools can help by removing grunt work and frustrating parts of the job, freeing more time for thinking and creative work.

The host added a finding from his own survey of tech workers: more hours does not equal burnout. Many people are working the hardest they have ever worked but enjoying their careers the most, especially PMs who are building, prototyping, and talking to customers.

Asked about processes for managing AI-written content across teams, Doshi said the team talks a lot about where such documents are useful. She cited a post-launch document as an example: two weeks after launching something, a team may want the latest on who is using it, how, and what the feedback is. AI is well suited to that because it can pull from dashboards, metrics, and analysis of social discussion much faster than a person.

She said the team should be intentional about separating what AI contributes from what people contribute. The model should provide the data, and the person should provide the insight. The team is also investing in skills for getting the right data so those artifacts can be structured well, which she hopes frees more space for creative thinking and writing.

Doshi said she believes things happen for a reason, and that even when you do your best, sometimes a door closes and another window opens, placing you where you are meant to be.

She also said she often talks about seeking serendipity. In her view, you should always put yourself in a position where you can get lucky, which means challenging yourself, stepping outside your comfort zone, and trying something new. Those are the places where you are most likely to get lucky. She mentioned a TEDx talk she gave about losing serendipity, and the host noted that he is a fan of the word.