Sam Altman on AI Progress, Safety, and Staying Human While Scaling Superintelligence

The host opened by welcoming Altman to the Stanford Graduate School of Business and said the audience was excited to have him. The host said they had prepared by talking to an AI version of Altman built with Codex. According to the host, it had answered most of their questions, but they still believed nothing beats an in-person conversation.

Asked about an internship he nearly took in investment banking, Altman confirmed it was true. He said he had never wanted to be an investment banker, but it was what the cool kids at Stanford wanted to do, and he fell victim to peer pressure. He stressed that anyone who wants to be an investment banker should go for it, but said it was not his natural interest. He described wanting to be seen as one of the smartest students, and reasoned that the smartest students were pursuing banking.

Altman said the banking firm flew him to New York for an interview and an impressive dinner, which made him feel the path must be right for him. However, a growing sense of dread set in, and he was sure he would not enjoy the work.

Meanwhile he had been working on a project with friends. When Y Combinator offered money for the summer to try a startup, he took it as a sign from the universe and went to do it instead.

Altman said he was sad to drop out of school because he loved it, and worried that he would stop learning. He said that one of the best things about building a startup was that it was the highest-rate learning environment he had ever been in. He recalled being told, possibly by Paul Graham, that you get an entire MBA in one month of doing a startup. Although he does not have an MBA and could not verify that claim, he said he learned a great deal very quickly about business, product, customers and sales, which he believes he could not have learned in school.

He said that the first period of his startup was an unbelievable rate of learning, and that he has never learned more in any other period of his life. Running Y Combinator was a different kind of learning. He compared it to something close to a government job, since he did not have the power of the companies it backed, but he did learn things about running a startup that helped at YC. He also said people almost always surprise themselves with how quickly they get good at something if they jump in and do it.

Asked why he and his team believed in artificial general intelligence when most people said it was impossible, Altman said that deep learning was working. He recalled that as a Stanford student in 2005, when he planned to work in AI, a Stanford computer science professor told him that the one thing they knew for sure was that deep learning was not going to work, and that AI would probably take at least another century.

He said that the 2012 AlexNet result came next, and that by 2014 deep learning seemed to improve with scale in many different ways. He said the world largely did not accept this, and that it is hard to overstate how unpopular the view was. Today, he said, the view that deep learning is a parlor trick is the radical one. He argued that his central career insight has been that there are surprising effects to scale, and that when you see a beautifully scaling exponential, you should keep going.

The host noted that OpenAI began as a nonprofit research lab with no intention of becoming a product company, and that over a billion people now use ChatGPT every week. Altman said that in the mid-2010s he, like many in Silicon Valley, was enamored with research labs, and that he had always thought research was awesome and wanted to run a research lab. He said it was supposed to be his retirement project, and that for the first five years it was the coolest job he could imagine, with a small group working on research.

Altman said the team was right about many things but wrong about how AI would develop. It had not occurred to them that the most likely path was building a product company. They had mostly thought they would develop an unbelievable superintelligence without then having to do the hard work of getting it usefully into people's hands and benefiting society.

He said that by late 2020 or early 2021 it had become clear they would have to build a big company. He did not think it would happen so quickly, and when ChatGPT launched almost four years ago, everyone suddenly understood they would become a large company very fast.

Asked which decisions are hardest for him today, Altman said the answer would take the rest of the time. The first one that came to mind was how to pause or pace progress, which people describe in different terms such as intelligence speed limits. He said that if people should always control the future, and should neither irreversibly hand decision-making power to AI nor accidentally lose control, and if power needs to be distributed in the future, then difficult questions follow.

He asked how much smarter than the smartest people an AI system should be allowed to become, how quickly that should happen, and how much people's ability to be smarter or to understand these systems needs to be enhanced. He said he thinks this is solvable but takes time. He noted that there are limited ways for the world to agree on those limits, and that people hold very strong opinions on all sides, from those who say any regulation is bad to those who say all progress must stop now. He described trying to be the pragmatic centrist in that debate as a very difficult position.

Asked how his thinking about being a pragmatic centrist and about AI safety has changed as capabilities accelerated, Altman said he still believes in his earliest tenets about how AI should be deployed in society. He still believes in iterative deployment, that society must contend with the technology as it develops even though the rate of change is difficult, and that people are incredibly resilient and adaptable. He said people do not care that much about the work of machines and are focused on other people, and that the societal shape of the future will be more similar to today than people think.

He said that watching progress over the last six months has tested the strength of his convictions. In some narrow senses, such as mathematical reasoning, he said that as far as he is concerned, superintelligence is already here, which he had not expected to happen so fast. If a similar rate of progress continues for another year or two, he said, it seems unlikely the current approach will work without changes, given how capable the systems are becoming so quickly.

Asked what business leaders can do to prepare for AI's risks, Altman said there are many levels, but that the one he would highlight is cybersecurity. He said he does not even know what word is dramatic enough, but that the problem coming toward us is still something that organizational leaders, countries and individuals have not taken seriously.

He described a moment earlier this year, which people called the Mythos freakout, after which some bugs were patched and things got somewhat better. However, he said people still did not intuit the exponential. Models can find many security vulnerabilities, and people assumed those were all the vulnerabilities that exist or that models would ever find. He said he did not think most people had concluded that the models would become much smarter and could find far more serious bugs, including ones outside the usual threat model. Altman said he no longer thinks the team will be able to patch all the bugs, and that the defense model will need to switch to using the same technology for a new kind of defense shield. He called it a huge safety and security issue and said he was somewhat astonished by how little attention it is getting.

Asked how students can shape what comes next, Altman said his honest belief is that AI will not change things as quickly as people think. He said it is true that huge numbers of tasks people do today will be done better, cheaper and faster by AI, and that wide swaths of current jobs will, at least in the way they are done today, be done entirely by AI.

He described the reaction of the mathematical community over the previous ten days. On day zero, the reaction was that it was fake. On day one, it was that they were cooked, their field was over, and they were telling graduate students to switch majors. On day three, some said it was horrible that OpenAI had done this and that no one should be allowed to do it. By about day five, the reaction had shifted, and mathematicians told him that while one kind of math job was over, most of what they did was not solving Millennium Prize problems. He said most of their work was making sense of knowledge, communicating it and presenting it in a human-intelligible way, and that many now expect a different but hopefully better kind of career.

He also described consultants who first thought their work was over, then found that clients still want to interact with another person, and that almost no one wants an AI consultant to show up and tell them what to do. Altman said he does not think there will be a job apocalypse in the way many people talk about it, but there will be huge short-term changeover in tasks and some kinds of jobs. He said people with high potential, drive and new skills tend to do best when the ground is shaking the most.

Altman said his chronic disappointment with MBA students is that they always want to do the thing that was cool five years ago rather than the new thing. He said this is a great time to do the new thing. When the host asked whether they should do the new thing even if AI is the cool thing right now, Altman said he does not think it is a good time to start another copy of OpenAI. However, he said many companies can be built on top of the technology OpenAI has released, and he hopes they will become much bigger and more important than OpenAI.

Asked what problem people still dramatically underestimate their ability to solve, Altman said that people, including skeptics, now say AI will solve every piece of knowledge. He said he does not fully agree but partly feels that way. He argued that having a great life, with real choice, freedom and flexibility, and the ability to make mistakes but look back and say the life was good, will still be hard, and that this difficulty may be important.

He said he hopes that in the coming years society moves toward what feels like a new renaissance rather than a new industrial revolution. He described the industrial version as automating everything, taking humans out of the equation, and having people become cogs running more smoothly in a machine. He said a lot of the energy in AI has gone toward that industrial vision, and that he hopes it gets the right outcome. He also said people actually want the renaissance version, with creativity, the ability to explore and the ability to discover knowledge, but that economic pressure toward ruthless efficiency means there will need to be some counterforce from companies, governments or people as a whole.

Asked what infrastructure OpenAI is betting on next, Altman first said robots. He said people talk a lot about recursive self-improvement for AI models, but there is another loop: if robots can build more robots, and data centers can effectively build more data centers, including by operating chip fabs, you can make far more compute all the way down to building more power plants. He said this is almost as big a deal as the potential recursive loop in AI research, though it gets much less attention.

He also said OpenAI made its own chip, called Jalapeño, with another one coming after it, which he described as huge wins in intelligence per watt compared with anything else on the market. He expects smarter models to help design smarter chips, creating another loop. He said OpenAI is investing across the whole supply chain, including fabs, power and all the equipment in data centers.

Asked about a strong belief he held in late 2020 that turned out to be wrong, Altman said he did not think they would be able to reach models that were superhuman in some important ways while alignment stayed easy. He said he had thought the alignment problem would be harder, and that it may still turn out to be very hard. He suspects it will get much harder, and that it is very important from here.

He said that since they reached models that are very superhuman in some important ways, and alignment has so far been easier than expected, he would have said with a high degree of certainty that one of those two things would be wrong. He worried that some people look at progress and assume alignment will stay easy as models get more capable, which he said is not obvious. He also said the potential cost of alignment being wrong has gone up enormously. He believes sandboxing and monitoring may work for a while, but eventually nothing but alignment will work. He said he worries some AI efforts around the world are so confident in alignment, given how easy it has been so far, that they are too willing to run ahead.

The audience member noted that Altman had recently said on a podcast that the world should accept some bad things in exchange for AI's benefits, and asked where he draws the line. Altman said he wished he had phrased that answer differently. He said open-source models will output a lot of hate speech, troll internet forums, hack into some systems, try to commit fraud on behalf of users, and that a new generation of phone scams will be much worse than the previous one.

He said he was trying to argue that banning all open-source models would be an extremely high cost to innovation and liberty, and that society should not do it. However, if open-source models exist and people can run them locally, there will be things nobody wished had happened, no matter how good monitoring is. He said there is a threshold above which models can do enough harm that people should not be given the liberty to use them however they want, and that society needs to debate where that is. He gave the examples that no one should be able to take down the internet or cause physical harm to another person.

Asked how software developers can build moats when better AI models increasingly absorb software functions, Altman said there have not been many software moats in history. He listed other kinds of moats, including network effects such as Facebook's, scale effects such as Google's, and data moats common to many enterprise software companies. He said software itself has never been a super strong moat. Instead, it is used to gain a temporary advantage that helps build one of the traditional moats of a business, and he expects that pattern to continue.

Asked what future leaders should focus on to drive change, Altman said his obvious answer is AI, and that one of the most tractable areas is changing organizations. He said it is now possible to design a very different kind of organization with a lot of AI relative to each person in it. He called the question of how to set up the best possible organization today an underexplored area of study.

He offered speculation that most organizations were designed by copying the military's hierarchical command-and-control model, which was the one example they had of something that could scale beyond a certain point. He said the tech industry has improved on this but is still inspired by the same basic principle. He found the way AI agents worked together to be very interesting. He pointed to 10,000 agents that solved Navier-Stokes, which worked together in a way unlike any organization he could imagine: there was no leader, no credit squabbles, perfect alignment and no bickering. He noted that humans have beautiful flaws that mean people would not work that way, but he thinks there is still something to learn from it.