WEBVTT 00:00:01.420 --> 00:00:17.140 Wednesday, November thirtieth, two thousand twenty-two. OpenAI publishes a short blog post. No livestream. No keynote. The title is two words: Introducing ChatGPT. 00:00:17.140 --> 00:00:34.900 The first paragraph says what it is. A model "which interacts in a conversational way." And then it lists four things the conversation makes possible. Read them slowly, because each one is a small confession about every model that came before. 00:00:34.900 --> 00:00:47.760 It can "answer followup questions." It can "admit its mistakes." It can "challenge incorrect premises." And it can "reject inappropriate requests." 00:00:47.760 --> 00:01:09.480 Admit its mistakes. Two years earlier, a software engineer had asked GPT-3 who was president of the United States in seventeen hundred, and it had told him, with total confidence, William Penn. Challenge incorrect premises. That's the same machine that told him his foot had two eyes. 00:01:09.480 --> 00:01:35.360 The post also says it's free, for now, during what it calls a "research preview." It says the company wants to "get users' feedback and learn about its strengths and weaknesses." And it has a section, right there in the launch post, titled "Limitations." The first one: ChatGPT "sometimes writes plausible-sounding but incorrect or nonsensical answers." 00:01:35.360 --> 00:01:53.560 Nobody at OpenAI thought this was the most important thing they would ever release. A year later, Sam Altman would describe it on stage as, quote, "a low-key research preview." Within five days, it would have a million users. 00:01:53.560 --> 00:02:18.600 But this episode isn't about those five days. It's about the two years before them. The years when the people who'd built GPT-3 split in two, and the half who stayed figured out how to make a model that does what you ask. Quietly. In blog posts almost nobody outside the field read. 00:02:18.600 --> 00:02:31.600 From Will Sigmon Media, this is The Role Models. Season one: OpenAI. Episode seven. The Quiet Build. 00:02:31.600 --> 00:02:48.900 We left Dario Amodei at an employee meeting in December twenty-twenty, thanking Sam Altman and saying goodbye after nearly five years. OpenAI's post said he and a handful of colleagues were starting a new project. It didn't say why. 00:02:48.900 --> 00:03:07.040 For a long time, Amodei didn't say much in public either. The fullest account he's given came almost four years later, in November twenty-twenty-four, in a long interview on Lex Fridman's podcast. Fridman asked him directly. Why did you leave? 00:03:07.040 --> 00:03:18.420 I'm going to give you his answer in his own words, at some length, because it's the only first-hand account we have, and because it's careful about what it doesn't say. 00:03:18.420 --> 00:03:46.720 He started with what he'd learned at OpenAI. He'd come to appreciate, he said, "the scaling hypothesis," the idea from our last episode, that bigger is predictably better. And he'd come to appreciate "the importance of safety along with the scaling hypothesis." The first one, he said, OpenAI "was getting on board with." The second, "in a way had always been part of OpenAI's messaging." 00:03:46.720 --> 00:04:01.720 Then, he said, "over many years of the time that I spent there, I think I had a particular vision of how we should handle these things, how we should be brought out in the world, the kind of principles that the organization should have." 00:04:01.720 --> 00:04:06.040 And he went straight at the two most common explanations for why he left. 00:04:06.040 --> 00:04:21.660 "People say we left because we didn't like the deal with Microsoft. False." He allowed that there had been "a lot of discussion, a lot of questions about exactly how we do the deal with Microsoft." But that wasn't it. 00:04:21.660 --> 00:04:33.960 "We left because we didn't like commercialization. That's not true. We built GPT-3, which was the model that was commercialized. I was involved in commercialization." 00:04:33.960 --> 00:04:54.420 So what was it? "It's more, again, about how do you do it? Civilization is going down this path to very powerful AI. What's the way to do it? That is cautious, straightforward, honest, that builds trust in the organization and in individuals." 00:04:54.420 --> 00:05:04.280 And then a line that's hard to hear as anything but pointed. "How can safety not just be something we say because it helps with recruiting." 00:05:04.280 --> 00:05:28.460 He refused to go further. "I don't want to talk about anyone else's vision," he said. "If you have a vision for how to do it, you should go off and you should do that vision. It is incredibly unproductive to try and argue with someone else's vision. You might think they're not doing it the right way. You might think they're dishonest. Who knows? Maybe you're right, maybe you're not." 00:05:28.460 --> 00:05:37.540 "But what you should do is you should take some people you trust and you should go off together and you should make your vision happen." 00:05:37.540 --> 00:06:09.740 Take some people you trust. Five months after that December meeting, on May twenty-eighth, twenty-twenty-one, a new company announced it had raised a hundred and twenty-four million dollars. Its name was Anthropic. It called itself "an AI safety and research company." It was led, the announcement said, by siblings. Dario Amodei, chief executive. And Daniela Amodei, president, who had been at OpenAI too. 00:06:09.740 --> 00:06:24.440 The announcement listed the research its team had done before. Read the list, and you're reading this season's footnotes. "GPT-3." "Scaling Laws." "AI and Compute." "Learning from Human Preferences." 00:06:24.440 --> 00:06:42.040 And it said what the company would work on: making large AI models more reliable, more interpretable, and, in its words, "building ways to more tightly integrate human feedback into the development and deployment of these systems." 00:06:42.040 --> 00:06:52.100 Human feedback. Hold on to that phrase. Because back at OpenAI, the people who stayed were about to bet the company's next product on it. 00:06:52.100 --> 00:07:13.960 One note before we move on. In later years, others would describe the split in more personal terms, and some of those accounts are contested. OpenAI has its own view of that period. In this episode, we're sticking to what the people involved said on the record, and to what the companies published. 00:07:13.960 --> 00:07:28.380 A week after Amodei's departure was announced, OpenAI had something to show. Tuesday, January fifth, twenty-twenty-one. A blog post called "DALL·E: Creating images from text." 00:07:28.380 --> 00:07:47.660 The name is a joke. Half Salvador Dalí, half WALL-E, the Pixar robot. And the idea is the merge we saw coming last episode. The post calls DALL·E "a twelve-billion parameter version of GPT-3 trained to generate images from text descriptions." 00:07:47.660 --> 00:07:51.480 You type a sentence. It draws a picture. 00:07:51.480 --> 00:08:18.040 The post walks through what that means with examples. One of them is a test of whether the model can keep track of details. The caption: "a hedgehog wearing a red hat, yellow gloves, blue shirt, and green pants." To get that right, the post explains, the model has to attach each color to the right piece of clothing, and each piece of clothing to the hedgehog, "without mixing them up." 00:08:18.040 --> 00:08:36.940 And it was honest that it often couldn't. Add more objects, the post says, and DALL·E "is prone to confusing the associations between the objects and their colors, and the success rate decreases sharply." Say the same thing a different way, and sometimes it gets none of them right. 00:08:36.940 --> 00:09:02.660 But there was one line in that post that sounded like the whole field talking. The researchers had found DALL·E could do certain image-to-image tricks, if you asked it the right way. Things no one had trained it to do. And they wrote: "We did not anticipate that this capability would emerge, and made no modifications to the neural network or training procedure to encourage it." 00:09:02.660 --> 00:09:17.840 We did not anticipate that this would emerge. You'll hear versions of that sentence for the rest of this story. Make the model bigger, give it more, and it starts doing things nobody designed it to do. 00:09:17.840 --> 00:09:40.080 The same day, OpenAI released a companion model called CLIP, which learned to connect pictures and words from hundreds of millions of image and text pairs found online. DALL·E would draw. CLIP would judge which drawings matched the words. Two bets from Amodei's portfolio, words and pictures, had become one. 00:09:40.080 --> 00:09:49.560 And DALL·E itself? The post showed samples. The public didn't get to use it. Not yet. 00:09:49.560 --> 00:10:04.540 Ten weeks after DALL·E, on March sixteenth, twenty-twenty-one, Sam Altman published an essay. Not on OpenAI's site. On a page of its own. The title: "Moore's Law for Everything." 00:10:04.540 --> 00:10:17.540 It's the clearest picture we have of what he thought was coming, written before ChatGPT, before anyone outside the field was paying attention. And it opens like this. 00:10:17.540 --> 00:10:30.980 My work at OpenAI reminds me every day about the magnitude of the socioeconomic change that is coming sooner than most people believe. Software that can think and learn will do more and more of the work that people now do. 00:10:30.980 --> 00:10:40.520 Even more power will shift from labor to capital. If public policy doesn't adapt accordingly, most people will end up worse off than they are today. 00:10:40.520 --> 00:10:45.040 Then he made predictions, with dates attached. 00:10:45.040 --> 00:10:57.740 In the next five years, computer programs that can think will read legal documents and give medical advice. In the next decade, they will do assembly-line work and maybe even become companions. 00:10:57.740 --> 00:11:05.920 Five years from March twenty-twenty-one is March twenty-twenty-six. You can decide for yourself how that one went. 00:11:05.920 --> 00:11:37.300 His argument was that AI would drive the cost of many things toward zero, the way chips got cheaper every couple of years. Imagine, he wrote, a world where "everything, housing, education, food, clothing, etc., became half as expensive every two years." And he proposed a way to share the gains: a fund, paid for by taxing large companies and land, two and a half percent a year, with the money going to every adult citizen. 00:11:37.300 --> 00:11:52.480 He called it the American Equity Fund. He put a number on it. A decade out, he estimated, each of about two hundred fifty million adults in America would get about thirteen thousand five hundred dollars a year. 00:11:52.480 --> 00:12:09.540 You don't have to agree with any of it to notice what it tells you. The head of OpenAI was not thinking about chatbots. He was thinking about the end of most jobs as we know them, and how to keep society stable when that happened. He called the change "unstoppable." 00:12:09.540 --> 00:12:16.647 And look at the thank-you list at the bottom. Ilya Sutskever. Jack Clark. Daniela Amodei. Some of the people on that list were, right about then, leaving to start a different company, built on the idea that the change might be stoppable enough to steer. 00:12:16.647 --> 00:12:23.753 Ten weeks after the essay, on May twenty-sixth, OpenAI announced something that looked like a footnote. A hundred-million-dollar fund to invest in startups building on its technology, with Microsoft among the backers. The OpenAI Startup Fund. Companies it backed would get early access to OpenAI's future systems, and credits on Microsoft's cloud. 00:12:23.753 --> 00:12:30.860 It made sense. If the API was a store, the fund paid people to build shops next door. But hold onto it. Years later, after the events of our next two episodes, the news site Axios reported that the fund, despite its name, was legally owned by Sam Altman personally. OpenAI said the arrangement was temporary and that he had no financial interest in it. A filing a few weeks later showed control had moved to someone else. We'll come back to why that mattered. 00:12:30.860 --> 00:12:48.520 Tuesday, June twenty-ninth, twenty-twenty-one. The announcement doesn't come from OpenAI. It comes from GitHub, the site where millions of programmers store and share their code. GitHub had been owned by Microsoft since twenty-eighteen. 00:12:48.520 --> 00:13:01.520 Its chief executive, Nat Friedman, writes: "Today, we are launching a technical preview of GitHub Copilot, a new AI pair programmer that helps you write better code." 00:13:01.520 --> 00:13:22.400 Pair programmer. In software, that's when two people share one keyboard. One types, the other watches and suggests. Now the second person was a model. As you type, Friedman wrote, Copilot draws on the code you're working on, "suggesting whole lines or entire functions." 00:13:22.400 --> 00:13:48.380 And under the hood: "Developed in collaboration with OpenAI, GitHub Copilot is powered by OpenAI Codex, a new AI system created by OpenAI." Codex, he wrote, was "significantly more capable than GPT-3 in code generation," in part because it had been trained on "a much larger concentration of public source code." 00:13:48.380 --> 00:14:16.680 Public source code. Code that people had shared on GitHub, under all sorts of licenses, for years. That detail would turn into an argument, and eventually a lawsuit, that's outside our story. But notice who was on each side of this launch. OpenAI built the model. Microsoft's subsidiary sold the product. The exclusive partnership from last episode, working exactly as designed. 00:14:16.680 --> 00:14:53.100 A week later, OpenAI's researchers posted the paper behind it: "Evaluating Large Language Models Trained on Code." They'd built a test, a set of a hundred and sixty-four programming problems, each with its own checks, and called it HumanEval. On its first try, Codex solved about twenty-nine percent of them. GPT-3 solved none. And the paper found a trick: let Codex take a hundred tries at each problem, and check which ones pass, and it solved about seventy percent. 00:14:53.100 --> 00:15:11.940 Twenty-nine percent doesn't sound like much. But programmers who tried it described something that felt new. You'd write a comment saying what you wanted, and the code would just appear. Sometimes it was right. Sometimes it was subtly wrong. Either way, people kept using it. 00:15:11.940 --> 00:15:14.894 In August, OpenAI opened Codex to developers through its own API. Greg Brockman described it as the successor to GPT-3. And without anyone saying it out loud, something else had happened. GPT-3 had been a model you talked to through a box on a web page. Codex was a model that sat inside the tool people used all day, at work, getting paid. 00:15:14.894 --> 00:15:17.848 That same summer, OpenAI quietly stopped doing something it had been famous for. Remember the robot hand from episode five, the one that learned to solve a Rubik's Cube? In a podcast interview released that summer, the co-founder who ran robotics, Wojciech Zaremba, said the team had been disbanded. 00:15:17.848 --> 00:15:20.802 His reason was data. A robot has to generate its own experience, slowly, in the physical world. Text was different. "There are actually plenty of domains that are very, very rich with data," he said. "And ultimately that was holding us back in case of robotics." 00:15:20.802 --> 00:15:23.756 The lab that had once tried everything, games, robots, music, was narrowing. Everything was flowing toward the models that learned from the text and code people had already written. That's where the data was. 00:15:23.756 --> 00:15:26.710 In November, OpenAI took down a wall it had built a year and a half earlier. The API waitlist was gone. Anyone in a supported country could sign up and start using GPT-3 that day. The post's subtitle explained why it was now allowed: "Wider availability made possible by safety progress." Tens of thousands of developers, it said, were already building on it. 00:15:26.710 --> 00:15:29.664 Two weeks before that, Microsoft had made its own announcement. At its Ignite conference on November second, it unveiled the Azure OpenAI Service: GPT-3, offered through Microsoft's cloud to its business customers, by invitation at first. The same model, now sold by two companies. One was a lab with a nonprofit on top. The other was Microsoft. 00:15:29.664 --> 00:15:32.618 And in December, the researchers took on GPT-3's oldest problem. Kevin Lacker's problem, from our last episode: it makes things up. Their answer was a project called WebGPT. They taught GPT-3 to use a text-only web browser. It could search, click links, scroll, and quote what it found. And it had to cite its sources. 00:15:32.618 --> 00:15:35.572 The results were good. On questions from the "Explain Like I'm Five" forum on Reddit, people preferred the best model's answers over answers written by the human demonstrators fifty-six percent of the time. 00:15:35.572 --> 00:15:38.526 But the post saw a trap coming. Answers with citations, it said, "are often perceived as having an air of authority, which can obscure the fact that our model still makes basic errors." And then something stranger. A capable enough model, it said, "would cherry-pick sources it expects humans to find convincing, even if they do not reflect a fair assessment of the evidence." There were, it added, already signs of that happening. 00:15:38.526 --> 00:15:41.480 A model learning to be persuasive instead of right. They wrote that down in twenty-twenty-one. 00:15:41.480 --> 00:15:54.360 Thursday, January twenty-seventh, twenty-twenty-two. OpenAI posts something with a title only a researcher could love: "Aligning language models to follow instructions." 00:15:54.360 --> 00:16:08.540 If you listen to one chapter of this episode twice, make it this one. Because this blog post is the hinge between GPT-3, the thing that predicts the next word, and ChatGPT, the thing that answers you. 00:16:08.540 --> 00:16:41.280 The problem, the post says, is that GPT-3 can be "coaxed" into doing useful work, if you engineer the prompt carefully. But it can also produce answers that are "untruthful, toxic, or reflect harmful sentiments." And it explains why, in a sentence that ties this whole season together. "This is in part because GPT-3 is trained to predict the next word on a large dataset of Internet text, rather than to safely perform the language task that the user wants." 00:16:41.280 --> 00:16:46.820 "In other words, these models aren't aligned with their users." 00:16:46.820 --> 00:17:21.680 The fix was the recipe from the summarization work. Here's how the post describes it. Real customers had sent prompts to the API. Human labelers wrote examples of good answers to those prompts. Then the labelers ranked several of the model's answers, best to worst. The paper says the team was about forty contractors, hired through Upwork and Scale AI, chosen after a screening test. A reward model learned to predict those rankings. And GPT-3 was tuned to score well against it. 00:17:21.680 --> 00:17:44.680 They called the results InstructGPT. And here's the number that changed the industry. "Our labelers prefer outputs from our one point three billion parameter InstructGPT model over outputs from a one hundred seventy-five billion parameter GPT-3 model, despite having more than one hundred times fewer parameters." 00:17:44.680 --> 00:18:01.140 A hundred times smaller, and people liked it better. Remember the escalator, and the scaling laws: bigger is predictably better. This was a different lever. Not more size. More human judgment. 00:18:01.140 --> 00:18:22.460 The post said the new models made things up less often, and were a little less toxic. It said the tuning used "less than two percent of the compute and data" that went into training the original model. It described the process as "unlocking" abilities GPT-3 already had, which were just hard to get at. 00:18:22.460 --> 00:18:52.480 And it named a cost. If you tune a model only on what customers ask for, it can get worse at other things. The post called this an "alignment tax." They found a way to shrink it, by mixing in some of the original training data. But the phrase stuck, because it named the fear at the center of this whole story. That safety costs you something, and that someone, someday, would decide not to pay. 00:18:52.480 --> 00:19:04.200 Then the line that matters most for where we're going. "This is the first time our alignment research, which we've been pursuing for several years, has been applied to our product." 00:19:04.200 --> 00:19:24.280 The safety team's technique, the one Paul Christiano's group had been building since the backflipping robot, wasn't a side project anymore. It was the product. These models, the post said, had already been "in beta on the API for more than a year," and were now the default. 00:19:24.280 --> 00:19:47.580 The post was candid about whose preferences were being taught. The models were tuned, it said, to "the preferences of our labelers," and of the researchers who instructed them, and shaped by customers and the company's policies. Those influences, it said, "do not guarantee our models are aligned to the preferences of any broader group." 00:19:47.580 --> 00:20:05.540 And it ended with a warning about the method itself. Teaching a model to follow instructions also makes it better at following bad ones. "Solving this requires our models to refuse certain instructions; doing this reliably is an important open research problem." 00:20:05.540 --> 00:20:17.900 Follow instructions. Refuse some. Make things up less. Admit mistakes. You've heard that list already, in this episode's first minute. 00:20:17.900 --> 00:20:27.680 Five weeks later, the full paper appeared. First author: Long Ouyang. Just as we said he'd be. 00:20:27.680 --> 00:20:39.880 Spring twenty-twenty-two. OpenAI is no longer only a lab that writes papers. It is shipping, regularly, and the people who ship are moving up. 00:20:39.880 --> 00:21:04.540 On April sixth, it announced DALL·E 2. Sharper pictures, at four times the resolution of the first one, and the ability to edit an existing image with a sentence. This time, the public could sign up, through a waitlist, as a research preview. It went out with a written document on its risks and limits, a practice OpenAI would make standard. 00:21:04.540 --> 00:21:19.940 In May, Mira Murati, the engineer who'd run the API launch two years earlier, was named chief technology officer. She'd oversee DALL·E, Codex, and the research that was about to become ChatGPT. 00:21:19.940 --> 00:21:43.100 Remember her title from the end of our last episode. Research, product, and partnerships, in one job. Now the research and the product weren't just in one title. They were one pipeline. A model got trained. It got tuned on human feedback. It went to customers through the API. Their prompts became the next round of training data. 00:21:43.100 --> 00:21:44.087 That's the loop. And by the fall, OpenAI had a model in the GPT-three-point-five series, finished training early that year, that was better than anything it had released. The question was how to put it in front of people. 00:21:44.087 --> 00:21:45.073 That same spring, the escalator itself got a correction, and it came from a rival. Remember the scaling laws from episode six, the OpenAI paper that said bigger models get predictably better? In March twenty-twenty-two, researchers at Google's DeepMind published a paper with a pointed finding. "We find that current large language models are significantly undertrained." 00:21:45.073 --> 00:21:46.060 The labs, it argued, had been making their models bigger without feeding them enough to read. For the best use of a given amount of computing, model size and training data should grow together: double one, double the other. DeepMind tested it by training a model called Chinchilla. Seventy billion parameters, a quarter the size of its own previous model, trained on about four times as much text. And Chinchilla, the paper said, "uniformly and significantly outperforms" much larger models, GPT-3 among them. 00:21:46.060 --> 00:21:47.047 The escalator was real. But the steps weren't only size. They were size and data, together. And the data, mostly, was us. 00:21:47.047 --> 00:21:48.033 Summer twenty-twenty-two is when this stopped being a story about one lab. 00:21:48.033 --> 00:21:49.020 In June, a Google engineer named Blake Lemoine told The Washington Post he believed Google's chatbot, LaMDA, had become sentient. Google put him on paid leave for sharing confidential conversations. And the company told the Post he'd been told "there was no evidence that LaMDA was sentient (and lots of evidence against it)." 00:21:49.020 --> 00:21:50.007 Most researchers agreed with Google. But think about what the story showed. Google had a chatbot good enough that one of its own engineers was moved by it. And Google was keeping it inside. Five months later, OpenAI would make a different choice. 00:21:50.007 --> 00:21:50.993 In June, GitHub Copilot came out of preview. Ten dollars a month for individual programmers. The pair programmer was now a product with a price. 00:21:50.993 --> 00:21:51.980 In August, a startup called Stability AI released Stable Diffusion, an image generator in the spirit of DALL·E 2. With one big difference. Stability released the model itself. Anyone could download it and run it on a decent home computer, with no waitlist and no content filter they didn't choose. 00:21:51.980 --> 00:21:52.967 Remember episode one. A lab founded on the idea that this technology should be open. In twenty-twenty-two, the most open image model in the world came from somebody else. 00:21:52.967 --> 00:21:53.953 OpenAI's answer, on September twenty-eighth, was to take away its own waitlist. DALL·E was open to anyone who signed up. The post gave the numbers: more than one point five million users, "actively creating over two million images a day." The company said the months of gradual release had let it make its filters "more robust at rejecting attempts to generate sexual, violent and other content." 00:21:53.953 --> 00:21:54.940 A week earlier, OpenAI had released something that got much less attention, and that matters more to this podcast than you might think. It was called Whisper, a speech-recognition model trained, the post said, on six hundred and eighty thousand hours of audio collected from the web. It "approaches human level robustness and accuracy on English speech recognition." And this time OpenAI open-sourced it. The model, and the code. 00:21:54.940 --> 00:21:55.927 Full disclosure. When we check each episode of this show against its script, the software that listens back is a version of Whisper. It's listening right now. 00:21:55.927 --> 00:21:56.913 So by the fall of twenty-twenty-two, OpenAI had an image model anyone could use, a speech model anyone could download, a code model selling by subscription, and an API anyone could sign up for. And it had one more thing, the thing Murati's pipeline had been building toward. A model trained on human feedback, good at conversation, sitting on the shelf. 00:21:56.913 --> 00:21:57.900 Two weeks before ChatGPT, there was a warning shot, and it came from Meta. On November fifteenth, Meta's AI lab released a public demo of a model called Galactica, trained on scientific papers and pitched as a tool to help write science. 00:21:57.900 --> 00:21:58.887 Within hours, people were posting what it produced. Fluent, authoritative-looking scientific text that was simply made up. Fake papers, attributed to real authors. MIT Technology Review's headline put the result plainly: the model "only survived three days online." Meta paused the demo on November seventeenth. 00:21:58.887 --> 00:21:59.873 Meta's chief AI scientist, Yann LeCun, wasn't happy about it. "It's no longer possible to have some fun by casually misusing it. Happy?" 00:21:59.873 --> 00:22:00.860 So in mid-November twenty-twenty-two, here's what the industry had seen. A Google engineer convinced a chatbot was alive. A science model pulled in three days for making things up. And sitting at OpenAI, a chatbot that also made things up, and whose own launch post would say so in its first list of limitations. 00:22:00.860 --> 00:22:08.060 Here's how ChatGPT was made, in OpenAI's own description, from that November thirtieth post. 00:22:08.060 --> 00:22:27.200 It used the InstructGPT recipe, "but with slight differences in the data collection setup." The first step is the one I find most striking. Human trainers wrote conversations "in which they played both sides — the user and an AI assistant." 00:22:27.200 --> 00:22:46.140 Picture that. People sitting at screens, writing a question, and then writing the answer they wished a machine would give. Then the next question. Then the next answer. Thousands of imaginary conversations, written by people pretending to be the thing they were building. 00:22:46.140 --> 00:23:06.780 Then the ranking. The trainers chatted with the model, and for a given reply, the system generated several alternatives, and the trainers ranked them. A reward model learned from those rankings. The model was tuned against it. And they did it, the post says, over "several iterations." 00:23:06.780 --> 00:23:24.480 Underneath it all was a model in the GPT-three-point-five series. And the post adds a detail we've been tracking since the billion-dollar deal. "ChatGPT and GPT-3.5 were trained on an Azure AI supercomputing infrastructure." 00:23:24.480 --> 00:23:31.660 The post was unusually frank about what was still broken. Listen to the limitations as they wrote them. 00:23:31.660 --> 00:23:51.180 It "sometimes writes plausible-sounding but incorrect or nonsensical answers." Fixing that is hard, they explained, partly because during training, "there's currently no source of truth." And if you train it to be more careful, it starts refusing questions it could have answered. 00:23:51.180 --> 00:24:01.820 It's "sensitive to tweaks to the input phrasing." Ask one way, it says it doesn't know. Ask slightly differently, and it answers correctly. 00:24:01.820 --> 00:24:18.060 It's "often excessively verbose," and it "overuses certain phrases, such as restating that it's a language model trained by OpenAI." The reason, they said: trainers "prefer longer answers that look more comprehensive." 00:24:18.060 --> 00:24:31.060 It usually guesses what you meant, instead of asking. And while they'd worked to make it refuse harmful requests, it "will sometimes respond to harmful instructions or exhibit biased behavior." 00:24:31.060 --> 00:24:43.560 Reading that list now, it's almost a map of every argument about chatbots that followed. It was all there, on day one, in OpenAI's own words. 00:24:43.560 --> 00:25:03.520 And the post put the launch in a frame the company would use again and again. "Iterative deployment." Release something, learn from how people use it, improve, release again. They even offered a contest: up to five hundred dollars in API credits for the best feedback on problems. 00:25:03.520 --> 00:25:25.440 Mira Murati described the thinking, a few months later, in an interview. "The underlying technology had been around for months," she said. "We had high confidence in the limitations of the model from customers that had already been using it via an API. But we made a few changes on top of the base model. We adapted it to dialog." 00:25:25.440 --> 00:25:46.300 Why release it at all? She said they'd been exploring it internally "with a small, trusted group," and "we realized the bottleneck was getting more information and getting more data from people. We wanted to expand it to more people out there in what we call a research preview, not a product." 00:25:46.300 --> 00:25:51.680 A research preview. Not a product. 00:25:51.680 --> 00:26:00.640 So, Wednesday, November thirtieth. The research preview goes up. Free. A text box and a reply. 00:26:00.640 --> 00:26:07.360 What happened next has been told so many times it's turned into folklore, so let's stick to the record. 00:26:07.360 --> 00:26:13.620 The following Monday, December fifth, Sam Altman posted on Twitter. 00:26:13.620 --> 00:26:19.360 ChatGPT launched on wednesday. today it crossed 1 million users! 00:26:19.360 --> 00:26:34.160 Five days. And the same day, in another post, he said the other thing out loud. Every one of those conversations ran on those Azure supercomputers. Every reply cost money. 00:26:34.160 --> 00:26:40.540 we will have to monetize it somehow at some point; the compute costs are eye-watering. 00:26:40.540 --> 00:27:07.860 Eye-watering. Remember the founding pitch, seven years earlier. A nonprofit, free from the need to generate a financial return. Remember Brockman's API post, two years earlier: a revenue source "to help us cover costs." Now the most popular thing OpenAI had ever made was free, and every minute it was more popular, it cost more. 00:27:07.860 --> 00:27:09.549 Mira Murati put it simply later. "We didn't anticipate the degree to which people would be so captivated with talking to an AI system. It was just a research preview." 00:27:09.549 --> 00:27:11.237 Five days after the million-user post, on December tenth, Altman posted something that read almost like a warning label. 00:27:11.237 --> 00:27:12.926 ChatGPT is incredibly limited, but good enough at some things to create a misleading impression of greatness. it's a mistake to be relying on it for anything important right now. 00:27:12.926 --> 00:27:14.614 it's a preview of progress; we have lots of work to do on robustness and truthfulness. 00:27:14.614 --> 00:27:16.303 A misleading impression of greatness. From the man who'd just watched a million people sign up in five days. 00:27:16.303 --> 00:27:17.991 And at Google, which had built the chatbot Blake Lemoine believed was alive, employees asked at an all-hands meeting why they hadn't released something like it. CNBC reported the answer from Google's leadership: a company Google's size carried far more "reputational risk" than a startup, and so it was moving "more conservatively." 00:27:17.991 --> 00:27:19.680 More conservatively. Hold that thought until the first minutes of our next episode. 00:27:19.680 --> 00:27:25.447 We did not anticipate. The same words, almost, as the DALL·E post. Except this time, the thing that emerged wasn't a capability inside the model. It was what happened outside it, when millions of people got to talk to one. 00:27:25.447 --> 00:27:31.213 Two weeks after the launch, on December fifteenth, the people who'd gone off together published a paper of their own. Anthropic called it "Constitutional AI." Its first line asked a question OpenAI would ask the following summer. "As AI systems become more capable, we would like to enlist their help to supervise other AIs." 00:27:31.213 --> 00:27:36.980 Instead of relying only on thousands of human ratings, Anthropic gave a model a short list of written principles, a constitution, and trained it to critique and revise its own answers against them. Human feedback, partly replaced by the model's own judgment, guided by rules people wrote down. Same problem. Different bet. And now two companies, run by people who'd worked together two years earlier, were placing them. 00:27:36.980 --> 00:28:08.760 So here's where the quiet build ends. The people who thought safety needed a different home had gone off together, as Dario Amodei put it, to make their own vision happen. The people who stayed had taken a safety technique, learning from human feedback, and turned it into the most useful product the company had ever made. Microsoft's cloud ran it. Microsoft's money had paid for the machines. And on the other side of a text box, the world was typing. 00:28:08.760 --> 00:28:34.180 A year later, Sam Altman would call this a low-key research preview, and add, quote, "That went pretty well." He wasn't wrong. But "pretty well" was about to cost a great deal of money, set off a race with the biggest companies on earth, and put an enormous amount of pressure on the six people who sat on a nonprofit board. 00:28:34.180 --> 00:28:42.180 Next time, on The Role Models. The Moment.