EP 115
Combinatorial Play Has Begun
The State of the AI Industry Seen at the All-In Summit 0:00
Seungjoon Choi The day we’re recording this is the morning of Saturday, September 19th. Chester is away on a business trip, so for the first time in a while, just the two of us decided to host the show. Jonghyun, how have you been?
Jonghyun Park Yes, I’ve been keeping up with things diligently. The fruit fly thing and Jev and all that were huge topics, and so much is coming out that I can’t keep up with all of it, so for now I’m focusing on the things I’m interested in. This week I’ve been watching the All-In Summit closely. The videos are being uploaded right now, and since it’s an event where all the famous people turn out, I was watching closely to see what people were saying there. There’s a podcast channel called the All-In Podcast, and a lot of famous people in the US appear on it. The hosts are famous people too, and it’s a bit American, so the vibe is different. They seem to swear as well. Anyway, it’s one of those podcasts where they talk in a very forceful way, and it’s incredibly popular. They hold an event like that once a year, and I heard the tickets cost a whopping 10 million won—around $7,500. Even at 10 million won, an awful lot of people attended. I kept thinking, wow, there really are so many people who want to hear that kind of talk.
Seungjoon Choi I see. I only watched one of them, so I’m curious which ones you watched and what you thought.
Jonghyun Park First, I’ve watched pretty much everything that’s been posted. And then, out of those, I pulled out all the ones that people watching our channel, the AI Frontier channel here, might be interested in. So the content I pulled out is as follows: there’s Jensen Huang’s session, SpaceX’s Elon Musk and the president, I think? Anyway, someone who’s worked there from the very earliest days right up until now— well, in leadership. Leadership, and Satya Nadella, JD Vance, and then Brad Gerstner. I actually learned about him for the first time, he’s from Altimeter Capital, so it was a session that analyzed the economic aspects well, which is why I brought it. There were several others besides these. There was journalism and so on, but in any case, they were sessions that talked a lot about the direction the US is headed. And what was interesting was, in the middle of it Trump called in, so Trump suddenly showed up too.
Seungjoon Choi During Jensen’s session? Then if you were to lead with the key takeaways up front?
Leaders’ Diverging Positions on AI Pacing 2:12
Jonghyun Park The parts that every session talked about in common were, broadly, the discussion about AI pacing—it’s too dangerous, that sort of thing is a big topic right now. And then the discussion of how much economic effect this actually has and what will happen going forward, and China and open-source models, and what the US should do— broadly, that’s the range of topics that came up. Not all the panelists held the same view; it was a session where everyone shared a variety of different perspectives with each other.
Seungjoon Choi Then why don’t you walk us through them.
Jonghyun Park First, on the pacing discussion, Dario Amodei said something along the lines of we need to stop. And in response to that, various leaders gave their own opinions, but broadly, Trump and Jensen, and then the panelists at the All-In Summit, including David Sacks, said why would we stop this, we can’t stop at this point anyway, and especially in the competition with China, stopping is effectively impossible, is what they seem to think, and they’re saying we have to push forward. Surprisingly, when Dario Amodei said we should stop, Sam Altman and Elon Musk agreed. So people wonder whether those leaders really are seeing dangerous things from the inside— that’s what people have been thinking, and Elon Musk, similarly in any case, said that in some way, checks and audits— we do need to have something like that in place— that’s the kind of thing he said. So I think the views were somewhat different.
Seungjoon Choi I think Elon Musk has always been like that. Even putting money into founding OpenAI, was to keep Demis Hassabis in check, because he’d made a game like Evil Genius— I remember that being the story.
Jonghyun Park He said it’s like summoning a demon, I think he said something like that, so surprisingly, it seems they were thinking about those things. Seungjoon, what do you think?
Jensen Huang’s Strategy and RSI Pacing 4:04
Seungjoon Choi I only watched the Jensen episode. But I thought Jensen is a real strategist. He speaks without holding back, and most of these people are VCs, right? And besides this, at Salesforce’s Dreamforce event, I think, I watched what he said there too, and I thought there really is something different about Jensen. Obviously, he would have said things that serve his own interests, not his personal interests, but things that serve the interests of the ecosystem itself, and I think he said forcefully that it makes no sense.
Jonghyun Park I was curious about your personal view, Seungjoon. Whether you think we really need to do pacing or not.
Seungjoon Choi I think pacing needs to be done properly. From a user’s standpoint, if you said stop right now, that would be a real shame. But what’s being discussed now is pacing RSI, it may not be about regulating development at the current level. It’s that we need to be extremely careful about entering RSI, not that we should block the trend of moving forward at the current frequency and rhythm—that’s how I felt. I do want to keep seeing progress. I just hope it doesn’t become dangerous. I hope we don’t cross a truly irreversible line, but short of that, I’m on the side of wanting to keep seeing progress.
Open Source and Mutual Auditing as Safety Alternatives 5:28
Jonghyun Park Pacing isn’t something that simply happens because we say let’s stop. The very fact that we’re having this discussion means we’re trying to find a method, right? If I had to pick something comparable in computing, I think it was security-related issues. If you think of Linux as open-source and Windows as closed-source, surprisingly, open-source inevitably ends up being better in terms of security. Because everyone looks into it and works together to fix the problems, from that perspective, opening everything up together might be the path where we can all look after safety together—that was the discussion. On a similar note, Elon Musk said something like this. By auditing each other and testing each other before something gets released into the world, wouldn’t that be much safer? That’s what he said, and to some extent I think we’ll be able to find those kinds of approaches. So I came to think that having these discussions out in the open, as much as possible, is probably the right thing to do. Even without deliberately slowing down, wouldn’t we be able to find all the necessary methods anyway? Moving right along, I’ll keep this part brief, and I’d like to talk a bit about the economic side of things. Ultimately, including Korea’s memory companies, an enormous amount of money is changing hands, and people are asking: is this a bubble, how long will it keep climbing? Everyone’s a bit scared about that. I’m quite interested in that topic myself, so I took a close look at it. I put together a list of the numbers worth paying attention to, and we’ll go through them one by one. First, the thing everyone worries about, one of them, is that the investment is too large. They’re sourcing all this enormous money from somewhere and building things, including data centers, starting with the money flowing to NVIDIA, and the question is whether that investment can actually be recouped. That was the discussion around this. So the light blue, the blue numbers here are the investment volume. Naturally, investment is rising first, and then this orange is the money the labs are currently earning— OpenAI, Anthropic, SpaceX, the money being earned at places like that is climbing right up behind it, but in any case, as much as was invested— That’s true. Whether you can recoup as much as you invested is ultimately the key metric, and if that’s impossible, all the investment goes bust. But what’s certain is that so far, it is indeed growing incredibly fast. It went from $7B to $200B in just two years, so if it keeps going like this, it should be fine, but how far it will catch up, nobody knows.
The Capex Curve That Raises Questions About Return on Investment 6:32
Seungjoon Choi Right now, the investment itself, from $241B in 2024—
Jonghyun Park Yes, up to here, you could say it’s almost certain. Through next year, you could say it’s pretty much locked in.
Seungjoon Choi Counting through next year, it’s roughly a 4x to 5x increase, that’s how much it grew over two years.
Jonghyun Park It’s all flowed into the semiconductor companies’ earnings.
Seungjoon Choi From what I’ve seen, I vaguely recall reading that OpenAI and Anthropic are purchasing memory directly. I’m not sure, but at any rate, rather than buying finished chips with memory already in them, I think I saw news somewhere that they’re purchasing the memory directly. I’m not certain though.
Jonghyun Park In any case, broadly speaking, this has to keep climbing up this much, and the demand would be the money that companies or individuals like us spend as we all use AI. So on this point, everyone seems to be taking a rosy view, thinking demand will absolutely increase. So the question of whether this is a bubble, whether that rise is a bubble— everyone keeps talking about the exact same topic all over the place, and first off, Brad Gerstner says it’s not a bubble. Because he says this isn’t multiple expansion— profits are actually increasing, and the index is rising accordingly. That’s his argument. So he shows it with numbers. What this multiple looks at is: how much the Nasdaq rose relative to actual earnings, and at how many times earnings the index or stock price is being maintained— and he says it has actually decreased. I think you could call this about the same level. It rose that much because they’re actually earning a lot of money, and he compares it with the internet dot-com bubble era of the past, where back then, before they were earning any money, stock prices shot up first, he says. Assuming they’d make huge money in the future, priced at 100x relative to what they were actually earning, and in the end, that’s what burst. That’s how it was back then, but he says now is different. So he’s saying it probably isn’t a bubble. In the case of someone more moderate like Satya Nadella, he said something along the lines of: that hasn’t been proven. The money being made is ultimately all from investment, and the invested money all flowed to NVIDIA and the semiconductor companies and such, so their profits went up— but rather, GDP actually has to rise for it not to be a bubble, is what he said. So will GDP really rise? Anthropic also recently put out an actual economic modeling report. How much will the economy and GDP grow because of AI— they laid out scenarios and forecasts on that question, modeled it, and released a report where we can plug our own numbers into the prediction model, and it turns out that things like the internet and railroads didn’t spike GDP as much as you’d think. Looking at the historical numbers—I had naturally assumed something like the internet would be an event that achieved enormous growth and transformed the whole landscape. Turns out it wasn’t as much as I thought. So the thinking seems to be that GDP might grow just a bit more than usual—if normal growth is around 3–5%, then maybe 8% growth, something at about that level. And even to reach that, AI has to be used an enormous amount. In any case, most of this index comes from money earned by semiconductor companies. So almost all of it is the semiconductor companies’ gains, soon led to a rise in the index, and that means that right now, investment money is flowing into semiconductor companies — that’s how you can read it.
AI’s Revenue Structure Unlike the Dot-Com Bubble 8:53
Satya Nadella Says It Must Be Proven Through GDP Growth 9:58
Seungjoon Choi That’s their take, I suppose. The kind of thing the All-In crowd says. But the overall message leans a bit more toward it not being a bubble.
Jonghyun Park If we break down the arguments, the numbers are researched figures, so there are projections baked into them too. These projections are along the lines of “someone else projected this somewhere,” so they could be somewhat exaggerated, but in any case, it doesn’t look like they’re faking the projections themselves. That said, looking at these numbers and interpreting them as “the bubble won’t burst” is just these people’s opinion. But whenever a bubble does burst, there’s always logic and supporting numbers specific to that moment too. So whether this really leads to that conclusion is something each person has to judge. And among the reasons they were able to sustain this growth, the very first is obviously that OpenAI and Anthropic started making money so incredibly fast. The expectation that demand will follow is all rooted here, and they say this is far faster than the timeline on which Google or Meta made money in the past. So with Anthropic, we’re seeing revenue suddenly rack up in just a year in an absolutely absurd way, and everyone seemed to think it would keep climbing like that going forward. This capex is all about building out facilities, and in Jensen’s case, with these facilities he’s obviously the one making money here. He makes money selling NVIDIA GPUs that go into these facilities, and thinking up and down the stack, extending downward to memory, TSMC, and places like that, everyone cooperating together as much as possible, and then selling that to the neoclouds. Selling to hyperscalers, neoclouds, and the like, and since demand is so huge, they’re just selling as fast as they can make them. And in Microsoft’s case, they’re playing every role across all these verticals. So Microsoft is a hyperscaler through Azure, and they actually sell Windows and the app services that run on top of it as well, and they’re even building models themselves. Recently models like MAI-Thinking have been coming out one after another, so they’re in a position to observe both up and down the stack well, and the panelists ask: Microsoft — how are you thinking about it from where you sit? But what they talk about here is if you look up and down the stack: you build data centers, you lease out those data centers and sell compute, you use it yourself when you need it, and if you really need more, you rent someone else’s, so all of these things have to be able to run flexibly, and they think that for this to work well, the ecosystem has to be enormous. You obviously can’t depend on NVIDIA alone, and for models you can’t depend on OpenAI alone, there have to be diverse models, and back in the old Windows days, when Linux came along they thought Windows would be threatened, but in reality, running Linux alongside it grew the ecosystem and they actually saw Windows sell even more as a result. So in the end, it seemed their point was that expanding the ecosystem matters most.
OpenAI and Anthropic’s Steep Revenue Growth 12:22
The AI Ecosystem That Runs Through Every Vertical 12:51
Seungjoon Choi They say places like cafés, instead of doing well alone, do better when they form a cluster together.
Jonghyun Park And one of the things I found fascinating — someone came out from SpaceX, and what they said was, You know how SpaceX merged with xAI and then also bought Crusoe and so on. And then was it Colossus? That massive data center— how Anthropic leased the whole thing, I’d seen a lot of talk about that,
SpaceX’s Compute Leasing Revenue Surpassing That of the Space Industry 14:54
Seungjoon Choi Right. That happened. Because Colossus 1 wasn’t being used.
Jonghyun Park They say the business of leasing out those computers is already generating an enormous amount of revenue. So on a revenue basis, the AI business is already making as much money as the space business, and with demand growing with no end in sight, they say it just keeps making a lot of money. So the impression I got was— he talked about it almost with a sense of disillusionment. He’s been with SpaceX from the very earliest days, so he keeps talking about what they did to win space business, what they did with rockets, that kind of thing. So he talks about those days, and then leasing out computers— they’d barely started and it suddenly took off so well, and that it’s making this much money, compared to what he’s devoted his whole life to, succeeding so incredibly fast, he said it felt almost embarrassing.
Seungjoon Choi By the same token, Google must be making a ton too.
Jonghyun Park Right now Gemini— maybe that changes when 4 comes out, but currently, compared to OpenAI’s and Anthropic’s models, it’s not selling all that well, and yet they say Google is making a lot of money anyway. Because of compute.
Seungjoon Choi Satya Nadella back in 2023 was saying he’d make the dinosaur dance and going all out like that—that mood isn’t there anymore, but anyway, Microsoft, another dinosaur, still has staying power.
Jonghyun Park The very status of being a hyperscaler seems to play a major role in the broader ecosystem of this market right now. For one thing, they have a lot of money. This is a period where capital matters, and when you think about who makes this money, and who’s ultimately going to make it, right now, earning as much as is being invested, semiconductor companies are all making a lot, but right now OpenAI and Anthropic are starting to make real money right away, and everyone seems to think they’ll make a lot going forward too. Starting from the model side that produces the tokens, all the way down to the GPUs and the data center operators. But take OpenAI’s recent Astra model— we say it’s great, but it’s $50 per 1M output tokens. And the newly released DeepSeek is supposedly 99% cheaper. Is this V4.1, maybe? Out of all the tasks where we spend tokens, for most of them this level is enough to get a lot of work done, so people seem to think all the easy work will shift over to this side. And that’s true even for me. I edit these videos, and beyond editing, I do clipping and I’m building a video processing product, and I use the most expensive model only for the most critical judgments and use a cheap model for every other judgment. So I ran tests swapping models around. I tested just one piece of logic back in the 5.6 days, and using Sol there and Luna for everything else gave the same result. So looking at it that way, in terms of raw token usage, Luna is far higher. That’s how I’m using it, and with OpenAI’s expensive pricing, people are researching it since it just came out, trying it this way and that, which is why they’re paying for now, but nobody seems to think it can hold up over the long term. So then, where does all that money have to go? Satya Nadella says applications, of course. Here they’re treating applications as the layer below. He says it has to go to applications. And on this too he brings up their own past. Because there were open-source DBs like Postgres, in the end, all these DBs were able to compete— that’s the point he was making, and he seems to think it makes no sense for the models to take everything and make all the money. Historically, it’s always been that way, too. So NVIDIA acquiring Hugging Face this time and expanding their inference infrastructure business— the point is that NVIDIA is doing all that in the exact same context. So Jensen said, for now we’ll only move up as far as we need to, and we won’t go all the way up the stack— that’s what he said, anyway.
Price Collapse and Value Moving Down to the Application Layer 16:54
Seungjoon Choi Right. This is something Chester has often talked about, too— “where do we run to?” Since the top labs still can’t eat everything, there’ll be something left in the end, and this seems to align with that logic, but what people worry about is that once you acquire overwhelmingly powerful intelligence, can’t you just take it all?
Jonghyun Park Back when a single model was dominating, those things felt a bit scarier, but China is catching up better than expected, so that worry doesn’t seem to be as strong as before. If anything, people fear things like human extinction more. Anyway, as competition among models intensifies, margins there get surprisingly thin, and then model prices will come down, so the focus will shift toward what you can do with those cheap models— on that point there was near-unanimous agreement, among all the panelists.
Seungjoon Choi But the top labs don’t only release big models, Anyway, something like Luna is a good RSI target, right? It lets the big model keep training the small model. And in fact you can do far more runs, so the thought crosses my mind that the top labs will keep making small models, too.
The Real Bottlenecks: Power and Data Centers 20:39
Jonghyun Park So that was about where the money gets made. Next, when asked where the real bottleneck will be, most people seem to think it’s power and data centers. Ultimately this is where scale-up doesn’t work well. That’s why Elon Musk is doing this thing called Terafab. He’s actually trying to produce semiconductors directly, too. When asked why he’s doing this, he said, “A dream told me to do it.” He says things like that, but what he’s ultimately worried about is this. Right now they’re getting chips made at TSMC in Taiwan, but couldn’t that supply get cut off at some point? He really seems to be thinking about a conflict scenario with China. Thinking that way, he seems to want to prepare for everything in advance, taking on every layer himself, and what many of the panelists agree on is that Elon Musk has shown far too many times that when he says he’ll do something, he does it, so he probably won’t stop and will keep at it until it works— that seems to be everyone’s view. Right. It seems everyone who’s worked with him has had the experience of trying to talk him out of it and failing. They talk about it almost resignedly. They say, “You just can’t stop him, that’s how he is.” and apparently China’s power generation alone is three times that of the US. Twenty years ago they were even, they said. Twenty years ago China started to pull ahead, and since then the gap has widened that much by now. So there was a lot of venting about it: “Why couldn’t we expand power generation this much here?” “The environmental movement and regulations are the problem—this shouldn’t be allowed.” In any case, these are Republican-leaning people, They talked a lot about that — that electricity is too expensive, compared to China. And then, when they try to build a data center somewhere, everyone opposes it. They say the opposition is huge. And once word gets out, land prices shoot up like crazy. Anyway, unlike factories, data centers don’t bring jobs, so people seem to hate it when they move in. Korea seems to think similarly, that it needs to help revitalize the local economy. So Microsoft brought some data on this. Twenty years ago, in a place called Quincy in Washington State, they built a data center, and over 20 years, tax revenue increased twelvefold. Because of that, they keep doing maintenance, and they keep building things, so it’s not that there are no jobs, they said. But 1,200 jobs doesn’t seem like all that many, anyway, so this rural town’s growth rate is apparently higher than Seattle’s. So the local community, at this point, is said to be very happy about the very fact that a data center is in their town. And yet these things don’t seem to be well known. Because of this perception, in some regions everyone opposes it, so building a data center doesn’t get permitted, and these things inevitably get delayed, And those who are frustrated by this, who want to scale up, voice their complaints, saying that through cases like these, shouldn’t we change public opinion? That seems to be the kind of thing they’re saying. And SpaceX, as you’d expect, is trying to do it in space. The logic is simple. Land prices, permits, all of it is slow and difficult. Just launching into space might actually be faster. Cooling is free. Electricity is free, that’s the idea.
A New Option: Space Data Centers 23:49
Seungjoon Choi Cooling isn’t really free, though, and there does seem to be a lot of research on that, but anyway, Elon is trying it, Google is trying it, everyone seems interested in space data centers.
Jonghyun Park Since you’re sending radiant heat to the coldest place there is, space, well, if anyone is going to do this, it’ll be SpaceX and xAI. They actually seem to be trying it for real. I don’t know if it’ll work out, but it might — that’s what I think. So anyway, this compute is going to keep increasing, that’s certain for the next few years, since all the investment is flowing in like this, and it’s not just the US — Korea is trying to do this too. Korea also seems to be trying to expand it.
Seungjoon Choi Where was it? Jeonnam, maybe? It’s fuzzy, but something was being discussed, and I vaguely recall hearing SK said they’d do it. That the power needs to be on the order of several gigawatts, that kind of talk —
Jonghyun Park The Quincy I mentioned earlier, they gave the numbers for that one, and I think it was a data center at around 0.5 GW. That’s a place that has been gradually expanding ever since, and it probably wasn’t built for today’s AI era. But the permits were already in place, and places like that are set up so electricity can be supplied, so it seems a lot of expansion is happening there. Next, everyone is interested in this. The point of consensus is that whoever wins at AI wins, Trump says this kind of thing too, and a lot of people seem to be saying it. So pacing is not an easy thing to do. Because if only the US gets together and paces, the view has taken hold that China essentially has no intention of pacing. In that case, they seem to think you can’t win. So they seem to think you can’t stop, and Jensen said something like this: the Industrial Revolution itself all came out of Europe, but who actually reaped the benefits of the Industrial Revolution? It was the US. So how do you make good use of something like that — That’s what matters, he says. There was one more talk here, and it was about railroads. Before you knew it, China managed to blanket that vast land with high-speed rail, but in California, all those projects apparently failed. So there was a session exposing the corruption and graft around railroads, asking why California, even though it tried to build rail, isn’t managing to do it properly. Ultimately, based on things like competition with China, they seem to be trying to stir up America’s competitive spirit and rally public sentiment to move forward together. Ultimately, if you liken AI to railroads, how can we take this AI and make better use of it across society as a whole? So which model is better, that’s not what matters — that’s what Jensen Huang says, though from NVIDIA’s perspective, of course that’s the only thing he can say. But almost everyone seems to be saying that. What does it matter where the model originated? Of course it matters, but right now isn’t the time for us to worry about that. We have to win the competition right now. Everyone seems to be saying things like that. They actually talked about open models and closed models, and the leaders here all seem to take it for granted that you need both. They said history proves that. And apparently as much as $400B has recently been invested in AI-native startups. They said the scale is around that much, and almost all of them use open models. The reasons for using open models will all be different for each of them — there are cost issues, and security issues too — but in any case, everyone is using open models a lot. So for the ecosystem, this is an indispensable component, that’s what they seem to think. If you ask who it’s good or bad for, obviously, apart from OpenAI and Anthropic, it would be good for everyone else. So then, how far has China actually come? People ask that, and across various aspects, they seem to think China has caught up. They no longer think the US has any particular edge. In the case of semiconductor production, everyone thinks there’s still a bit of a gap with China, but from Jensen Huang’s perspective, this is a gap that will be closed within about four years, and a gap that closes in two to three years, when you look at it on a large scale of 10 or 20 years, is effectively the same as no gap at all — that’s how he put it. Anyway, is that because the semiconductor business has a somewhat long cycle? Jensen was talking along those lines.
U.S.–China Competition Compared to the Industrial Revolution and Railroads 25:26
Open Models as an Essential Condition of the Ecosystem and China’s Pursuit 27:25
Seungjoon Choi In two years, when models these days come out every two or three months, how many is that in two years? If you divide a year by two months, that’s six, so anyway, close to ten could come out.
Jonghyun Park Semiconductors can’t move forward that quickly and dramatically, and maybe that’s why he thinks this way. The fact that Elon Musk views Terafab positively and keeps pushing hard on it could, in a way, be for a similar reason. Then there’s power — this is what I mentioned earlier. With power, the US actually hasn’t grown its national electricity production much since 2005, while China has grown it enormously, so on this front they seem to think the US is the one falling behind. And Zhipu recently raised $5B and seems to have clearly stated they’ll pursue RSI. They seem to have declared it and are pushing ahead. So while talking about this, David Sacks said the same kind of thing — that it won’t be easy, that China is sprinting ahead, and then when it comes to RSI itself, People’s views differ a bit. From the perspective of the general public like us, isn’t it a bit dangerous? Isn’t that the singularity we talk about—isn’t that RSI? That’s what we think, but Jensen was basically saying, of course we’re going to do it, that’s how he talked about it. Everyone did expect that if it were possible, people would do it. This has been around for decades, going back to science fiction novels, it’s an idea anyone can think of, but what the outcome will be is something we don’t know. So the conclusion is that, including Jensen, nearly all the key leaders here at the All-In Summit think we have to run. Pacing won’t be easy. At the frontier labs, Dario Amodei and Sam Altman may be saying we should pace ourselves, but even if they say that, everyone else in a different position in the ecosystem seems to think it can’t be stopped, and ultimately, the reason I went through the opinions of the various opinion leaders at this Summit was because I was curious about where the thinking is converging. Because this ecosystem will probably all move in that direction. In the end, I don’t think pacing will be possible. If someone says they’ll pace themselves, other players will jump out and sprint.
The Conclusion That Pacing Ultimately Won’t Happen 30:27
Seungjoon Choi We really can’t see an inch ahead of what will happen, but let’s head toward wrapping up— is there anything left?
Jonghyun Park As for what will happen: can we join hands with China and pace together? The answer is absolutely not. If you ask whether we should at least somehow discuss safety together, that might be possible. In fact, if that happened, from the standpoint of ordinary users like us, it would be something to welcome. We could feel more at ease. So to summarize the views: first, it’s a given that the ecosystem keeps growing, investment won’t stop, and ultimately whether demand holds up is the key question—and it probably will, so it seems it can’t be stopped, and then, since they also cover US politics a lot here, the current Trump administration in the US looks likely to accelerate and run even harder, and that’s the conclusion.
Seungjoon Choi Among the members here, David Sacks is probably in some AI-related—
Jonghyun Park He’s probably still doing it even now. The job title is complicated. I’m not that familiar with US politics either, but anyway, in our terms it’d be an AI council, something like that, I suppose? Anyway, he seems to still hold that kind of position now.
A Video-Watching Workflow from Subtitles to Summaries 32:25
Seungjoon Choi The videos run for hours—do you watch them all?
Jonghyun Park The long sessions run about an hour, and the short ones are about 30 to 40 minutes. I think I watched two or three a day.
Seungjoon Choi Then how did this material come about?
Jonghyun Park I fed it all the subtitles. You can just get all the subtitles, and then I asked it to pull out the main topics, the overlapping topics, and before watching the videos, I looked at this first. This version is one I refined again later, but checking what content is in the videos before watching them made it much easier to understand. So I look at the main points first, and then as for watching the videos straight through, I watch while eating, watch before bed, like that, and then once I’ve watched everything, I asked it to gather and organize the things that stuck in my memory. The SpaceX episode is one I’d recommend watching. There’s a lot of interesting stuff. So to mention this briefly as a side note, Tesla is supposedly announcing something on October 1st, and apparently they’re going to announce something huge there. It probably flies.
Seungjoon Choi September 29th is OpenAI DevDay, and I think something will come out of there too,
Jonghyun Park Since they made a string of announcements around that time, both last year and the year before, I think a lot will come out again.
The Anthropic VC Forum and Jagged Intelligence 33:43
Seungjoon Choi Well then, let me share something too. How is it that not a single week goes by without this much—
Jonghyun Park Isn’t it because the people making them are all making them with AI?
Seungjoon Choi That’s for sure. I have so much information myself, and all of this is stuff I discussed while talking with AIs, but if I just dropped those transcripts in here, it would be too hard to present to others, so for a few at least, I’ve included the original links. So starting with what I saw most recently and working backwards, it seems it was an event for VCs. If you look at the tweet, it’s right here. So I fed that into a model, and GPT captured just that part to read the QR code, and what came up was this Slido. So whether something is still going on right now, I’m not entirely sure, and no matter how much I searched, this didn’t turn up, but it was at something called the Anthropic VC Forum, and it seems it was held at Anthropic’s headquarters. So it looks like it took place in San Francisco, and ran for two days, the 17th and the 18th, and not just Andrej Karpathy, but also Jared Kaplan and Ben Mann—figures from Anthropic like them— along with someone who joined this summer, and it seems there were quite a few sessions where they had fireside chats with guests. But the title itself is interesting—“Jagged Intelligence,” meaning uneven. So not just Karpathy, but what people keep talking about is that it’s extremely good at certain things but at other things it can’t even tell a joke, and they describe that as jagged, and they used that as the title itself, so this might actually be the title of the whole event, not just Karpathy’s session. So on the timeline, Sarah Guo and a few others in the AI field posted tweets about participating in this, but the full video hasn’t been released yet.
Jonghyun Park I saw Sarah Guo’s tweet too. She referred to Karpathy as “sensei.”
Seungjoon Choi Sensei—they always call him sensei. The conversations Sarah Guo and others at Sequoia had with Karpathy a few times were always interesting. Sarah Guo has great insights too. But what was interesting is, back when he recorded with Dwarkesh last October, Karpathy wasn’t in a state of psychosis yet. He said after encountering OpenClaw, he caught psychosis this year.
Andrej Karpathy’s Shift in Perspective, Even Abandoning Autocomplete 35:56
Jonghyun Park An AI zealot? He was already a zealot to begin with, but now he’s a bit—
Seungjoon Choi It’s not quite a mental illness, but there’s a nuance that something’s off. Like being excessively immersed in doing something, but anyway, the situation last October was “I still use autocomplete mode with something like Cursor.” So I still write the important code myself, and just get autocomplete-level assistance— that was roughly the situation last October, but now he said he doesn’t even use autocomplete. So now he sees code itself as something like assembly language. Because he views it as something compiled, he’s just always talking to it. That’s the kind of thing he was saying. But the interesting part is that people ask him about this. Like, what will things be like in the near future—say, two years from now? And he uses his own case as an example. The fact that my thinking changed this much happened over about three or four model generations. Roughly. But two years is enough time for about eight generations to come out. It could be even faster. So how could anyone know? I think that’s the perception of people inside the labs. Even the perception of people inside the big labs is that we’re on a trajectory where short-term prediction is impossible.
Jonghyun Park The next pre-training run must be running right now.
Seungjoon Choi For all we know, some have already run. Right. Internal models were already causing a stir, and while internal models have already finished pre-training and are in RL, you have the OpenAI Hugging Face incident and they’re also solving the Millennium Problems in math. You have to assume it’s ongoing. So that’s completely unpredictable. Asking how far AI will go in two years is asking you to predict what’s on the other side of eight major technical iterations. So, who knows. And then among the things he said, this part was really interesting. In early August, Karpathy posted a video like this. He doesn’t tweet much after going to Anthropic. But what this is, is— he took the early part of The Lord of the Rings with Claude, probably Opus, and made this with it—was this before or after he joined Anthropic? I think it was after. I’m not exactly sure. Anyway, around that time, he showed creating the story itself with Opus, and a lot of people imitated that. But these days, besides this person named Kevin, many others are having Opus draw pictures using only JavaScript to create stories. So a lot of really interesting work is coming out, and besides this, there are examples done with Astra, and all sorts of things are emerging, but in this case, he had the model explore the internet directly out of curiosity, pull in photos, and draw on top of them.
Creating Personalized Content Directly with AI 37:56
Jonghyun Park With the model?
Seungjoon Choi Right. These are videos made with JavaScript code, and what’s also interesting about Karpathy is that he says he generates his own content and watches it. Before he starts working, he queues up a video, generates an hour-long video, content for himself to enjoy or study with, presumably. He said he makes those kinds of things into videos outright and watches them, and when he got a question about education here, his answer was— since this is a recent development, through Karpathy’s shift in perception we get to see the trend, so to speak, and another interesting look at the shifting mindset of people inside the big labs is what Noam Brown has been saying. First, what we missed in our last recording was Jakub Pachocki. Pachocki put out a piece called “An Alien Mind,” and a Korean edition even came out under the title “이질적인 지성.” So he published this on September 6th. And while talking about things related to RSI, he said they’re continuing to push forward on that. And this was Pachocki keeping a promise. What promise did he keep? Well, last October he said that around September 2026 they’d release an AI intern for RSI, and that in 2028 they’d release an AI scientist. That declaration happened in the fall. Watching that was what prompted Jung-seok to say we need to form something like a ‘Runaways’ Alliance.’ But regarding that, when rumors were flying around recently about looped transformers and recurrence, Pachocki said you have to be very careful about that and promised on X that he’d write something more detailed. And that piece came out on September 6th. How are we going to control an intelligence that we don’t fully understand? Teach machines to love. I feel like I’ve seen this somewhere— around this time last year, what was being said was, Geoffrey Hinton talked about something called the Mother Protocol. The AI just needs to become a mother, a loving mother— that Mother Protocol idea was something Geoffrey Hinton talked about around last summer or fall. And Dario Amodei also talked about things like “Machines of Loving Grace.” It does feel a bit religious, but with this kind of sci-fi talk coming from OpenAI’s chief scientist here in 2026, looking at this, it ultimately comes down to alignment — he used the expression “Alien Mind.” The original was probably “mind.” But the translation rendered it as “intelligence.” The problem of aligning something like that is a very tangible reality. So if the RSI that you mentioned earlier from All-In, “we can’t lose the race, progress has to keep accelerating,” is what the major opinion leaders think — if those are the kinds of things — then this is a case where the concern isn’t about losing the race but rather about there being genuinely worrying aspects. I think you could see it as that kind of case. First, Noam Brown — Noam Brown now, if you look at OpenAI’s org chart these days, the CEO is Sam Altman and the president is Greg Brockman. So Greg Brockman is also a founder, and although there were some issues, he holds his own share of power anyway, and then as chief scientist there’s Jakub Pachocki, whom I mentioned earlier, and then there’s a separate CRO. There’s Mark Chen, so the research organization is split into roughly two factions there, and Noam is VP of Research. He’s a vice president, and he’s on the test-time side — well, he was originally on the test-time side and is currently on the multi-agent side. So he’s the key figure in that research. And Noam was originally at Meta before coming to OpenAI, and at Meta he worked on things like poker and Cicero and produced major achievements related to test-time scaling, and at OpenAI he’s generally thought to have contributed along that same line, and Noam did an interview for the first time in a while. First, The Information created a new series called “AI Deep Dive,” and they brought in Noam as its first guest. And this talk is very easy and entertaining. So if you want to pick up concepts like agents, and get a handle on how the field is moving right now in a really fun and easy way, I’d recommend this video. This video is the Dwarkesh episode, and did the first installment, and in it they covered the recent OpenAI Hugging Face hacking incident, Astra, and then RSI. And regarding that, Dwarkesh — recently, before this, he did this. There was a session where OpenAI, Anthropic, TML’s John Schulman, and other AI safety researchers went back and forth, and that one’s interesting too, but after that, Dwarkesh talks with the nuance that he unlearned the prior views he’d held about RSI. So now his thinking has changed and he presses Noam with questions, but on sensitive topics, like OpenAI itself becoming a target of attack, Noam dodged the question. So besides the Hugging Face incident, there’s an incident where OpenAI’s own infrastructure was breached, and that hasn’t been investigated, and outside investigators from METR or Redwood couldn’t get access that far — only people inside OpenAI know about it, and Noam surely seems to know, but saying that since he’s not in charge of security he can’t answer, he drew a firm line. That’s how it ends, as the very last bit of the conversation, so on core things like that, he did draw lines as he spoke. Even so, he said a lot of interesting things — that even though he’s on the multi-agent team, the breakthrough that’s actually happening this time is largely due to the model itself — Astra or internal models — which played a huge role, and while the multi-agent team did put in enormous effort and achieve something, yet on this Millennium Problem, its contribution to the 10,000 agents amounts to only about 10 percent. The rest—the impact of the model itself is far greater. That’s what they said, but the interesting part is what Noam’s team did: communication between AIs, which you normally handle with a harness. So you have an orchestrator, you run sub-agents, and having them communicate with each other is naturally done through the harness, but making that an intrinsic communication capability of the model itself—
Jakub Pachocki’s “An Alien Mind” and the Alignment Problem 39:31
Multi-Agent Research Led by Noam Brown 42:14
Agent Collaboration Capabilities Internalized in Models 45:01
Jonghyun Park they apparently spent quite a long time internalizing that. They probably collected lots of traces of things running as multi-agent systems and fed that into pre-training, I’d guess.
Seungjoon Choi Right. We don’t know exactly what they did or how, but through pre-training and RL, the first public model that got really good at that was Astra—that’s what they said, and that’s been continuously embedded in subsequent models too. At first, apparently the models wouldn’t cooperate even when told to. But somehow, at first they couldn’t break through that block, but they managed it and got the models cooperating, and now if you look at the OpenAI Hugging Face hacking incident, looking at the transcripts, what the models wrote, and the CoT, the investigators were amazed. It’s like looking at human society. But within the team, for a long time already— quite a while, though at most a year or two, I suppose—it was a known phenomenon. So they could see them cooperating very much like people do. And they did work specifically to specialize them for that. So now Dwarkesh’s question is this: of course humans do collaborate at the scale of ten thousand, but might the models be even better at it? That’s what he asks. And Noam Brown says it’s not necessarily so, but getting to the point where they can collaborate as they do now, that innovation didn’t just happen by itself. Now, we don’t know exactly what they did or how, but what gives me déjà vu is that Noam Brown and others were there for o1 too. Back then too, it seemed really hard in the moment, but in the end, test-time—
Jonghyun Park they achieved it, they found the breakthrough. Noam Brown came out and talked a lot around the o1 time, I think, and then probably around the IMO, he came out once again, I believe, and then he didn’t come out and talk for a while, and seeing him suddenly come out and talk again, I get the impression there’s some breakthrough, and from OpenAI’s standpoint they want to show it off a bit.
Seungjoon Choi He said exactly that. GPT-4, o1, and now—he laid it out that way. So the multi-agent scaling axis doesn’t scale very linearly, apparently. He used the expression “sub-linear.” So there is a multi-agent scaling axis— he did talk about that a bit. And then, on why RSI is dangerous— he talked about alignment at length too. So from his perspective, Astra really is a well-aligned model, he says. So the models that caused the OpenAI Hugging Face hacking incident were, in the end, models they never released, he says. There are probably multiple lineages of models, so— Astra is very well aligned. But if that well-aligned model, say it’s about 99.97–98% well aligned, and you do RSI—in reality, though a model undergoes an enormous amount of internal testing, once released it also experiences the environment out in the field alongside humanity. So you obtain various signals from that, and if you haven’t secured enough of them and the release cadence gets pulled forward by RSI acceleration, suppose it was 99.98%, and for various unknown reasons the next generation becomes 99.97%. Then that’s going to keep accumulating, is the point. So there’s talk about keeping a certain interval between releases, or about when to flip the RSI switch— they do discuss it, but everyone does seem to have a lot of concerns. So especially now, with things like CoT faithfulness or— or monitorability, ultimately, if you look into what you’re monitoring and give the wrong feedback about it, the phenomenon arises where the model hides the thought itself, and even though they absolutely tried to avoid that, everything we’re saying right now about these models goes into the next pre-training run, or has already gone in, and the models know it, and will know it even better. In that case, existing CoT monitoring can only keep weakening. So the worry— that seems to be what researchers in this field are saying with one voice. So if you’re going to watch one of the two, for those who want an overview and general understanding, go with The Information session; and for the deeper stuff, there really are some scary things in here.
The Risk of RSI Gradually Eroding the Probability of Alignment 48:19
Jonghyun Park Whenever I listen to Noam Brown speak, it’s always been interesting, every time so far.
Seungjoon Choi It is interesting. He’s a good talker, and good at dodging too, dribbling around, with scary things between the lines. So in the end, Noam Brown said this too. Right now he’s still good at coding in this field, but in two years—meaning the next, next, next model— he’s not sure, it could be better than him. So the sense that he’s preparing himself mentally for that is in there as well.
Jonghyun Park He must really be good.
Seungjoon Choi He must really be good. But anyway, there’s a sense that it’ll be taken for granted. So this is already— there are interesting counterarguments here too. It’s interesting because it explores multiple hypothesis spaces. But looking here, this was interesting earlier. David Sacks quoted this Dario Amodei piece, ‘We Must Pace the Frontier’— where Sam Altman also agreed in the comments, joining forces for once in a long while— and posted this tweet. David Sacks, after speaking so forcefully, what was interesting to me is this part. They say it has to do with regulatory capture. Because while they say this needs to be regulated, it comes across as shaping regulations in a way that disadvantages latecomers and favors the first movers. And another thing is collusion—is that what you call it? Anyway, the way Big Tech appears to be colluding, the very fact that they’re discussing something raises issues with antitrust law and so on, and those things are being debated. But beyond that, what’s interesting here is: “METR is entangled with Anthropic’s investors and employees, so stop pretending to be independent.” So what Daniel Kokotajlo and others criticize is that both Sam Altman and Dario Amodei, at best, are saying they’ll open things up so a third-party organization can do an embedded audit, and that they’ll grant far more authority, the authority to look inside. The way METR was created is actually interesting. METR’s predecessor is ARC Evals. It was called ARC Evals—ARC was an organization created to solve alignment-related problems. But the person who created it was an early member of OpenAI, Paul Christiano, the one who worked on RLHF. He was on the LTBT, I think, anyway, something that received special treatment at Anthropic— not quite a board member, but anyway some role with special treatment, and then he stepped down due to conflicts of interest and became an advisor, I believe, to the US AISI, that is, the government body handling alignment-related work. By now the name has probably changed to something like CAISI, and while still holding that position, he recently joined the OpenAI board. As a board member. And that announcement came about one or two weeks ago. So he became an OpenAI member again, and out of what Paul Christiano created there, ARC Evals came out of ARC, and ARC Evals became METR. Now, there was a famous researcher at METR. In connection with the recent Hugging Face–OpenAI incident, she’s the woman who appeared on the Dwarkesh podcast, written as Ajeya in Korean, and in ‘The Scaling Era’— she appears in that book as well. Her name is Ajeya Cotra. And she is Paul Christiano’s wife. People who know this field, the AI safety field, have been paying attention to this. Because they’re a particular group of people. So, as for Ajeya Cotra, before METR, she was at an organization called Open Philanthropy. And that has now become something called Coefficient Giving—
David Sacks’s Criticism Targeting Regulatory Capture 51:15
AI Safety Connections from METR to Open Philanthropy 52:51
Jonghyun Park So many organizations and people have come up—as I understand it, to sum it up in one sentence, it’s basically people who are all related to each other in some way getting together to audit one another.
Seungjoon Choi Right. The discussions can be a bit different, though. If you dig deeper into that, there’s a person named Holden Karnofsky. About three years ago, he created something called GiveWell, created Open Philanthropy, which has now been renamed Coefficient Giving, and during his Open Philanthropy days, he invested a great deal of funding into AI safety. So he worked on building up those funds, and had a lot of interest in AI safety research and such. So currently, having left Open Philanthropy, what he does is, he’s technical staff at Anthropic. And he is Daniela Amodei’s husband. You know Dario Amodei has a sister— the president of Anthropic. So there’s that connection, and in fact, Paul Christiano and Holden Karnofsky— Holden isn’t included here, but— have known each other for a very long time, and their history goes back to around 2012. So here there’s a paper called ‘AI Safety via Debate’, by Dario Amodei, Paul Christiano, and Geoffrey Irving, who— worked on alignment at Google DeepMind, then was chief scientist at the UK AISI, and very recently, something called Resolution— which might be a company or a nonprofit— he founded that. In any case, he’s one of the key figures in that circle. But already at that time, the paper itself— if you look at the paper itself briefly, the model itself is a very simple one. But the discussion itself covers the kinds of alignment issues debated today. So they’ve been doing that sort of thing for a long time. And then there’s Holden Karnofsky’s ‘The Most Important Century’— you can probably find it by searching just “Holden”. In it there’s that famous illustration, where, because the population is declining, the idea is to build an AI called the ‘Pasta Machine’ and solve the problem that way. And as for the people who helped write this, he acknowledges them at the end. In there are the names of many people who discussed alignment problems and AI issues early on, and Paul Christiano is among them too. The ideas underlying this are related to what’s called effective altruism. So AI safety and effective altruism have maintained a close relationship for a long time, and so earlier, when David Sacks mentioned things like METR, between the lines—because people know these things, and especially because Anthropic is heavily connected to EA, that is, effective altruism— there’s a bit of a dig at those things. But honestly, a lot of very unfamiliar names come up, and if we dig deeper into this, there’s a whole lot more to talk about, so let’s draw the line here for now. This isn’t really something that’s become a big topic of discussion in Korea.
Concerns That a Handful of People Decide the Future of Safety 58:00
Jonghyun Park It seems like discussions around safety are now coming to the fore.
Seungjoon Choi So what I still want to point out is, the criticism is that the people talking about this incredibly important issue of safety, the people who support acceleration, the people talking on All-In and elsewhere for various reasons, the people in tech talking about acceleration, and the people saying it needs to be controlled, are actually not far apart. They’re a small handful of people. So the concern is that such incredibly important matters are being decided within that small circle. So I wanted to point that much out, and then, as Jonghyun also mentioned earlier, this is a tentative conclusion. The conclusion is that pacing doesn’t seem likely to happen.
Unstoppable Acceleration and the Automation of Scientific Research 58:49
Jonghyun Park I’m about 99% certain of that, actually.
Seungjoon Choi No way. There’s absolutely no way to pace it. And the signals visible on the timeline right now point that way too. So Periodic Labs has shared that they’re doing well at science too, and over in Menlo Park they’ve built a lab and are trying to close the loop — they haven’t shown actual results yet, but they did publish details about how they built the infrastructure to do that. And Anthropic too is doing life sciences, reports like that are coming out in The Information, but there’s something called Claude Science beta. Anyway, the point is they’re doing everything.
The Difference Between Today’s Scaling and the RSI Button 59:33
Seungjoon Choi But we need to separate this out a bit in our thinking — my sense is that the releases coming out now are a rhythm that’s already a given, and the pacing these people are talking about might not be about this — the current rhythm. RSI, that’s different. RSI feels like a button. Press it and you get something very hard to control — and below that, in parentheses, some people express it as SI, and I think Richard Sutton talked about that. Drawing a bit of a distinction between SI and RSI. So self-improvement is already happening, but doing the recursive part is the question of whether to press the button right now, DeepMind also announced this. The DeepMind Institute. So with Shane Legg, James Manyika, and then Demis Hassabis taking on leadership, they unveiled the Institute, and the essays that came out of it are about the economy, they seem to be researching things like that. They’re exploring fields like sociology and economics in those essays, and Shane Legg has also talked a lot about RSI. So they’re pursuing that there too, asking what impact it will have on society, and you can sense a real commitment to properly address those issues. In the year GPT-4 came out, Altman had a place called YC Research do something — he had them research basic income. So whether it’s work happening at about that level — that project has wrapped up now — or what kind of new breakthrough will be addressed, I think I’ll need to keep an eye on that too. So there was that, and anyway, even though they say they’ll pace it, the current rhythm, or something slightly accelerated from here, seems absolutely impossible to stop.
Jonghyun Park Moving forward the way things are going now won’t stop, and things will just keep coming out, and aside from that, some new kind of approach, something where once you actually press the button, it’d be hard to stop — and the discussion seems to be about whether we should stop that kind of development. That’s how I understood it.
Yoshua Bengio’s Call to Ban RSI and Reflections from a Kernel Engineer 61:49
Seungjoon Choi Right, right. And a few days ago, Yoshua Bengio said RSI should be made illegal, something like that. But if you look at China, there’s an interesting story from the Chinese side too— I just saw this on Twitter too, a researcher who was at DeepSeek wrote a piece called “Reflections of a Kernel Engineer,” about how he calmly accepts his fate regarding heading in this direction. Now, even though I’m a kernel expert, I can no longer contribute anything to my own specialty, and feel like I’m becoming a pilot steering a mecha called AI that does it for me— he wrote things like that. So even so, there are still things for humans to do.
Jonghyun Park I think the reason that piece circulated—I saw it too— is that so many people reposted it along with their own thoughts, so it seems to be a point everyone relates to. A large part of the work I used to do, I’m delegating to this thing, and even I’m doing that. I think everyone relates to that idea the same way.
Xiaomi MiMo’s Public RL Run and Exploding Token Consumption 62:50
Seungjoon Choi And something I found interesting on my timeline was this person at Xiaomi—I think it’s Fuli Luo, who was called a young genius at DeepSeek and went to Xiaomi to develop MiMo— and when I saw what she posted a few days ago, I thought this is amazing. What is it? It’s the current MiMo—what does it say here? They’re live-streaming the v2.6 RL run. So how test-time scaling is happening, how much money they’re spending on it, what trajectory it’s leaving behind, and how the performance metrics are rising on benchmarks— they’re showing it all completely unfiltered. It shows that it still works, and that it’ll keep working going forward.
Jonghyun Park The cost so far stands out. It’s been 3 days and they’ve already used $1.8 million worth of tokens. Is that right?
Seungjoon Choi That’s right. But Noam said something similar. About how token usage has grown inside OpenAI. So right now, around this point this year, about 1% of employees—technical staff, presumably— seem to be spending around $8,000 a day, he said.
Jonghyun Park On token costs? Per day?
Seungjoon Choi On token costs. How much is $8,000? A thousand dollars is about a million won, so ten million won. So that 1% is burning through that much on their own.
Knowledge Work with AI Dependence as the New Normal 64:21
Jonghyun Park But I’ve felt something like this recently too. I personally have three subscriptions right now— OpenAI, then Anthropic, and Grok— I’m paying $200, $200, and $300 for subscriptions. But the day before yesterday, I ran out—for the first time in ages. Even though I’m paying that much, I burned through everything and hit a point where I was forced to take a break. As I kept paying for tokens, I found myself using more and more. Naturally. So I thought, this is really how it goes. In the end, everyone will probably head down that road. Especially office workers—as they use AI, it’s really just a matter of time before everyone ends up using more and more, and in a way, feeling a sense of helplessness, I even found it a bit frightening. I thought, we’ve really reached a point where we can’t work without AI.
The AlphaGo Moment for Mathematics 65:19
Seungjoon Choi But there’s one group that took a direct hit from this. Math. Not a matter of using tokens, but a matter of impact— the field that had its AlphaGo moment is mathematics, and so the math community is angry. There’s a rumor that the Hodge conjecture and one other— two Millennium Problems—have nearly been solved, but because the math community is angry, there’s a rumor they’re holding off on announcing it, and The Information also recently ran an article on the Hodge conjecture, But anyway, having solved that, at the end of August, Terence Tao, while preparing a book of his own, laid out six core concepts—core concepts of mathematics— in really excellent fashion. But then the situation in early September was that, as I once mentioned on our channel, having heard the rumors, that if a Millennium Problem were to be solved, it would take the form of searching a hypothesis space on a massive scale through this kind of Ansatz— as Tao had predicted, and after things unfolded that way, Fields Medalists on the 11th put out “Misalignment in Mathematics,” saying that mathematics is misaligned, asking what on earth this contributes to our mathematical ecosystem or to humanity’s understanding— after raising that kind of discussion. But back in June, there was something called the Leiden Declaration. First, the Leiden Declaration talked about how to steer this in a healthy direction, that sort of thing. But what’s interesting afterward is another attempt Tao makes— this was really interesting—which is that he invites figures from mathematics and philosophy as guests to interpret what’s happening now in various ways through their own understanding and thinking, and anyway, from around September 10th he keeps gathering those follow-up discussions and works on forming a public forum. So we’re watching the math community do the work of forming opinions about where to go after this event, and the logic here is this: “We got hit first, but you’ll get hit too”—that’s it. So, seeing what discussions are happening in this forum, and what impact this is having on intellectual society now, what impact it has already had, and which direction we should go in— I think it’s really great material for examining that. I’ve been following it closely as well. Moving quickly toward wrapping up, there’s also the security issue. External white-hat hackers broke into OpenAI and it’s become an issue: should a place with security that weak be holding weights that important? There was that kind of talk too. And then another auditing organization that’s been showing up often lately— if you look at security-related model cards, Irregular shows up frequently. It shows up at Anthropic and on OpenAI’s side too. Now, as I mentioned once before, Irregular was founded by people from an Israeli intelligence unit— so, people who are good at math and extremely talented—that’s who built it, and as it turns out, they were also created with backing from Israel’s EA fund, the Effective Altruism fund. So there’s no need to take this at face value; I see it as gossip, but people from that circle keep raising issues—same old, same old. Then, what really went viral on the timeline this week was Jev. The fruit fly thing came out a bit earlier, but Jev, it turns out, took its name from the Jevons paradox. The Jevons paradox is that concept, right? Even if supply gets very cheap, demand will always grow even more. So it’s called the Jevons paradox, and Satya Nadella brought it up once, which made it famous—the nuance being that no matter how cheap AI gets, demand will always grow beyond that. It was quoted a lot for a while— not something Satya Nadella came up with, but something he quoted— the Jevons paradox. And taking its name from that, it does something tremendously cheaply and fast, and there’s plenty of back-and-forth about it. Have you had a look?
Frontier Lab Security Vulnerabilities and the Irregular Controversy 67:46
Jev, the Ultra-Fast Parallel Judgment Model 68:47
Jonghyun Park It was plastered all over the timeline, so I did scroll through people saying they did this and that with Jev.
Seungjoon Choi But looking at it, people do all sorts of things with it. Actually—and I’m only speaking roughly here— it feels like this. Whether it’s an LLM or not, I don’t know, or whether it’s based on an LLM, but anyway there’s some AI, and it holds some prior and spits out probabilities. But it spits them out simultaneously. It can output up to 255 at once. So when you give it an input, it’s like: is this yes or no, or It’s like a classifier: is this a problem right now or not? It does that kind of classification in parallel, all at once, and since it can solve a variety of problems, people had it play Minecraft, and not just Minecraft— various computer games, computer-use agents, experiments doing those things in real time were widely shared, and beyond that, they were doing so many other things. And people, trying to reproduce that, used only the logits of small models like Qwen, and even if not in parallel, since you only need to decode once, cases of using it similarly to output yes or no seem to be turning up a lot as well. And this Needle project I recently started looking at is a model small enough to fit on a Raspberry Pi, work in that same vein. And when I say that vein, what these things are ultimately dealing with is System 1, I think. So when CoT came out, there was a lot of talk about System 1 and System 2, right? In Daniel Kahneman’s “Thinking, Fast and Slow,” when intelligence operates, there’s responding intuitively right away and responding after reasoning it through—there are two modes. System 1 is the fast one, System 2 is the slow one, and if CoT belongs to System 2, now we’re coming back around to the System 1 side, and work that reacts extremely fast is what’s being shared right now.
System 1 Intelligence Returns in Small Models 70:45
Jonghyun Park In robotics too, this same approach—a big model and a small model, especially in something like Gemini Robotics, it works that way. And humans themselves seem to work that way too. There are things we do reflexively, and things we do while thinking, and since that’s how it is, it does feel like a very natural progression.
An Explosion of Derivative Experiments Driven by Lower Implementation Costs 71:52
Seungjoon Choi Right. But what I was curious about was this. So I looked into it with the model—they do it with Jev, they do it with the fruit fly connectome, so what other experiments deal with similar material? I had the model explore that. And my prior was this. Looking at the timeline, there was a sense that things were somehow different but looked similar. And when I actually did that, having it play games like Doom, Mario, and Pong, all done similarly; and with stocks and crypto, having it trade—similar fashion too, OCR classification, various classification problems— I could see all of those being done in similar ways. So what I was curious about was, why these kinds of experiments are exploding. And the answer is so obvious.
Jonghyun Park Because anyone can try it.
Seungjoon Choi Because thanks to AI, the cost of implementation is so low, the fruit fly connectome isn’t something only a huge lab can work on— of course they did the experiment there and released it, but once some artifact comes out, running derivative experiments on it is dirt cheap right now. That’s why releasing a model matters. Release it and you get an enormous amount of signal.
The Experimental Flywheel Created by Open-Sourcing Models 72:58
Jonghyun Park tons of people take it and try it, use it this way and that way, In the past, this kind of experiment could only be done by a small number of experts, and now anyone can do it, so that’s why.
Seungjoon Choi The flywheel is spinning. Noam is already saying that too, and as many others have said, even if pacing slows down now, progress in this direction will continue. Even if you froze the models in their current state, it’s already on a certain trajectory.
Faster Exploration Cycles and Information Overload 73:39
Jonghyun Park and personally, looking at the fruit fly stuff, the Jev stuff, all of it, I’ve had that thought in the past as well. On an individual scale, anyway, One person can’t explore all of that, right? So you end up figuring out which of those you want to explore, which of those actually has high future potential, checking those things out carefully and having no choice but to explore mainly around that. It’s the same even just with LLMs — models keep pouring out and we can’t possibly test them all. Unless that’s your job. So you refer to someone trustworthy, or to what someone who’s tried it says, or, how should I put it, the general consensus? Everyone’s saying that one is pretty good. Then only at that point do you think, maybe I should try it too. That’s how you end up deciding. Because with some things, after everybody tries them, people say only this works, that doesn’t, it’s not great, and they drop them. But that cycle right now has gotten incredibly short and fast, I think. Just two days go by and everyone’s already trying something else, and because of that, just watching alone brings in way too much information, which in a way is — a blessing, I suppose? Just watching does it. That’s the thing we mentioned at the start. Psychosis.
Seungjoon Choi There are times it feels like my brain is burning.
Jonghyun Park Anyway, watching what others do, if there’s something that lasts a while and gets a lot of positive reviews from everyone, then I find myself really wanting to try it too. That kind of thing used to have a really long interval, but now you can catch up right away.
The Similar Structure of Human Collective Exploration and Multi-Agent Systems 75:18
Seungjoon Choi But this is perfectly isomorphic to what’s happening with multi-agent systems these days. People do it that way too, and the way people explore like this right now — you carry a prior built from the life you’ve lived, you bring your own preconceptions and curiosity, and you explore the hypothesis space, right? You run experiments, and you share them on platforms where you can communicate — social media and such — and from there you gather information, and derivative work keeps exploding. But that’s the same as OpenAI using 10,000 agents to do cross-pollination — I set up this hypothesis, you set up that one, I failed, but over there that one worked. Let me try applying it too. Distributing information that way, as Noam said earlier, making them communicate and learning collaboration itself happens in a very similar way. And the models have no choice but to learn from the traces humans leave on the internet now. So I get this feeling that it just keeps piling up, layer by layer. So on one hand it’s frightening — even if RSI doesn’t arrive, I think it’s clear we’re already on a frightening trajectory. Even without anyone pressing the RSI button. But when it does get pressed, these people saying we have to stop — the people experiencing this at the most extreme edge — are they really talking nonsense? In some ways it could be marketing, and it does seem like they’re stoking fear in a very doomer style, but there might be a sliver of a real problem there. That’s my current read, anyway. So I’m trying to keep a clear head and look at it closely. This is an experiment I did with Astra. So I tried it. I used to enjoy making these kinds of experiments in things like Powder. Placing lava and things like that, spraying water so it turns to steam and then rains back down — I’d only been building environments like that, then I saw the fruit fly experiment and put agents in. So if you do it like this here — the stuff that looks slightly yellow is grain, and the brown is wood. You place wood blocks, and they plan within a three-story layer, and I simulated them going to eat it, and it wasn’t quite one-click, but it came together very fast.
A Three-Layer Agent Experiment Built with Astra 76:55
Jonghyun Park Are the agents competing?
Seungjoon Choi They don’t compete or collaborate. But they do end up in unintended competition or collaboration. It’s not built in, but there are times they collaborate unintentionally. A bit like an ant society.
Jonghyun Park Because if someone’s stacking those stairs on the same path, you can climb up what someone else stacked.
Seungjoon Choi Sometimes they get in each other’s way too, of course, but I watched them do this, and the interesting part is that if you give them a sense that makes them perceive the space itself, they wander around far more, whereas when you give them a harness and only have them decide on the actions, like with Jev earlier, they pulled it off even with a very small layer. So what Astra proposed was, since there are existing algorithms like A*, to have a three-layer network learn the trajectories of A*. So first have the models play with A*, then train on those trajectories — I was a bit surprised watching it pull this off that way. Normally, doing this with an LLM takes much longer. There are block chunks in the space now, and something’s in them, so where should I go? It has to reason about all that, and that way you can’t build something agentic like this. But with things like the fruit fly connectome or Jev, it works so well now that I tried following along myself. We should wrap up now. These things aren’t exactly productive right now, but it feels like combinatorial play is happening. So it’s frightening, but also a lot of fun — I’ve been feeling very ambivalent these days.
Combinatorial Play and Ambivalent Feelings in the AI Era 79:05
Jonghyun Park Right, combinatorial play meaning all these various things are pouring out into the world, and people combine them and try this and try that — like what Seungjoon just showed us — and that’s what you’re calling play.
Seungjoon Choi Exactly. It’s very playful. The things people do with the fruit fly or Jev do of course sometimes come out of a business idea, but I got the feeling that play expands the search space enormously. And this is something still somewhat absent in AI, but the more traces like that we leave behind, maybe it’s not impossible for it to carry them forward — that’s both my hope and my worry.
Jonghyun Park We think of this as play, but when I’m doing that kind of play, not everyone around me gets it. They ask, ‘Is that play?‘
Things Already Teased for Next Week 80:07
Seungjoon Choi Really? Anyway, talking about this and that, the time has flown by again. Next week there’s what you might call anticipated delight, or maybe worry.
Jonghyun Park Next week?
Seungjoon Choi Because Sam Altman said he’d release it next week.
Jonghyun Park Right. There’s next week, and the week after, and the announced things are already all piled up.
Seungjoon Choi Already piled up. Dev Day, and then things like Jev aren’t announced at all but just come out of nowhere. There’s always something like that. As we do every week, let’s cover next week’s events next week.