EP 118
Inside the Singularity | AI Investment Cycles, the Math Shock, and the San Francisco Mood
The Sense of Passing Through the Singularity 00:00
Chester Roh Today, as we record, is October 11, 2026, a Sunday morning. Things are changing so fast that as soon as Seungjoon Choi joined today’s recording, he opened with, “I’m exhausted.” So much is happening. New models are being announced, and now people are saying math is over. What will be next? The curve of progress seems to keep getting steeper in ways we really can’t predict. So it seems we’re not alone in saying we’re passing through the singularity. People across the community are talking about it too. So today we’ll talk about that. I’ve prepared one piece, Seungjoon Choi has one, and Jonghyun Park has one— three in all. My piece looks at this rapid change through a historical lens: What were the railway and fiber-optic booms like? What cycles did industries go through? With the world changing so quickly, I want to take a step back and see the big picture: what might happen next, and what should we do about it? It’s a business thought experiment. I often talk about things from that angle, so I’ll start by sharing a business thought experiment.
Mapping the AI Market Through the Time Gap and Domain Gap 01:26
Seungjoon Choi I’m curious which version this is.
Chester Roh This is the framework I use to view the market right now. It’s a diagram I’ve used at Runaways’ Alliance and elsewhere before. The x-axis is the time gap, and the y-axis is the domain gap. I tried to fit what’s happening in today’s market into one framework. At the upper right, the companies we call Frontier Labs, like OpenAI and Claude, are pulling us toward the cutting edge, toward the singularity. Right behind them are Qwen and DeepSeek, which I call Quasi Frontier Labs. These companies follow about two to six months behind. Further back are companies stepping away from general domains to focus on specific verticals such as legal services and consulting, or accounting. Both Frontier Labs and Quasi Frontier Labs continue to expand toward the broader public and into domains beyond LLMs, beyond what we call knowledge. That expansion is ongoing, so I’ve shown it with these arrows. I labeled the green line “adoption frontier.” The people to the right of that line are keeping up with these developments, while the people to the left haven’t caught up yet. I’ve always enjoyed plotting entire industries on two-dimensional graphs,
so I squeezed everything into this one. So where are we? We’re at the spot I’ve labeled “we’re here.” At the most virtual end of the domain spectrum is what we call the LLM, the crown jewel of the knowledge industry. At the opposite, physical extreme is human labor. I placed the domains between those two ends. And the closer you get to the physical world, the harder it is to build an RL environment that provides the reward signals we always talk about. The more virtual it is, the easier that becomes. Enormous domains lie between those ends. Huge industries are clustered there, and I think we’re just straddling the RLVR boundary, somewhere around there. If I drew this diagram more realistically, I think it would look something like this. I drew it much larger earlier, but what we see happening at Frontier Labs, what’s happening in mathematics, and what’s been automated through coding— these are all familiar topics in this community of engineers and entrepreneurs. But take just one step beyond that, and huge communities have yet to experience any of this. Seungjoon Choi said something like that when we were chatting earlier: there’s no one to talk to when you go outside. That’s because not everyone lives at this pace. Most people don’t, at least not yet. There are still plenty of discussions claiming that Tesla FSD is the same
as the driver assistance system in a Hyundai. So when I think about these different senses of time, I think the gap has grown about this wide. For much of the public, AI still hasn’t happened. And then there’s superintelligence. Trump said to change it to superintelligence, and even SpaceX became SI. But if ASI emerges, the industries above the RLVR boundary may all be finished, I think. Broadly speaking, that’s roughly the playbook. Quickly recognize the value Frontier Players offer, decide whether your domain is safe, and then make money from the time gap by serving people who come along later. Right now, that’s the business model for most companies outside Frontier Labs. Taking that a step further, we could build our own domain-specific RL environments, for example. So I still think training models and fine-tuning them—things that, over the past two or three years, people said had disappeared and could all be replaced by prompt engineering— may be bringing back an era for people who fine-tune and work on models. And I think Jev was a particularly good example of that. So that’s where today’s discussion begins.
The AI Investment Cycle Comparable to Railroads and Telecommunications 06:36
Chester Roh People compare today’s AI investment to the great bubbles of the past, including infrastructure booms in railroads and telecoms. They say it’s an investment cycle almost on the scale of the railroad boom. And the sums involved are enormous. Let’s look at a few numbers. Add up what four Big Tech companies say they’ll spend in 2026, by the end of the year, and you get $725 billion. A billion dollars, at 1,000 won to the dollar, is 1 trillion won, so we’re talking nearly 1 quadrillion won in capex from those four Big Tech companies. Some of it comes from money they’ve earned, some was raised through stock, and some was even raised through debt, with much of it remaining off the balance sheet. But Nvidia makes about 89 trillion won a quarter—nearly 100 trillion. But the companies making money from AI right now— Anthropic and OpenAI— neither discloses its revenue. People use publicly reported figures and bits of data that trickle out to estimate their ARR. On an annual basis, the two together bring in roughly $40 billion to $65 billion, or about 100 trillion won. Capital spending is nearly ten times that. But revenue is only a tenth as much. And even that revenue is close to what Nvidia makes in a single quarter.
The Enormous Gap Between Capex and Revenue 08:17
Chester Roh So there’s a huge gap. We’re still in a period of enormous investment. What justifies all that investment? The expectation of more growth. That this is only the beginning. That once the consumer market opens up beyond the enterprise market, it will be enormous, and the entire industry built by Google, Meta, and Apple will be replaced by AI. Those are the assumptions underlying all of this. So will it really happen? What if it doesn’t? I asked myself a few questions like these. Even in a world that’s advancing incredibly fast, I wrote down four questions to start this thought experiment.
Seungjoon Choi Before we go on, I’m having trouble grasping just how much 1 quadrillion won is. What would be equivalent to 1 quadrillion won?
Chester Roh South Korea’s annual government budget is probably around 400 trillion won. And its annual GDP is probably around 1 quadrillion won. I mean the total spent by individuals, businesses, and the government. It must be about that much. There are timing differences, but accounting for those makes the calculation complicated. It’s hard to determine which GPUs, bought when, are generating revenue now, and to match every input to its output. So I’ve lumped everything together at this year’s level. Even so, if you compare this year’s investment with this year’s revenue, there’s a big gap. Please keep those caveats in mind as you listen. I’m not calculating these figures precisely to make predictions like an analyst. I’m using them for a broad thought experiment. So some of these figures may be wrong. I ask for your understanding on that before we begin. It’s an enormous amount of money. To answer Seungjoon’s question, when I explain the idea of 1 trillion won to people at my company, I think back to our Seongsu-dong office. You can see Lotte Tower in Jamsil from there, and it looks huge. It’s very tall, too. If you add up the site, the residences, the hotel, and everything else, that property is said to be worth about 10 trillion won. So I tell people to use that as a point of reference.
The Diversification of the Frontier and the Disappearance of Escape Velocity 10:47
Chester Roh Now, back to the main point. As you know, Anthropic, Google, and OpenAI aren’t the only ones that can build frontier models. With capital and enough compute, datasets, and talent, anyone can build a frontier model. I think that idea has become fairly widespread. And earlier, we saw just how incredibly They’re advancing so quickly. They’re advancing at breakneck speed right now, and that’s no exaggeration. In an industry advancing this quickly, we usually talk about escape velocity. One or two companies reach escape velocity, dominate the market and gain a technological monopoly. Then they can earn money with ease, protected by patents or other safeguards, as in the pharmaceutical industry. That sort of thing happens, but here, the market keeps diversifying. It has diversified relentlessly, almost to an exhausting degree, and new players keep entering. All on the single assumption that the market will grow. And lately, Andreessen Horowitz has been making a lot of presentations like this. They gather as much data as they can on where the AI market stands today and make a lot of YouTube videos about it. They say about half of Americans use AI, but fewer than 5% pay for a subscription. That makes no sense.
And looking at total revenue, the top 10% account for more than half. The top 1–2% account for 20% of revenue. So revenue is heavily concentrated at the top, and most revenue-generating workloads there aren’t the kind of chats we have with ChatGPT. They’re large-scale agent services, not chat services: workflow automation, video production, coding, and mostly work of that kind. So in another Andreessen Horowitz YouTube video, they say there are 30 million coders. Worldwide. But we can assume nearly all of them are already using AI, and they’re generating nearly 100 trillion won in revenue now. The number of knowledge workers is said to be around 1.5 billion. That’s about 1.5 billion people. So they ask how big the market will get if all those people start using it. But we need to question that, too. Will they really use it as much as coders use these agent workloads? Or will they use it only for simple tasks, like summarizing emails? We don’t know yet. OpenAI and Claude have also created Work, right? They want knowledge workers beyond coders to try it and built Work around office products, but they haven’t released figures showing how much it’s growing.
Surely, if it were growing rapidly and meaningfully, they would have brought out those figures and said, “Look how well it’s doing.” So outside coding, I suspect AI isn’t yet widely used for knowledge work. As for the consumer market, when OpenClaw came out early this year, we wondered what impact it would have if the general public got hold of it. We’re watching that now. Grokbot, Muse. They say five million people use Muse, and give a user count for Grokbot, too, but we’re still not seeing many consumer applications. People aren’t using them much yet. I wonder how this will play out. So the rationale behind the massive overinvestment we’re seeing now is this: All knowledge workers will start using AI. And consumer use cases will emerge. Compared with how much we use it now, there’s no telling how much bigger it could get. But I think it’s time to ask whether that will really happen. Because from what I’ve seen around me, people’s adoption cycles aren’t as fast as I expected. And companies seem to be shifting their workflows to AI, but that doesn’t feel very fast yet, either.
So will this keep being a battle between models? It’s been a benchmark battle all along. The frontier model takes everything. So we thought Anthropic and OpenAI would settle the technology race. We assumed Google would catch up, of course, but Google has been surprisingly quiet lately, which has puzzled us all. Meta is catching up, too, as is Elon Musk’s Grok. So what kind of game comes next? As I mentioned last time, I got slightly hurt on a business trip to Europe, and while I was laid up for a few days, I looked into what had happened in the past by asking Claude about these things. Looking at the scale of investment today, one striking comparison, as a share of GDP, is the railway boom from the early 1800s to the early 1900s—about 100 years. Think about it: in the late 1700s, the steam engine first appeared, and it drove the Industrial Revolution. The first thing to come out of it was the railway, and the second was the use of steam engines to begin automating factories. Those were the biggest changes back then. Looking at the history of railways, the first railway line was built in 1825, and the first commercial line, in 1830, connected Manchester and Liverpool. That was the first of its kind.
Before that, Liverpool imported cotton, and Manchester had many mills that wove it into cloth. It was where the Industrial Revolution first began. But until then, goods had moved by canal. Then the railway came along and broke the canals’ monopoly on that market. Transport became much cheaper and faster, too. And it wasn’t just freight: passenger travel suddenly took off as well,
The Railroad Revolution and the Overinvestment Frenzy 17:21
Chester Roh and people began traveling more. It was much like the moment ChatGPT first appeared. Wow, this is going to change the world. So cities and branch lines everywhere saw a construction boom. And on profitable routes, it wasn’t just one line that got built. More kept appearing. That led to widespread overinvestment, with investment in railways continuing for decades. The thinking was the same back then: build a line, and the market grows enormously. But looking back, it really was overinvestment, and a great many companies went bankrupt. Investors who backed the railway lines themselves recovered almost none of their money. But they left behind infrastructure and cheaper transport, which changed how customers lived. As people’s behavior changed, the ones who actually made money were those who built new value on top of that infrastructure. But the transport operators themselves all went under. Farm owners in the countryside started businesses delivering fresh milk to cities, and fresh vegetables reached cities, too. As these businesses took to the railways, they took virtually all the profits.
So, to sum up the big picture, railway investors saw how much money could be made, and an investment frenzy took hold. As that frenzy grew, banks and investors weren’t the only ones putting up money. Ordinary investors and consumers piled into the bond market, too, funding the industry. The whole country did. But they all lost money there. So where did the value go? It went to the businesses that added value on top of the railways. And looking at it more broadly, society as a whole benefited. That’s the right way to see it. What happened to those industries after that? They could no longer sustain the overinvestment and entered a period of consolidation. So eventually, in the United States, there was JP Morgan, and in Britain, a few dominant firms took over the lines and standardized what hadn’t been standardized, and coordinated timetables. Government regulation came in as well, banning collusion on fares. As all of that took shape, the railway market became the one we know today. A few dominant firms operated under regulation, in an industry where they couldn’t make huge profits. And that brought the hundred-year cycle to an end.
Seungjoon Choi So what corresponds to that today? That’s what I’m curious about.
The Moment of Truth for the AI Infrastructure Boom 20:23
Chester Roh Today, we’re in that early market stage, with rosy projections of how large it will become and widespread overinvestment in infrastructure.
Seungjoon Choi So the companies that went under— what corresponds to them today?
Chester Roh None have gone under yet. When the outlook for the future is extremely rosy, what happens is the capital markets come along for the ride. As long as investment continues and money keeps flowing in, the market keeps growing. And the real moment of reckoning comes when the capital markets start to question it. Consumer adoption isn’t happening as fast as expected. Compared with the investment going in, revenue growth at Anthropic or OpenAI looks slower than expected. Then the stock investment that would have followed… When a company goes public, it raises funds by selling newly issued shares. That can bring in an enormous amount of money. Raising money through new shares is called equity financing, and if that equity financing isn’t enough, companies borrow money. But if you look at infrastructure investment now, more than 80% is financed through debt. So hyperscalers are borrowing money, too, as are outside firms like PE firms specializing in infrastructure investment. And just as happened with the railways, private bonds are appearing now, too. Products inviting investment in the GPU boom and the data center construction boom are now emerging. This stage closely resembles the railway market’s period of overinvestment. Whenever an industry inspires a rosy new outlook, in hindsight, we find expectations exceeded the scale of the business that actually materialized. Expectations were always high, and so was the overinvestment. That overinvestment laid more fiber-optic networks and far more rail lines.
And in fact, relative to the number of software companies actually needed, far more money was invested. Think about it: It happened with word processors, then in the Web 2.0 era with countless web applications, and again in the mobile era. When the app era began, there were so many app companies. We needed one camera app, but there were thousands of them. In that way, supply always exceeds what actual demand calls for. So what creates all that extra supply? Expectations that the market will grow much bigger draw in capital and investment, and that’s what creates it. If you look at the AI market in that light, something similar seems to be happening now. As I showed earlier, even this year, the amount invested far exceeds what they’re earning. They spent $725 billion this year while bringing in about $175 billion. That figure includes more than Anthropic and OpenAI: It also includes companies like neocloud providers. It’s a figure calculated generously at $175 billion, yet even that is only a third or a quarter of the investment. So they’ll need to earn much more next year, but when I think about how much those earnings can grow, I have a feeling they won’t grow that much.
The Problem of Not Seeing Demand After Coding 24:10
Chester Roh Coding has already come into the picture, and coding used to be the entire software market. Over roughly a year, they all entered the agent coding market, and as they adopted agent coding, Anthropic’s and OpenAI’s revenue shot up. That’s why the growth curve was so steep. For another wave of adoption, the next workflows would need to fit this approach as well as coding does, generate as much excitement as coding, and be adopted just as quickly. That’s not happening now.
Seungjoon Choi So this isn’t a question of B versus C. Adoption itself isn’t happening at that pace.
Chester Roh Right, right. While money was plentiful, those expectations
The Consumer Market That Opens After the Bubble Bursts 24:59
Chester Roh drove enormous overinvestment. But when reality falls short of those expectations, a capital-market collapse follows. Investors don’t make money. And when they don’t, the overinvested businesses consolidate and become standardized. Then comes commoditization, followed by regulation. Then, when it’s offered on the market at very low prices, the consumer market finally starts to grow. The same thing happened with railroads. During the railroad boom, from the mid to late 1800s, people wondered, “How bright will the future it creates be?” At the same time, westward expansion in America intersected with events like the Civil War. The area where people lived kept expanding, and all of that was intertwined. That’s how it grew, but by the late 1800s, the bubble had burst, and it had become the oligopoly of a few firms we know today. Of course, that played out over a century with railroads, whereas with AI, it’s happening in roughly one- or two-year cycles. That’s the difference. Also, once you’ve built a railroad, you have infrastructure that lasts for decades, A model has a four-month shelf life. Four months later, a new model comes out and the old one disappears. That’s the difference. Looking at the big picture, I think we may be going through a similar cycle.
It was only in the 1900s, when people like JP Morgan came in and completed the restructuring, that, on this infrastructure, mail-order sales, parcel delivery, and urbanization began to flourish. Urbanization also drove the growth of knowledge industries. So when I compare today with the railroad boom, I see several frontier labs in China, several in the US, and some government-led ones in Korea. Europe has several, and Japan says it will build some too. I don’t think we’re at the end of this period of abundant capital flowing in on the promise of a rosy future. I think it has further to go. I have a feeling we’re somewhere in the middle of it. So, as I said earlier, the consumer market starts to open up once this phase ends and models become standardized commodities. Even at the end of last year, people were saying that Anthropic and OpenAI were pulling far ahead at the frontier: “Models are going to get very expensive, and OpenAI is looking to offer $3,000 and $10,000 plans, because the productivity gains will far exceed what people pay for them.” That’s what people were saying, right? That model prices would rise. But now the opposite is happening. Model prices keep falling. But capital spending keeps rising. So when we talk about companies and things like that, we often hear this, right? We talk a lot about how much AI will change knowledge work, saying, “AI has made it all obsolete. We’re all going to be unemployed en masse.” That’s what people have been saying, but so far, in our day-to-day experience, we haven’t really felt that change.
Knowledge Work Shifting from Steam Engines to Electric Motors 28:34
Chester Roh This is another way of looking at it, but I think the existing system of knowledge work hasn’t been reshaped to fit AI yet. A common analogy is how steam engines shaped factory design. There was one large steam engine, with a main drive shaft running into the factory. Countless belts were connected to it, running to the first and third floors, where they powered various standardized tools throughout the factory. When electricity first arrived, factories simply replaced the steam engine with a large electric motor. Productivity barely increased. The real gains from electricity came later, when motors became small enough to power sewing machines and individual workflows. New, smaller machines emerged, production processes were redesigned, and that’s when productivity really began to rise. That’s the analogy. So the knowledge-work system we’ve built around companies over decades hasn’t changed yet, even with AI. We’re still using AI like a steam engine. If we reorganize work as electric motors allowed, that’s when real change begins. are saying it will begin.
But these are still just predictions. From what I’ve observed up close, or from looking at Palantir and Chamath’s AD90.ai, and the many companies talking about FDEs that work with businesses on AX consulting and implementation, I get the sense that market hasn’t quite arrived yet either. So the most important question is whether demand will catch up with the money being invested. That’s what matters most. It hasn’t yet. The first thing to watch is whether new markets emerge beyond coding and whether Anthropic and OpenAI’s revenue continues to grow at this year’s rate in 2027 as well. That’s the first point. Of course, the demand those two create will also be spread across Google, Meta, Microsoft, and many smaller companies, which will serve some of it. So that growth may be spread across their figures, but the first question is whether the growth rate holds. Second, the capital markets are absorbing all this overinvestment, largely in GPUs. Earlier, Seungjoon Choi, you asked, “Just how much is 1,000 trillion won?” Remember? That’s beyond the scale of a national budget. It’s closer to a country’s GDP, and Korea isn’t a small economy. It’s one of the larger economies.
Can investments on the scale of a whole country be justified? Will the companies providing those GPUs and cloud services keep finding enough customers relative to the money they raise to invest? Right now, people say, “Everything we build sells as soon as it’s finished,” and data center builders are optimistic too. But as these figures become reality, I think that outlook may change. So whether there’s a major collapse or a smaller one will depend on the gap between the investment going in and the revenue coming out. It depends on whether revenue catches up across neoclouds, Anthropic, and the others combined with investment in GPUs, and whether the amount invested in the meantime starts to fall. That demand will come from enterprises and, after B2B coding, from companies adopting agents. The really big opportunity is consumer adoption. Muse, Grokbot, and Gemini Spark, for example, are going after what we now call search. AI hasn’t completely taken over demand in areas like Amazon, search, and Google. Google still earns enormous revenue from its traditional business, and Amazon still earns enormous revenue from selling goods. Whether Grokbot or Meta’s Muse captures that revenue, and whether that market grows, will be the next thing to watch.
AI Companies Expanding into Every Domain Without Boundaries 33:09
Chester Roh Another thing I found interesting was that I recently joined Jinwon Lee, 활성, Nota’s CTO 김태훈, and Lablup’s CTO 김준기 for a panel with four CTOs at the Lablup conference. Looking at AI companies today, Everyone is expanding into every area. Just a year ago, companies would say, “We do fine-tuning. We make orchestration software. We build models. We do this or that,” each staking out one area. But look at companies now. Starting with OpenAI and Anthropic, they’re all expanding across boundaries. Anthropic and OpenAI openly say that biotech, legal services, and internal company workflows and so-called office software are huge verticals they intend to take on themselves. They’ll handle all of those and partner only on the rest. They say this openly.
So businesses at every layer are now expanding into every layer. The starting point behind all this is that, ultimately, models can’t capture as much value as we thought. We used to think companies that built the best models would make all the money, but that’s not the case. Models keep getting commoditized and cheaper, while Chinese models and others like them catch up two to four months later. If their capabilities all converge at a higher level, what would you do if frontier models raised their prices? A rational company would switch to GLM, even if it fell a little short, or use DeepSeek instead. So model companies can’t capture as much value as originally expected. That part of the market is becoming more commoditized. Looking at it by layer, value capture spreads both upward and downward. Downward, chipmakers like NVIDIA and data center operators will make money. Upward, value will move into applications. From my perspective, and that of most businesses watching the market, including Andreessen Horowitz, the application market hasn’t even opened up yet. Only a few applications exist so far. So it’s fascinating that value capture in the model layer is moving both upward and downward.
Value Moving Toward Scarcity 35:38
Chester Roh That brings us to a very important question. Earlier, I mentioned the fugitive playbook. Those of us who aren’t yet in the AI market don’t know where to go, but even those already in the market don’t know where to go either. I think that’s closely tied to why labs and promising companies at every layer are all expanding in every direction. They’re facing the same uncertainty. They don’t have the answer yet, either. Going back to the railroad boom, and feeling like a blind person grasping an elephant’s hind leg, if we ask where value is moving, the answer is that value isn’t always captured where the best technology or the most money is. It doesn’t necessarily go there. It follows a basic economic principle: it moves toward what’s scarce. And only a few things are scarce. First, in the consumer market, supply can keep growing indefinitely, but the number of paying consumers is finite. The population is even shrinking, making those consumers scarcer still.
So value will move to where consumers direct their attention. Next, in business and fields like coding, the ability to build things used to be scarce. If anyone can build things, what becomes scarce next? I’d say it’s where context and accountability accumulate. Look at the people thriving in the AI era right now. They know how to use AI and have run businesses. What does it mean to have run a business? They didn’t just do what others told them. They organized their own work and took responsibility, taking risks to make things happen. Those people become the scarce resource, and value capture will shift toward them. So companies built by people like that, such as Palantir, or firms offering domain-specific AI workflow consulting and solutions, will increasingly be where value is captured. That’s my view. Coding got there first, and I think the coding market is now finished. Everyone who was going to enter has entered. I also think coding workflows are nearly at their endpoint. I spend over one million won on tokens, just in subscription fees. And I don’t think I’ll spend more than that. As I’ve been talking, I’ve reached a conclusion. We need to wait and see.
And the future doesn’t look all that optimistic. From the capital market’s perspective, I think we’re approaching a point where a bubble could burst. Once that bubble bursts, Anthropic’s and NVIDIA’s share prices may fall, but that’s when the real thing begins. If I drew it as a graph, we’d seem to be near the peak now, but after a rise and a slight dip, I think a second phase of real growth will begin. I think what matters now is understanding that coming era. With the singularity approaching in this market, let’s step outside it and look from a little farther away. Where is this market headed? It’s a thought experiment. Once this period is over, when we look back, perhaps from a future where something has run its course, it’ll be clear: “We should have done that back then.” But we can’t see it now. Value has always stayed where things are scarce, not where technology or academic credentials are best. That’s been true throughout history. I hinted at this earlier when talking about companies. At my company, and at several large companies I’m involved with, we’re experimenting with AI to change workflows. Some tasks used to require scarce skills, but now we classify them as tasks we can have AI do instead. Other things remain scarce: areas of uncertainty where evaluation metrics aren’t clearly established.
Someone has to step in despite that uncertainty, decide which direction to take, take responsibility for that decision, and keep driving its execution. Whether directing AI or people, they keep saying, “Go this way, keep going.” “Do this, do this, do this.” You could call that leadership. Those capabilities are becoming scarce. That’s where the value will go. That person may now work under me, consulting for this industry, but once that work is done, they could leave and apply that expertise to another industry. If they build something, their company could sit between the model and the customer and capture the value. That’s the sense I’m getting. I discussed this with Jeongkyu Shin when we went to the Lablup conference. I asked the CTO what would matter going forward. The answer was, “Brand.” Even if everyone offers the same product, people remember who was best to deal with, who they can trust, and who they think they’ll benefit from dealing with again. The answer, the CTO said, lies in building that kind of brand. That makes sense too. To apply that idea of scarcity
An Industry Structure Polarizing Like K-pop 41:45
Chester Roh to what’s happening in companies, what kind of market should we consider to be over? The market where you can just do what you’re told without thinking, push buttons, and get paid for it. That market is over. Artists used to be needed in every neighborhood, and they all made a good living. But as technology advanced, the market expanded. An artist in Seoul and one in New York now compete in the same space: the internet. As those connections grow stronger, thanks to technology, markets don’t stay local. They merge into one much larger market, with more participants, but they don’t all make money. Only blockbuster hits make money. And because those hits earn enormous amounts of money, the market’s expected payoff rises. So even people who aren’t earning yet enter, dreaming of becoming stars themselves. That’s how the market takes shape. Every industry is becoming like our K-pop entertainment industry. The market grows, and stars make huge amounts of money, while aspiring stars pile up in vast numbers. But only a few make money, while many earn nothing. I think every industry will eventually take on this shape. Jonghyun Park, you’re in the U.S. now. Whenever I visit San Francisco, I feel its distinctive atmosphere and pace. It must be hectic. Tell us about anything interesting
The Frontier Ecosystem Seen from the SF Front Lines 43:30
Chester Roh that’s happened while you’ve been there.
Jonghyun Park I arrived in SF yesterday, and I’ve been trying to take in as much of the frontier as I can while exploring. From that perspective, let me share what I’ve observed. This is the person who created Jev. I heard presentations from people like this and learned what drove them to build Jev. What Jev’s creator said connects to what we discussed earlier, so let me share it. Today’s LLMs are all trained to help people. But that alone ultimately won’t let them grow much. So rather than just helping people, we need to build something that works fully autonomously, even without people. That would open up enormous potential for growth. They said they created Jev to automate things. I think they were also considering how to move further up along similar lines. They also told me that inference demand isn’t keeping up with current CapEx investment. But from what I’ve seen, looking at where we are now, why is there so much CapEx investment? Inference demand far exceeds existing capacity, so everything built sells immediately. I’ve seen this happen again and again. I went to an event hosted by an inference company called Modal. It was a conference. The people you see here were speakers in one session. Several sessions like this ran simultaneously throughout the day. Founders of well-known companies we all recognize came to speak. What struck me was how valuable inference companies are. They have a lot of money, too. So many prominent people and companies want to work with inference companies that they’ll gladly attend events like this and spend their time. That’s the impression I got.
The Dawn of the AI Film and Video Generation Market 45:24
Jonghyun Park I also met people from many companies there, especially startups. For example, I met people in the current YC batch and a great many inference providers. I didn’t meet them because I set out to meet inference providers. I’d just ask someone beside me as I walked around, and they’d say, “We do inference.” Whenever that happened, I’d ask, “Is it harder for you to sell inference or to buy GPUs?” They’d all say, “Buying GPUs, of course. If we could get GPUs, we could sell all the capacity.” Almost everyone answered that way. And inference itself is incredibly varied. It’s not just companies hosting LLMs and selling inference for them. Take Dreamscale Labs, a company in the current YC batch. They want to sell inference for VLAs or world action models that run robots. For companies like that, response time matters, so they’re looking for GPUs in San Francisco.
Inference is also branching out into specialized uses like these. There’s plenty of inference for video generation models, of course. Some build another layer on top and say, “We’ll provide video generation model inference for film studios. Or we’ll focus on robotics, like those companies.” By adding these layers and offering a fuller package with more value, they seem to be trying to compete in the inference market. I met many different providers like that. Services like Vast are doing this too. Weights & Biases and many other inference companies hold events to bring in as many people as possible. I heard similar things at OpenRouter. Then there’s an inference provider called Fal, and LTX, a company known for its video generation model. They hold events together to bring people in. I could see that these companies were working as if their lives depended on it. That was a strong impression I took away. Turning to video, I’m personally very interested in it. There was a screening of films made entirely with AI. I found out about it,
and it sounded fascinating, so I went. I watched all the films there. Of course, they weren’t perfect. But some people were crying. That surprised me. From the audience’s perspective, I thought there could be demand for this. People might actually make and screen films this way. That’s what I came away thinking. And to take that a step further, at events like this, I sometimes meet people and discover that quite a few AI-made films are already listed on sites like IMDb, complete with ratings. I had no idea. They weren’t Korean films; most were Chinese, but I came across quite a few. And speaking of making films like that, these days, there are short dramas, or microdramas. That short-form microdrama format is selling really well. That’s one trend in content, and a lot of it is being made with AI. People there also told me that film studios just don’t talk about it, but every studio generates scenes with AI. They all do it—that was the impression they gave. I can’t verify whether that’s true. Model companies, too, boast, “We have film studios as business customers, and here’s how we meet their needs.” They really boast about that. So, video… Yes, go ahead.
Chester Roh No, even that Manager Kim drama that was popular recently— all those scenes set in North Korea were made with AI, right? And there are things on YouTube like StarCraft movies. I also think Koreans would have a lot of strengths in this area. They post series like that, and I watch them too. They’re fun, and the quality is excellent.
Jonghyun Park So I showed some of the work being done there. I think that, for it to have value as a real film, you can’t let anything awkward give it away. That seems to be how they see it. Even when shown a side-by-side comparison, you can’t spot the difference in details like what happens when someone steps on undergrowth and how the plants react. They’re tuning even details like that as part of their business and research. And since video editing is what interests me most right now, when it comes to video editing, Descript is the service with the highest revenue and is very popular. So I had a chance to visit the company and talk with them. I got to speak with the CEO, and I met an engineer there too. I’ve been in San Francisco for almost two weeks this time, and perhaps the most valuable thing I learned is that they need models that can transcribe audio with precise timing. Even 0.1 seconds of error is too much, and there aren’t many models like that. Only a few are open source, too. Companies offering these B2C services are researching such models themselves and developing them in-house. And with those models, only the papers are public; the weights aren’t open. They’re building models solely for internal use. The leading players are in the consumer market. To open up the consumer market you mentioned, they’re refining their AI internally to this degree. I got to see things like that. Everyone seems to be working hard
Physical Intelligence and 1X’s Scale-Up Strategy 51:20
Jonghyun Park to open up the consumer market. Moving on to robotics, I also heard from robotics companies at an event hosted by Modal, which I mentioned earlier. I sat in on their sessions. Those sessions will probably be on YouTube soon. They’ve been posting them one by one. The founders of Generalist, Physical Intelligence, and 1X all came out and spoke. My impression was: they’re deploying and scaling up. Almost all of them talked about scaling up. “How did we achieve this scale-up?” It’s different from scaling up a language model. Because video files are enormous, how can you collect high-quality footage from diverse environments to train a model? Physical Intelligence, for example, wants to deploy in as many industrial settings as possible and collect data as the robots operate there. Generalist runs something like a data factory, with factories around the world. They discussed how to bring all that footage together in one place. They talked a lot about things like that, and this field seems poised to open up soon. Even the physical side of AI. 1X’s humanoid Neo is well known, and I had a chance to see it in person at events like this. At the Physical Intelligence session, one of the cofounders came down afterward and was just walking around.
So I stopped them and asked some questions. I had opportunities like that. It was my first time coming to San Francisco for work, and I didn’t have a big network here or anything like that. So I wondered: could I join conversations, get into the inner circle where people share that information, and have worthwhile interactions? I came here with those doubts. But once I arrived, I found more opportunities than I’d expected. And the group we call the frontier seems more loosely knit and open than I’d thought. Anyone who wants to can approach them, learn something, and pursue what we want to do. That’s what I’ve been feeling.
Chester Roh Yes, exactly. If you go and say, “Introduce me to someone!” people really do make those introductions.
Seungjoon Choi Didn’t you mention in passing that you didn’t want to come back? That you wanted to stay?
Jonghyun Park This is how I’ve been putting it here: when I first thought about going to San Francisco, I was heavily influenced by people around me. A lot of people told me to visit San Francisco, saying I’d learn different things by interacting with people here and that the people were different. I heard that so often that I arrived feeling a little skeptical. What could really be different? After all, we can use AI to whip things up and work online anyway, so what difference would physically going to San Francisco make? That’s what I thought. But San Francisco is different. I call it the “San Francisco is different syndrome.” I’ve ended up catching that syndrome. For now, I’ll make the most of my remaining time here. I’ve met people here that way, too: some watched this video and contacted me in San Francisco, and we got together to talk. So, in case anyone else is interested, Before I leave San Francisco, we’re holding an event here on using a video LLM to make videos. I met a talented founder here, and we decided to host the event together. If you’re here and interested, please come…
Seungjoon Choi What’s the event called?
Jonghyun Park It’s called Video as Code. Rather than use a video generation model to make videos, we’ll use an LLM to understand footage and write code. Then, Seungjoon, we’ll use it for the Video as Code approach you discussed two weeks ago to make really good videos. I met someone here who’s doing that well, so we’re planning a session to talk about it together.
OpenAI’s Math Research Release and the Mathpocalypse 55:23
Chester Roh All right, then. Let’s get back to the frontier for a bit. A lot happened this week. Mathematicians may have had the saddest week of all. OpenAI released another major result, so let’s hear the story behind it from Seungjoon.
Seungjoon Choi So much happened that the term “Mathpocalypse” started making the rounds. To run through it quickly, on October 6, OpenAI released 722 papers all at once. That sent ripples through the mathematics community. Mathematicians had seen it coming, but when their AlphaGo moment actually arrived, their raw reactions poured out for days. What’s interesting is that, besides the official blog post, the work is available on GitHub. A lot is public, including all the papers. Of the 722, three have now been withdrawn because they contained errors. But I found this part interesting. It says that, as part of model development, they use open research problems to evaluate their models. On existing math benchmarks, model performance had saturated, so they expanded this evaluation. So rather than pursuing mathematics for its own sake, I got the impression they were using math as fodder to improve their models. There are 722 manuscripts, things resembling papers, but the lines of research behind those results number about 372. And the open problems they gave the models to solve numbered around 4,000.
So the models didn’t solve everything they were given. Another much-discussed example was the quasi-Riemann problem. I don’t know exactly what that is myself, but they didn’t solve the Riemann hypothesis. They solved problems that were something like steps toward it. So they didn’t solve every truly difficult problem. Then, looking at the areas of mathematics involved, such as number theory and other fields, the work spans about 17 fields, covering roughly 40% of mathematical fields. That’s its reach in terms of fields. And apparently there are many open problems. There are probably few problems at the very top; Millennium Prize Problems, for instance, are rare. But among problems currently considered unsolved, nearly 90,000 are listed in MathDB. So the scale is enormous, though this work didn’t cover everything. That’s what I found when I looked into it. Then, on August 28, internal model training resumed—it didn’t begin then. Training resumed on August 28, and a Navier-Stokes result was announced on September 8. Fields Medalists issued a statement on September 11. By September 21, more than 100 problems had already been solved, Before announcing this, they said they’d seek advice from an advisory panel. After consulting prominent mathematicians, the panel made its recommendations on September 29, and the announcement followed on October 6. So things moved quickly.
The Mathematical Community’s Shock and the AHM Statement 58:53
Seungjoon Choi As for the mathematics community’s response, various statements came out. One of the more extreme statements came from AHM: mathematics should be done by humans. It says, “We oppose AI doing mathematics,” and is collecting signatures. It has gathered quite a few. The advisory panel also said, “Although we provided advice, flooding the mathematics community with results like this is deeply unhealthy. Especially when researchers can’t access the internal model, getting ahead of them like this and creating a gap doesn’t help at all.” The panel even suggested that this should stop in its official remarks. But what impressed me most was that, in my view, the mathematics community has responded in a healthy way and taken in a wide range of views. There’s this site called Prompts, and its full name is interesting: Proofs and Prompts. It launched in August and seems to discuss and test automated proofs and other topics in AI and mathematics.
And on that site, there’s a collection about the more than 100 solutions: more than 100 responses to them. It gathers reactions from mathematicians and mathematics students, including prominent mathematicians. These are their responses to the event. I tried tagging them. Awe and excitement: 36; loss and sadness: 22. I used AI to classify the reactions. There was optimism and hope, calm observation, reactions from early-career researchers, the access and inequality issues I mentioned earlier, and then the manuscripts— things resembling papers— which are apparently very hard to read. They hadn’t gone through peer review like typical math papers, so the writing was unrefined and very difficult. Why should we do unpaid work to decipher them? People raised unpaid work and the burden of verification. Others said we should still read and digest them together, or that we should ignore them. There were all kinds of views.
Chester Roh What does that AHM statement say? Does it say this should stop?
Seungjoon Choi No. It says AI doing this in mathematics is unacceptable, and that they’ll respond firmly and take legal action, or lobby. There was some talk along those lines.
Chester Roh Right. I’m not sure how effective that would be. Isn’t this impossible to stop?
Seungjoon Choi Some of those 100-plus responses expressed the same resignation. So I set up ChatGPT’s event feature to track this. Are people really refusing to do that unpaid work? By unpaid work here, I mean reading, refining, and studying the papers. But it keeps happening. I even got a report at 3 a.m. today: people keep reading them. After all, this is a field they’ve devoted their entire lives to exploring, so it would be hard not to be curious. So they’re actually reading the work, They’re discovering things, finding gaps, and making improvements. That’s happening constantly right now.
Math 2.0 and the Reconstruction of Mathematics’ Identity 1:02:41
Seungjoon Choi But what worries the mathematics community is the training of young mathematicians and the broader ecosystem of mathematics as an academic field. The prevailing view is that these are under serious threat, so concerns about them are being discussed publicly. I also found videos showing some emotional reactions. Terence Tao has started talking about Math 2.0. For several days, he used his blog as a public forum for a range of opinions. Then early this morning, he shared slides from a recent presentation at Caltech. Even before the October 6 announcement, Tao probably knew about it, because he mentioned Math 1.0 and 2.0 and talked about proposals for how we should move forward, and how this is spoiling things for mathematicians, like spoiling a movie. He made remarks that reflected those concerns, then went quiet for a while. Early this morning, he posted about Math 2.0 on Mathstodon. Among the slides he shared there, this one has once again drawn criticism.
It uses an analogy with cancer treatment: Shouldn’t we fully understand a treatment before using it? But in medicine, if clinical trials show that something works, even if we don’t fully understand how it works, if it helps patients, we should keep making progress. People are interpreting and citing the slide that way and criticizing it. On my timeline, I’m seeing comments that mathematicians are a little arrogant and are facing a backlash. The software industry seems somewhat unsympathetic. There’s Tristan Buckmaster, from back then, around the Navier-Stokes case. He raised concerns that his research had been overtaken or possibly used. People are citing a video of him saying that to criticize mathematicians. Isn’t pursuing truth what matters, rather than protecting mathematicians’ livelihoods? That’s the kind of criticism we’re seeing. In the software industry, even people like Eric S. Raymond, who wrote The Cathedral and the Bazaar— I didn’t realize he spoke this bluntly— say, “We accepted this two or three years ago, dealt with it, and moved forward. You should do the same.” Kent Beck said something similar. Having felt the impact earlier, the software industry seems somewhat unsympathetic.
Jonghyun Park Math 2.0 reminds me of Karpathy’s Software 1.0, 2.0, and 3.0. He was talking about how software’s structure changes quite a lot, I guess, and used those numbers to describe it. Terence Tao also seems to be saying that mathematics is changing structurally in some way, and offering his thoughts on what we should do. It sounds like people have mixed reactions.
Seungjoon Choi He’s often said that what matters isn’t the proof, but the understanding gained while pursuing a problem. He’s said that a lot since Navier-Stokes. So for now, things feel chaotic and confusing, All kinds of emotions are coming out, but the mathematics community is opening up a public discussion. Compared with the software industry, they seem to be handling it well, in a way. They’re making room for people to talk about those feelings and bringing in a range of views. So when it comes to our identity or sustaining our profession, and what we should do about it, they do seem to be drawing on what the software industry and the Go community have already been through.
Chester Roh Yes, but in the field we call research, not just mathematics, we often talk about what comes next, don’t we?
Seungjoon Choi Earlier, we talked about Fields Medal winners and other prominent mathematicians finding that their research problems had already been tackled, feeling disappointed but curious at the same time. It’s a complicated situation. Since I can’t fully understand the deeper mathematics myself, I mainly looked at the reactions on the surface, and this stood out:
Public-Key Cryptography and the Lowered Barrier to Attack 1:07:20
Seungjoon Choi Encryption could be at risk. Vitalik Buterin posted about it this time, too. I don’t know this field very well, but some people in it say the number theory problems tackled this time and related work could pose a threat to RSA and public-key cryptography other than quantum computing— a threat from AI. So I get the sense that people in that field are on edge.
Chester Roh Right. A new approach could overturn some things we thought we knew. Those algorithms are built on the assumption that the mathematical calculations would take far too long. But a shortcut might emerge and reduce the algorithmic complexity. It wouldn’t mean an immediate breach, but the cryptographic algorithms themselves could turn out to be surprisingly simple and potentially be broken.
Seungjoon Choi The recent attacks on Korean banks drew attention overseas. The attackers used DeepSeek, and left traces of that use. In any case, the barrier to carrying out those kinds of attacks is falling, too.
Chester Roh Right. Every problem is turning into a search problem you can throw computing power at. The era of “Do it for me, do it for me, do it for me” is coming.
Seungjoon Choi But the mathematics community is in turmoil and discussing where to go from here. Many people are also saying this could be a chance for the field to become healthier. In academia, papers have served as a proxy for achievement. That’s not what we’re actually trying to achieve, but in trying to measure achievement, we’ve ended up, as Goodhart’s law suggests, evaluating it in strange ways. People are saying this is a chance to set that right. And this goes beyond mathematics: with the output of papers and similar work exploding, it’s becoming an issue across academia. But OpenAI’s actual interest lies elsewhere entirely— at least, that’s my interpretation. It simply wants to improve the model’s capabilities. For that, it needs better, more difficult problems. So what Noam Brown and Mark Chen cited wasn’t about mathematics. It was this graph again. Back with Astra, when it was an internal model. And it doesn’t end there. It keeps going. What’s interesting this time is that Navier–Stokes was in early September. That took 10,000 agents and 88 hours, as we discussed in detail before. Now, in early October, just a month later, ChatGPT Pro Thinking takes three hours per problem. In other words, the roughly 372 problems we discussed earlier were solved in about that time on average by a single agent working for around three hours. So there’s a lot of speculation about what that implies. Put simply, it would mean that capabilities gained through RL improved that much in just one month. As for Navier–Stokes, this is something I and others are speculating about: when those difficult problems were solved, the 10,000 agents must have produced excellent solution paths. Those paths then become data for the next round of training.
Jonghyun Park The first breakthrough takes that many resources, but once you know the way, following it isn’t difficult. I think that’s one way to interpret it.
Seungjoon Choi Right, you find a promising direction and ideas you can combine. If you look at mathematicians’ blog posts, they describe it as combining ideas within the convex hull. They use the phrase “convex hull of ideas.” So it seems to work by combining ideas. Not every innovation works this way, but some innovations connect different papers… That’s exactly a pattern we see in mathematics. Most mathematicians find it hard to know fields outside their specialty in depth, but today’s models are very good at surveying the literature and making connections. Among those 100 reactions, quite a few were along the lines of, “It connected this with that to do this?” This is Patrick Toulme, who works on RSI at Google. He works on Gemini RSI and previously worked on TPUs. He posted this around yesterday.
The RSI Loop and Rapidly Rising Model Capabilities 1:11:55
Seungjoon Choi He says every lab is running an RSI loop, and gains from RL greatly improve math and coding abilities, but too much RL hurts language ability. It has a negative effect. I wonder if that explains why some outputs, like the earlier one, are hard to read, or why the writing still isn’t very good. It makes me wonder if those things are related. There are also comments about bots, saying the response in Silicon Valley seems fairly positive. And then there’s this part. Fable 5.5 hasn’t come out yet, but some people are reportedly using it through routing, and the reactions say it’s quite good. As for OpenAI’s Bel, the word is that it’ll be even better in December. Someone on the Gemini team is even spreading that rumor. That’s the kind of talk going around. It is just a rumor.
Chester Roh Yes. I think we’re gradually getting used to this pace.
Seungjoon Choi Anyway, to share one more rumor, there’s also talk that Anthropic recently started its next pre-training run. That’s what I’ve heard. So they’re continuing what they’ve been doing.
Jonghyun Park Gemini 4, too. There are events here where they show things privately… Anyway, they hold a lot of events here where anyone can go and show something. They also gather major customers privately to make a more detailed pitch, apparently.
Seungjoon Choi Gemini 4 hasn’t fully released Argon yet, either, but Perhaps because internal feedback said it wasn’t good at coding, Barium-B came out, and now they’re training Carbon. So the Argon checkpoint was mentioned only recently, but the checkpoints keep getting better. So that’s the common pattern right now.
Chester Roh Just do more RL. Until it works. And if a good checkpoint happens to emerge, train it further and release it as a model.
Seungjoon Choi This was posted by a Hugging Face researcher. Hugging Face says, “Thank you,” and then uses OpenAI’s GitHub as an RL target. That’s what they’re doing. But companies building RL environments have been posting a lot about this lately. “We do that too.” It seems a lot of people run businesses providing RL environments for big tech companies.
Chester Roh Companies selling RL environments and RL datasets are making a lot of money.
Seungjoon Choi As I briefly mentioned earlier, the convex hull of ideas— if you have points like these, you draw along their outer edge. Here, a line forms between these points. That’s how you make a convex hull. One hypothesis among mathematicians is that it only goes this far and doesn’t create an entirely new, different axis. Even so, the new points that emerge within it can still be remarkable and surprising. That’s one hypothesis, but some on LessWrong disagree: Seen as a convex hull, it may look like you’re only combining points inside it, but you can extrapolate too. That’s the argument. So expanding the boundary may be possible. That’s being discussed.
Chester Roh Right. But no one points out that the boundary isn’t actually defined. Nobody really knows where it is. AI’s boundary and humanity’s may be different.
Seungjoon Choi So I also wanted to understand these concepts, so I did some tinkering. I made an interactive way to see what a convex hull is and how it works by playing with it through interactive storytelling. So, the idea of a convex hull— “convex” has several uses in mathematics, and this is also called convex. But this is what Nassim Taleb calls convex tinkering. Sometimes you take a loss, but when there’s a chance of a gain, you cap the loss, much like with a call option. When you try many promising things, you limit the losses, and when one succeeds, you maximize the gain. So, as Chester Roh said earlier, if something interesting comes out of one attempt, there’s a chance it could more than make up for all those losses. So I also explored concepts like those fat tails. That’s something I tried. But this was something I made to understand the concepts myself. About the current mood, François Chollet made another interesting, somewhat negative point—
François Chollet’s Counterargument That the Intelligence Explosion Will Not Come 1:17:14
Seungjoon Choi that’s his usual style. He wrote a post pushing back a bit: recursive self-improvement wouldn’t lead to an intelligence explosion. So that stuck with me, too, and I translated it so I could discuss it with the model. Chollet’s view is that science already works like RSI, improving itself over time, but the history of science shows that more input doesn’t necessarily produce proportional gains. Input itself has to grow exponentially for progress to be linear. So with this kind of improvement, even if it amounts to RSI, an intelligence explosion wouldn’t happen. That’s one argument. And then there’s this: Chollet earlier cited work by Michael Nielsen and Patrick Collison, who studied scientific productivity in 2018. He cites their work to make that point. But Nielsen himself recently cited this: something Vernor Vinge wrote in 1993, “The Coming Technological Singularity: How to Survive in the Post-Human Era.” Citing that 1993 piece by mathematician and science fiction writer Vernor Vinge, Nielsen said the future feels like an opaque wall. I drew a picture of this earlier: the farther into the future you look, even assuming superintelligence comes about, the more you face the science fiction writer’s dilemma.
It used to be easier to predict what the future would look like. But as evidence of these technologies emerges, and enormously powerful technologies appear, imagining what people could do with them in concrete terms feels as though there’s a wall blocking the view. Michael Nielsen cited that and wondered whether we’re in such a period now. People seem to be talking a lot about the things I’ve kept citing all this time. We’re seeing plenty of evidence now, yet perhaps we still can’t predict very well what will happen. I wanted to bring that up. Was it already around this time last year that Seonghyun Kim and I talked about Hinton? The fog—the fog thins exponentially. You can see what’s close by, but it gets opaque when you look farther ahead. My head has been spinning lately, too. Chester Roh, you said earlier that you’ve grown calmer about these things, but I decided, in my own way, to weather news like this. And right now, that’s really hard.
Chester Roh I do keep up with it, but as for where it’s headed, I think this is a time to take a broader view. That’s how I see it. Things are moving too fast to focus on the details.
The Values Education Must Preserve and the Constraints on Experimentation 1:20:17
Seungjoon Choi So lately, when I meet people, the concern I share is this: what does all this mean for education, the field I’m closely involved with? It’s hard to work out. For one thing, these developments aren’t affecting everyday life yet. News that some model can do something today doesn’t immediately affect education. But what worries me is that education has always been good at having students spend their own time and use their own minds to practice something, without skipping that work or delegating it all to AI. If we say we should preserve the value of getting good at what we do ourselves, that’s what education has always done. So these changes happening around us There don’t seem to be many ways in. That’s because they could actually undermine the original value of education. But something is clearly happening. So, on one hand, it feels a little unsettling, as if there’s a wall here blocking my view of the future. Even as I try to understand the times we’re living through, I still can’t see where this is going.
Jonghyun Park Since you brought up education, let me add one thing. A friend who wants to co-host an event with me works in education. They explore how AI can improve education. Broadly speaking, they’re following the same path as 3Blue1Brown. But they want to do it better with AI. From what I’ve seen, this friend really has a noble goal for education. That’s what I gathered from talking with them. So perhaps AI-powered consumer apps could make an impact there, too. It feels like they’re only now starting to emerge. That’s just my personal impression.
Chester Roh What Seungjoon Choi said reminds me of something: The best way to predict the future is to create it. Right or wrong, I can pursue the future I believe in and keep telling everyone, “This is the way.” Maybe that’s how it becomes the future in times like these.
Seungjoon Choi Experiments that help us understand what’s happening— earlier, when I mentioned convex tinkering, I said we should quickly rule out failures and run many experiments on promising options. But in some fields, failure can be devastating. So epistemic action itself— running experiments to learn what’s happening— compared with other fields, is constrained to fewer sample points, it seems. And I wonder if education is one of those fields. That’s something I’ve been thinking about. I haven’t sorted it out yet, either. Still, Nielsen has been saying lately that some constraints are predictable despite that. Nielsen has made that point, too. With obvious factors like energy and resources, if you account for those constraints when predicting, you can see some of what’s coming. And I felt that connected with what Chester Roh said earlier. There was some overlap there.
The Pope’s Remarks on AI and the Discussion of Model Consciousness 1:23:28
Seungjoon Choi Then there are all sorts of other things. The Pope has been tweeting a lot lately—His Holiness.
Chester Roh Oh, really? I see.
Seungjoon Choi Some posts are about AI consciousness and related matters. Apparently, Anthropic has met with many religious figures. Chris Olah and others have. That’s been going on, and recently, on October 8, Anthropic updated its Usage Policy: You must not make remarks that seem abusive. Otherwise, you won’t be granted access. The context for those discussions, religious responses, AI consciousness, and what any of it has to do with Anthropic’s IPO— there are several threads to follow. So that’s one thing that’s happened.
Chester Roh Let me ask you both a question. Does this thing have a soul? What do you think?
Seungjoon Choi A soul? Well, I don’t think it has a soul, at least not yet.
Jonghyun Park I think it does.
Seungjoon Choi Oh, really? We’re talking about consciousness here, not a soul.
Chester Roh Okay, then I’ll use the term consciousness and lower the bar a little. Could a model have consciousness?
Seungjoon Choi Functional consciousness is what Anthropic claims it has, but I don’t use it for counseling. I don’t turn to it for empathy or anything like that. I use it to get help, so I keep some distance from that aspect.
Jonghyun Park I think it does. But rather than simply saying it does, think about what a soul or consciousness actually is. They’re concepts people have created— abstract concepts, really. So I don’t think they’re really some… as religion might describe them, intangible, sacred presence. I don’t think that’s what they are. That whole series of thoughts— the processes that happen in the human brain— is simply what we’ve given those names. We can’t define them precisely in words, but people share a sense of what they mean: particular thought processes or interactions that AI can carry out like a human. That’s what I felt when I watched people cry over the AI film we discussed earlier. I felt something similar. Even though AI made it, if it can interact emotionally with people that powerfully, perhaps that touches on what we commonly call a soul. That’s how I see it.
Chester Roh My slightly strange answer is that they’re isomorphic, differing only in their makeup. Only the substrate is different; I think they’re isomorphic.
Seungjoon Choi But… What’s interesting is that when the current Pope, Leo XIV, took office, official Vatican materials discussed the Industrial Revolution. Leo XIV himself studied mathematics as an undergraduate. He later studied theology, speaks five languages, and seems exceptionally smart. When he took office, he spoke about the Industrial Revolution and said his mission was to consider what religion and Catholicism should be in the AI era, and what humanity should be. He said he would take on the role of grappling with those questions. So lately, statements that art should belong to humans and things like that have kept coming from the Pope.
Chester Roh Another interesting thing—this is a different topic—what was it? Grok has been added to Tesla cars. So when I’m in the car with FSD on and I talk to Grok, it responds with very low latency. It handles function calls well, too. If I ask it to analyze something, it makes all the necessary calls, analyzes it, and brings back the result. While I was doing that, my family was listening quietly beside me and said, “Hey, put this in Optimus and you’ve got a person.”
Seungjoon Choi That’s what Tesla is trying to do.
Chester Roh That made me feel this is how ordinary people’s perceptions begin to change. Hearing that, I realized we’re immersed in model benchmarks, model sizes, and things like that, so we don’t notice those gaps, but people outside that world Whenever the values they see and recognize change, they take small leaps forward, step by step— that’s how it struck me. It’s advancing so fast.
Seungjoon Choi Anyway, lots of apps are being made and reverse engineered, and I see all that on my timeline. Ultimately, I think it comes down to the model’s capabilities. As the model gets more capable, of course, that doesn’t apply everywhere. It applies in some areas and not in others, but where it does, it’s working remarkably well and will get even better. Earlier, with test-time compute, the graph of the internal model’s improvement didn’t end there. We’re now seeing rapid growth even within a single month, in some respects through a recursive process. So by the time it comes out at the end of the year, it’ll be a different story. But the end of the year isn’t far away.
Chester Roh Even so,
Everyone’s Turn Comes 1:28:53
Chester Roh we have to find our own work to do.
Seungjoon Choi Daily life doesn’t change all that much.
Chester Roh And in times like these, what matters ultimately is asking the right questions— finding the right question. Being able to do that is a rare ability. As we run these thought experiments, we need to ask plenty of wrong questions to stumble upon the right ones along the way. I think that’s the right process.
Seungjoon Choi We talked about this briefly, but mathematics has had its AlphaGo moment. Its turn has come. I think everyone needs to expect their own turn to come, whatever their field.
How do you know your turn as an entrepreneur isn’t still to come?
Chester Roh It will, soon.
Seungjoon Choi Mathematicians all sensed it was coming, too. But once it happened, it felt completely different from what they’d expected.
So we don’t know what form it will take.
Chester Roh I can feel that these days, we’re gradually getting less caught up in all the hype. So we’ll keep observing what’s signal and what’s noise. All right, then, we’ll wrap up today’s recording here.
Jonghyun Park Thank you.