EP 105
From the GPT-5.6 Launch to ICML
Intro to ICML Week and the Surprise GPT-5.6 Announcement 0:00
Chester Roh Today, as we’re recording, is July 11th, 2026, a Saturday morning. This week was ICML week, so everyone was incredibly busy. In the midst of all that, GPT-5.6 was also unveiled without warning, so today, we’ll talk about GPT-5.6, take a look at what happened, and also, in relation to ICML, we hosted various events and met a lot of people, so we’ll briefly share our impressions with you. Then shall we start with GPT-5.6? It was a hectic week. First, once word got around that this was coming out,
The Mood Around GPT-5.6 Before and After Launch and Early User Reactions 0:31
Seungjoon Choi whenever there’s a major event, others always try to tag along. So everyone was trying to make their presence felt: SpaceX xAI introduced Grok 4.5, and Muse Spark bumped 0.1 up and introduced it as 1.1. But Gemini 3.5 Pro was supposed to come out in the summer, and now that early July is passing, it still hasn’t been released. So if you looked at the timeline on Twitter, the people working on Google AI Studio and posting about Gemini every day— Logan’s expression seemed rather sad, or at least that was the impression I got.
One interesting point was that Sam Altman and the official OpenAI account were rumored to be announcing it on Thursday. The rumors had been circulating for a while. People kept saying Tuesday, then Wednesday, but once they said it would be announced on Thursday, people finally started flooding the timeline with their reactions after trying it. So I looked through the reactions from prominent figures
on Twitter, and what was interesting was that some of them had been using it in advance for as long as two months. Of course, this wasn’t subject to export controls, though they said it was controlled by the government, but it showed that they had distributed it to quite a lot of people. The common reaction, though, was:
it doesn’t have the same big-model feel as Fable 5, but it’s very good; people commonly described it as their new daily driver. So there definitely seems to have been a performance improvement in coding. Have you tried it? I’ve used it a little in Codex.
Codex’s Granular Effort Levels and the Felt Difference in Cost Deduction 2:14
Jonghyun Park You mentioned that it’s a new daily driver, but even though I’m using it through a subscription, I felt that it burned through my allowance fairly quickly.
Seungjoon Choi What did you set the effort to? For effort, with Sol, the largest model,
Jonghyun Park they added High, Max, and Ultra, right? Anyway, something appeared at the top, like in Claude, so I just clicked each one from the top down and tested them all.
Seungjoon Choi I see. I only tried it at around High.
Chester Roh Right. There are something like six levels now, so deciding which one to choose leaves me feeling very conflicted. It definitely feels like OpenAI is putting a lot of effort into test-time compute. They’ve done a good job of breaking it down into granular levels too. Right. So we need to learn about those things,
Seungjoon Choi and humans now need to develop an intuition for which effort suits their work in order to use it cost-effectively. At least, that seems to be the era we’re in. For now. And then they also unveiled GPT-Live without warning. The day before, they used GPT-Live during the GPT-5.6 Sol announcement, even using it for simultaneous interpretation at the very end of the livestream. So GPT-Live was released in a full duplex format, and I tried this too. I don’t know whether I’ll keep using it, but it was definitely better than before. So this enables simultaneous interpretation during a conversation. Of course, during the livestream—the GPT-5.6 livestream— it did mess up once. That was an interesting moment too, but in any case, it’s here. So in the startup scene, I occasionally saw posts saying that people who had been working on this kind of thing were devastated overnight. Those kinds of posts did pop up here and there. They had been trying to crack this problem, but when it came out in a full duplex format, it seems to have put some of them in a difficult position. That’s the real problem. As for the news related to GPT-5.6, I don’t really examine the graphs
GPT-Live with Full Duplex and Simultaneous Interpretation 3:17
ChatGPT Work Separation and Signals of Recursive Self-Improvement 4:10
Seungjoon Choi or things like that in detail anymore. One interesting thing, though, is that when you open the web interface, it’s divided into Chat and ChatGPT Work. So for things like research where you use effort and have multiple agents working in various ways, ChatGPT Work is the right choice, while Chat responded very quickly. Even after I updated the app, the options for adjusting effort or changing the model weren’t immediately visible. There may be a way to do it, but the responses were extremely fast. So it seems it’s probably designed to be used like Instant, the way we used it before, but we’ll need to learn more about it. One thing I found interesting during the livestream
was an anecdote related to recursive self-improvement. It was something we’d mentioned several times: on October 31st last year—or rather, the 30th— they said they would introduce an automated AI research intern on the 30th. In September, in September 2020, and then in March 2028, they said they would create a fully autonomous AI researcher— an AI researcher that is itself AI. During this announcement, they showed Sol, with the smallest model, Luna, being post-trained. I’m not sure what kind of post-training it was. I don’t know whether they meant SFT or RL, but they redacted the sections containing important assets and the sections containing figures. They also revealed part of the prompt, and it was ordinary. But using that prompt, they showed Sol post-training Luna into something that could actually be used in practice. So there continue to be signals related to RSI, and now Lilian Weng has joined Thinking Machines Lab as a co-founder. Lilian Weng, formerly at OpenAI, wrote about self-improvement and harness engineering, organizing many of the things happening now, combining harness engineering with self-improvement. In any case, self-improvement can involve updating the weight, but Autoresearch-style self-improvement that improves the harness itself is also possible. Lilian Weng wrote a comprehensive post covering all of that, and it’s worth reading. I won’t go into detail now and will just introduce it.
Chester Roh Is the core message that we need to improve the weight, or that improving the harness is faster? Currently, the focus is on improving the harness,
Seungjoon Choi but signals from OpenAI suggest that improving the weight is also being done through a self-improvement loop. Since Luna was post-trained. So both seem to be happening. And Anthropic also gave a fitting introduction to that about a month ago. There was a post about it being time for AI for self-improvement to build itself. So if we jump straight into the pricing and speed, it was simply priced at the Opus tier. Right now, Opus 4.8 costs 5 dollars per million input tokens and 25 dollars for output, so output is a bit expensive. Output costs about 5 dollars more, which is how the pricing is structured. So while it’s priced at the Opus tier, what interests us is that this is included in the subscription plan. So for now, it’s still subscription-based. Thankfully. But there’s a rumor.
GPT-5.6 Pricing Structure and Key Watchpoints for a GPT-6 Subscription Model 7:04
Seungjoon Choi A rumor about GPT-6. The fact that it has already been out, and that people have been using it for two months, means it must have been available internally much earlier. So timing-wise, when we looked at the intervals between Claude model releases, there were signs that they were shrinking to less than two months. So there are rumors that GPT-6 is already quite far along, and the prevailing sense is that GPT-6 will be a large model, in the Mythos tier. That’s the general atmosphere.
If so, the key question will be whether OpenAI includes it in the subscription or switches to usage-based pricing. And it feels like that may happen sooner than expected. Right. Fable 5 also
Jonghyun Park extended the deadline a little, but I think there were signs that it would no longer offer a subscription plan. And to improve profitability, OpenAI will probably do the same, naturally, so I’m trembling with fear. Right. I heard that if you use it the way you normally would,
Seungjoon Choi you could easily spend $714 a day. So as you just mentioned, it was originally supposed to close on July 7, so everyone was testing the waters, trying to decide when to use it, but on July 7, they extended it. So people were like, “I stayed up all night because of this.” Things like that. So it became available until the 12th, which was what happened.
Another interesting thing is that after Sol came out, they did a reset. They reset the 5-hour and weekly rate limit, and Thibault appeared on the livestream this time. “I smell fear. Are you scared?” Because it would be a problem if people made an exodus to GPT-5.6. But it does feel that way.
We’ll have to see again on the 12th, but if GPT-5.6 is good enough, people will naturally make an exodus. So I think this is still a battleground. How the market will respond, and what will happen with GPT-6. Those were the key things to watch. While Seungjoon was speaking just now, the Codex app updated,
Codex App Integration and the RTS-Game-Like Feel of Agent Management 9:35
Chester Roh and the Codex app and ChatGPT app have been merged. Now it’s all just the ChatGPT app. The Codex app no longer exists separately.
Seungjoon Choi It did ask whether I wanted to keep the Codex icon.
Chester Roh That’s the only thing it asked. So I was also observing how people around me were reacting, and everyone seems to be playing a game.
Seungjoon Choi Deciding when to invest which resources, and which agents to use for which tasks feels like managing a game. When I looked into it, people had already been saying since early 2025 that this feels like playing an RTS, or like playing StarCraft, and those comments were fairly common. So you deploy tanks at a certain moment, though people of the current generation may not know StarCraft very well, and approach these things like a strategy simulation. So I tried generating that analogy as well, and when writing prompts, you get near-hit, miss outcomes, which gives you a dopamine rush. So it’s very often compared to a slot machine, but beyond that, now the question is what to do with all this, and how to manage it to accomplish the goal. So even choosing the timing of a rate limit reset or things like that becomes something you approach strategically. You look at how much you have left and decide whether to use this or that, whether to raise or lower the effort. And once everything is about this ready, you think, “Let’s make one big attempt with Fable 5,” and you end up devising strategies in that sort of way. But then people lose their minds.
Surpassing All Human AtCoder Competitors and Proving a Math Conjecture in One Hour 11:21
Seungjoon Choi I’m experiencing it too. Another interesting thing is AtCoder on July 9, which holds competitions involving heuristics and algorithms. There was an issue last year. Apparently, saiho is well known among people familiar with that scene, and within that field, saiho had still said, “I came in first.” That was around this time last year, in July. And that summer was when the news came out that OpenAI had achieved a breakthrough at the IMO. So AtCoder and the IMO, the International Mathematical Olympiad, saw breakthroughs around last summer, and so Sam Altman said, “Good job, saiho.” Sam Altman once gave this shout-out, and a year later, saiho posted again. So AtCoder has concluded, and OpenAI, although the prizes go to humans, OpenAI’s score was enormously higher than the human score. So in both the algorithm and heuristic divisions, the OpenAI agent achieved the maximum score of 8,300, though I don’t really know how it solved each problem. Even if I looked, it would probably be too difficult for me to understand. Anyway, it won by a wide margin, and then OpenAI, in the heuristic division as well, outperformed every human contestant. So in just one year, it became capable of solving top-tier coding problems, which it couldn’t do last year. And whenever a new math model comes out,
since OpenAI has people like Sébastien Bubeck, they talk about math, and this news came from someone else: they quietly uploaded something like this to a CDN. This is the Cycle Double Cover Conjecture, which, translated, would probably be called the cycle double cover conjecture, and it solved that too. So it hasn’t yet been properly verified by the mathematics community, but a math model has demonstrated its performance again, and it didn’t spend a very long time on this— it took about an hour. It wasn’t done with a publicly available model. It looks like they released the prompt too. So, using 64 sub-agents, for graph decomposition, flow theory, algebraic representations, induction, embeddings, and so on, it handled those things through an agentic workflow and, in just one hour, solved a conjecture related to mathematics, so I need to look into this too. I don’t know much about it, but it’s something I need to investigate. We’ve been talking about this with Jonghyun,
and Dwarkesh—was it 3Blue1Brown? The interview with Grant Sanderson covered many interesting topics related to mathematics, but the broader message it sought to convey through those specifics was also interesting. So I think it would be good for us to cover that next time. So in an era when these things happen automatically, what should human understanding and things like that look like? I’ll leave that question as a preview. What I’ve been thinking about lately is, last time, I said there were two schools: being outside the loop and being inside the loop. But it isn’t a dichotomy—you need to experience this, and experience that as well: doing something with an understanding of it, and then getting it done anyway without understanding it. I’ve been thinking about experiencing both sides, and that would also be good to cover next time.
The AI 2040 Translation Experiment and a Preview of the Next Discussion 14:29
Seungjoon Choi Lastly, on July 9, there was that famous prediction called AI 2027, remember? It was a forecast. Daniel Kokotajlo left OpenAI and created AI 2027 with Scott Alexander, and early yesterday morning, they released AI 2040. So they released it about an hour before GPT-5.6 came out. I need to read it too, but it’s extremely long. And since it’s in English, I simply gave GPT-5.6 Sol this link in Codex, had it clone it locally, and just said that I wanted to translate it into Korean. Then, 15 minutes later, it produced this. So it goes all the way through like this. The LLM didn’t translate it directly; it seems the LLM used some tool to translate it. I saw Python running and things like that. So now that the entire thing has been translated into Korean like this, I’ll read it. I don’t yet know what it’s about. But I’m sure it contains another interesting forecast. That’s everything I’ve prepared for now. Exactly. During ICML, researchers from the world’s top frontier labs
The On-Site ICML Atmosphere Where Being AGI-Pilled Went Mainstream 16:00
Chester Roh came, along with all kinds of media outlets and companies, and we met an enormous number of people. If there was one perspective they broadly agreed on, it was that almost all of them were AGI-pilled. People who thought this was a game that was virtually already won made up the mainstream, and a great deal of research— I think last time I interviewed Nikhil, a venture capitalist from Silicon Valley. When asked, “What did you find new or different at ICML?” he said that most people were AGI-pilled, and that much of the research, rather than focusing on model algorithms or things like that, focused on how to conduct evaluations better and how to advance evaluation metrics. I remember Nikhil saying there was a lot of research like that. That comes to mind.
And even amid all that, almost everyone working on LLMs is a scaling proponent, but there are fields trailing behind LLMs. A prime example is AI for Science, which has been drawing intense attention lately. After moving from LLMs to Physical AI, everyone is now trying to establish AI for Science as the trend that follows Physical AI, but that field still seems divided into two camps. There are scaling proponents and people who say, “That won’t work. Biology can’t be solved that way.” The two camps are clearly divided, so when they encounter each other at gatherings, they seem uncomfortable around each other. For the time being, like oil and water,
The Clash Between Scaling-First and Domain-First Views Around AI for Science 17:03
Chester Roh I don’t think those groups are going to mix. The scaling proponents’ perspective is extremely clear. They’ve seen far too many examples showing that, by investing in data, models, and more compute, the Bitter Lesson will continue to prevail, so there is a group that advocates that view. One defining characteristic of those on the other side is that they have spent a long time in their respective domains and only recently entered the world of AI, so they haven’t yet fully developed a sense of just how powerfully the community has experienced the Bitter Lesson over the past three or four years. But they have the problems they’ve solved and the experiments they’ve conducted, so there’s this attitude of, “These kids who don’t know anything come in, some lowly computer geeks come in and think it’s a problem they can solve by tapping away at a keyboard, but that’s not how things work here.” I noticed a clear divide between those two camps. I still don’t know which one will win.
Whether there really is something domain-specific. You don’t know?
Seungjoon Choi Haven’t you practically made up your mind already? Chester? I’ve made up my mind. I’m a scaling believer, but from an observer’s perspective,
Chester Roh I can’t recklessly say which one will win. Personally, I’ve bet on scaling, so when I see them doing this and that, I do have a sense that, “That’s a solvable problem.”
Seungjoon Choi But we said this would be the year of AI for Science, and yet there haven’t been that many signals. But it turns out that it’s already been nearly a year since Periodic Labs emerged, and they’ve been hard at work building a factory. To close the loop, they need to connect it all the way through to experiments, but they’re not there yet. But they’re also working extremely hard on RL. Periodic Labs, too. Right, things in the real world will take time.
Big Tech Booth Tours and a Hiring-Centric Expo Floor 19:38
Chester Roh Then shall we really get into our discussion of ICML? Jonghyun, since you visited many of the ICML booths, perhaps we could first hear what you have to say about them. I should have gone too,
Seungjoon Choi since it was an opportunity to attend a conference normally held abroad when it came to Korea, but I couldn’t make it.
Chester Roh The area around COEX went through quite an ordeal. There really were a lot of people. First of all, I attended ICML as well.
Jonghyun Park Since it’s fundamentally an academic conference, paper presentations are the main focus, but these days, the industry seems to have a particularly strong influence, so let’s first go around the sponsor companies’ booths and take a look at what kinds of companies have come in large numbers to observe and contribute to this field.
Seungjoon Choi But how did you apply in the first place? I was there as press.
Jonghyun Park I applied for press credentials in advance and signed a pledge of sorts before they registered me. In any case, there were an enormous number of people. They were also giving away lots of merchandise, so to get a hoodie, people were lining up in huge numbers. Naturally, all the big tech companies were there. OpenAI, Google, Apple, Meta, Amazon, Microsoft, and other companies were all there, and most of them operated in similar ways, holding lightning talk sessions where their researchers would come and share what they were working on, while researchers also rotated through and remained at the booths so that if people had questions, they could answer them when they came by. That’s how they were operating. One of our stars, Noam Brown, also came, and I saw photos of Noam Brown there, but unfortunately, I wasn’t there at the time, so I didn’t get to see Noam Brown. And when you go around and talk to everyone,
they do answer your questions to some extent, but everyone’s main objective was recruitment. Since it’s such a major global conference, there were especially many graduate students nearing graduation, and I felt that recruiting those students was their biggest objective. So the Google and Microsoft booths, and OpenAI as well, would say at the booth, “The current scheduled session is for Q&A.” They were all operating along those lines. Meta was there too, and Meta had its glasses on display, with sessions where people could try those glasses, and they were running things like that as well. Next, in a similar vein,
there was Naver, a company in Korea with a position similar to big tech, and Naver was operating in almost exactly the same way. Researchers from each field at Naver were all stationed at the booth, and if you said, “I’m curious about this,” they would do their best to connect you with researchers working on the relevant research so that you could speak with them. So at Naver, I also met someone working in robotics, asked how they were approaching it and things like that, and was able to learn a lot. Then, apart from big tech, there were also companies that could reasonably be called frontier labs, and Mistral in particular is French, but they were recruiting founding members in Korea. It seemed like they were planning to open a Korean branch.
Chinese Company Booths and the Presence of Chinese Researchers 22:27
Jonghyun Park There were also many Chinese companies, and in the case of ByteDance, people seem to really like Seedance these days. Since Seedance is so good at generating videos, they also unveiled a new model during this period, and Alibaba, as well as Xiaomi—Xiaomi also has a strong large LLM model these days, which people like, so they were there too, and they operated in almost exactly the same way as the big tech companies. What was interesting as I walked around and looked at the Chinese booths was that, since so many Chinese students had submitted papers to ICML and come to present them, I heard a great deal of Chinese. I strongly got the impression that they had opened booths at ICML to recruit Chinese people in China. The proportion of Chinese people really is very high.
Chester Roh There’s even a joke that the AI industry is being advanced by Chinese people in the United States and Chinese people in China, isn’t there? But statistically, that’s true. Strangely, there aren’t many Indians. In this field. I felt something similar,
Jonghyun Park because since ICML is an academic conference, the paper presentation sessions are the main sessions, and when you attend one, well over half of the presenters have Chinese names. Though I can’t know their nationalities, So I thought, “People of Chinese descent really are doing incredibly well.” That was the impression I got. Then, as I walked around the booths,
NeoClouds and the Rise of Inference Serving Companies 24:07
Jonghyun Park there were so many companies that I couldn’t see them all. Broadly speaking, what stood out to me were neocloud providers, companies that rent out and sell GPU capacity, and the sheer number of companies selling inference, which made me realize that inference really is now developing fully fledged business models at the commercial level. That was what I felt. Well-known companies include RunPod, Together AI, and Nebius, and I personally use RunPod all the time. I rent and use GPUs, and there were companies like these. Then, if I may highlight just a couple, there were companies founded by Koreans, though I think they are all headquartered in San Francisco. To briefly explain just two of them, there was a company called VESSL AI. VESSL AI similarly buys up GPUs
Korean Infrastructure Players Like VESSL AI and FriendliAI 24:56
Jonghyun Park and rents out and sells GPU capacity. I wasn’t very familiar with VESSL AI either. But VESSL AI seems to do a lot of B2B sales. So, like Lambda Labs, VESSL AI provides entire blocks of GPUs to frontier labs. That was one cluster provider I saw. Then, in a similar part of the same ecosystem, there is FriendliAI, a company that sells inference. So if you use APIs on OpenRouter to serve LLM models, quite a few of you may have seen FriendliAI. I haven’t used FriendliAI myself, but I remembered seeing it on the list.
FriendliAI rents GPUs, serves models on them, and sells access through an API, so I asked what kind of technical moat it had. I was told the moat lies in how efficiently FriendliAI can perform inference: meeting latency and bandwidth requirements, maintaining uptime as close to 99.9% as possible, and then lowering inference costs. FriendliAI said achieving those things was its moat. Probably the most famous project for this kind of inference serving is a project called vLLM. I use vLLM frequently as well. One of the people who created vLLM is its co-founder, Dr. Woosuk Kwon. Dr. Kwon developed vLLM while researching it at UC Berkeley.
That was all I knew, but when I asked, I was told that Professor Byung-Gon Chun of Seoul National University had conducted this kind of research in his lab, and that FriendliAI was founded based on techniques developed there. So I asked how FriendliAI differed from vLLM, and I was told that before Dr. Woosuk Kwon worked on vLLM, and before Dr. Kwon went to UC Berkeley, Dr. Kwon co-authored a paper with the lab here. So in a sense, FriendliAI and vLLM came from a similar lineage and branched off into separate projects. I think that is one way to look at it. For reference, FriendliAI is not open source, so our only option is to pay to use it. vLLM and SGLang seem to be
Chester Roh the two most famous options in inference, and vLLM really made a breakthrough with PagedAttention— That’s right. It was a landmark achievement.
These days, it seems like almost everyone doing inference uses either vLLM or SGLang.
Jonghyun Park Right. PagedAttention was the paper Dr. Woosuk Kwon wrote, which then became the basis for an open-source project and eventually a company.
Seungjoon Choi But Jonghyun, in what context are you using it?
Jonghyun Park vLLM? I download open models and serve them, and when I run inference with them, I usually rent GPUs from RunPod, which we mentioned earlier, deploy vLLM on them, and serve and use the models there.
and serve and use the models there.
Jonghyun Park When you use them yourself? Yes. When we work on projects, there are quite a few where we serve the models ourselves and use only the data generated through inference. We mainly use this approach in cases involving security concerns or various other issues. These days, the AI industry
The AI Infrastructure Stack Becoming a Comprehensive Art Form 28:11
Chester Roh is becoming almost a comprehensive art form. We used to talk about doing fine-tuning and things like that, but then the consensus became that it was better to use RAG with frontier models, and when the harness trend took off, everyone rushed in that direction. But as that approach became too cumbersome, the trend shifted toward grouping smaller tasks together again, putting them into a small specialized model, and running inference with that model. That’s the direction things are moving in, so RAG, harnesses, fine-tuning, post-training, and datasets all seem to be integrated into a single interconnected system. As a result, the inference strategy also needs to change completely depending on the model size and the type of workload, as well as whether prefill or decode is the appropriate focus. The strategy has to be entirely different in each case. When you listen to people working in this field, the ones who really know it practically have to master a comprehensive art form.
So just the other day, we were recording with Nikhil at Lablup with CEO Jeongkyu Shin. After the recording, partly to introduce them, I introduced Nikhil and CEO Jeongkyu Shin to each other. I wasn’t in the meeting with them because I was cleaning up in the back. About an hour later, Nikhil came out and said, “Wow, who is that guy?” Praising CEO Jeongkyu Shin, Nikhil said that since arriving at ICML, Nikhil had met countless people over several days, but none of those meetings had been as valuable as the hour Nikhil had just spent meeting with Jeongkyu. Nikhil said Jeongkyu understood everything from chips and data centers to inference and application workloads, with a command of virtually the entire stack from the ground up. Nikhil asked, “Why doesn’t that company enter the data center business? It seems like the company best positioned to do it.” “They were too busy to do it, so we joked, ‘Should we give it a try?’” Companies like that are all emerging,
but for people who are only now jumping into this to understand it all at once, it feels like there are an enormous number of prerequisites they need to study. But among the players who understand those things, at some level of this total art form, they are discussing this industry, and that was one of the things I realized this time.
Seungjoon Choi That means it’s at a tier where you can’t just say, “Do it for me, do it for me.”
Chester Roh To make “Do it for me, do it for me” work perfectly, many other things also need to go underneath it. I think that’s the right way to put it. To take the total art form one step further,
Dedicated Accelerators and the Expansion of the Hardware Optimization Ecosystem 30:45
Jonghyun Park if everything up to this point involved businesses that rent out clusters based on NVIDIA GPUs, or businesses that optimize inference on those clusters, there’s one more category: making entirely new chips. Accelerators. I didn’t include a photo of this one because they asked me not to, but HyperAccel, where Jinwon Lee, who also appeared on our AI Frontier channel, serves as CTO, also had a booth. They said they plan to release a dedicated accelerator soon, and OpenAI also seems to be focusing heavily on this lately. GPT is already running on dedicated chips from Cerebras, and then, together with Broadcom, OpenAI will probably design and soon release a chip dedicated to inference, or so we seem to be hearing. So there are similarly companies that actually manufacture accelerators, and then, across various accelerators, Nota AI is a company that optimizes how to run LLMs effectively. The ecosystem now seems to have solidified into a business model encompassing hardware, optimization for that hardware, and even serving LLMs. Earlier, when we talked about GPT-5.6,
we said that moving away from subscription plans was a little frightening, and I said that I personally found it frightening as well. If so, a more affordable alternative for us might be to serve GLM or open-source Chinese models on infrastructure like this, and use them at a somewhat lower cost. That could be an alternative. That was my thought.
Seungjoon Choi What exactly does Nota AI optimize? I don’t know exactly what they do either, but from the brief explanation I heard,
Jonghyun Park for example, chips from Qualcomm have an NPU inside. There are needs to deploy LLMs on hardware like that and run them, and as I understand it, Nota AI helps with those tasks.
Seungjoon Choi It might involve scheduling or something along those lines. I’m not sure, though. I think it’s probably about porting.
Jonghyun Park Because the hardware is different, you need to adapt it to that hardware—for example, to the quantization methods and ALUs it supports— which might require quantization, or splitting up the memory and restructuring the Transformer in one way or another. I believe that was the general idea. Nota AI probably operates across a somewhat broader range of businesses, but what I heard was that this is one of the things they do.
Seungjoon Choi HyperAccel was the one talking about a chip whose name starts with B, right?
Jonghyun Park That’s right. I think they’ll probably unveil it soon. And then they seem to focus on A, B, and C,
Chester Roh naming their chips A, B, C, D, using those letters as codes. Is that the naming scheme they use?
Jonghyun Park It feels like the naming scheme from the old days of Android. To categorize the next group of companies, there were also quite a few booths related to data. The best-known one is probably Scale AI. Their business is about how they create and sell data. But because this was an academic conference, their booth seemed focused more on recruiting students than promoting their business, giving it the feel of a booth showcasing their research, at least to me. Then there were companies I had never heard of before, such as Voxel51 and Toloka, particularly companies that collect and sell data for Physical AI. Because, as you mentioned earlier when discussing Scale AI, there are aspects of robotics that are difficult for Scale AI to handle at the data level. So there were quite a few companies collecting the relevant data. Those were some of the company booths, and one that stood out to me was a company I hadn’t known existed, but its booth was enormous. It was a company called Handshake AI. Handshake AI mainly contacts undergraduate or graduate students, each of whom presumably has a domain they specialize in. For example, a mathematics student would be quite an expert in the mathematics domain, while a student proficient in a particular language would be an expert in that language. Handshake AI brings all these students together and asks them, “Please create this kind of data.” Handshake AI then collects that data and sells it to frontier labs. That’s their business model, and the scale seems much larger than I expected. Apparently, they’re quite well known in the United States.
The Expert Data Market Through Scale AI and Handshake AI 33:28
Chester Roh Companies like Mercor and AfterQuery don’t create datasets in a substantially different way, so this seems to have expanded that model through a more open-sourcing approach.
Seungjoon Choi In any case, they’re collecting expert data.
Chester Roh Once the domain is defined, whether it’s finance, legal, physics, or chemistry, after that has been determined, neither AI nor frontier labs have any way of knowing what areas exist within that domain. So companies like those now play a kind of crawler role analogous to the role crawlers played in the search era. They go deep into a domain and keep broadening their coverage of it, and for the numerous subfields within it, people define some intent or objective. Then, when it comes to forming the dataset, the model recombines the material. As far as I can tell, there are no datasets that people write entirely by hand. But the models create datasets within that framework,
and to turn them into a complete post-train set, ultimately, someone who knows the subject well Since people have to evaluate it, it requires quite a lot of people. For highly advanced areas, expensive experts need to be brought in, while for areas like ordinary coding, math, physics, and the so-called core school subjects, even undergraduates can do the work. So that’s the approach used by companies like Handshake AI or the dataset companies we’re familiar with, and what’s really interesting about that approach is, once they create a new data domain that way, they take it and sell it to frontier labs.
Because those labs need to broaden their data coverage anyway. So once one lab over there buys a certain amount, they can sell the exact same thing to all the other labs as well, and that seems to be how this market works. And there’s still evidence that as datasets grow, they lead to improvements in model performance, so frontier labs are spending enormous amounts of money on those datasets, as I understand it.
Seungjoon Choi The name is rather suggestive. Handshake AI—it sounds a little ominous. Categorized this way, there were quite a few neocloud providers,
Jonghyun Park and there were data collector companies. Aside from labs researching models, these were the major categories, or at least the major categories as I saw them. Next, out of the various booths, I’ve brought up a few that I’d like to highlight. This company is called General Intuition. A personal friend of mine and co-host of our sudoremove channel, JC, strongly recommended that I visit, so I did, and they have a gaming platform called Medal. It’s a platform where gamers, while playing, can clip the exact moments when they think they played really well and share them. As a result, it’s a platform with an enormous amount of gameplay video data. Moreover, one important thing is that they collect the gamers’ input as well. So they can see how the game changes when a particular key is pressed. If you only have video, you normally don’t have those input keys, so you don’t have the input data corresponding to the actions. Since they have this kind of data, they have data that’s ideally suited to the world models we’ve been talking about lately.
General Intuition Building the World Model MIRA with Game Data 37:24
Jonghyun Park They created the company General Intuition from that, and it has a world model called MIRA, which they were demonstrating there. We were curious about it, so we asked them some questions. There’s a game called Rocket League, which is popular in the West and involves playing soccer with what look like miniature cars. I don’t know it that well myself. Normally, when we press a key in that game, the game engine and program run and render the scene, which we then see on the screen, but instead of that, the world model simply sends the visuals to us. The conventional approach disappears entirely, and if you think of a game as a physics simulation, the world model performs that simulation. They had built and were demonstrating a model along those lines.
They had also published quite a detailed technical blog. I hadn’t realized that it was publicly available, but while visiting and asking questions at the booth about how much data they had trained it on and what the model size was, I found that quite a lot of that information was available. First of all, it has 5B parameters, and compared with typical video models today, its architecture isn’t substantially different. The reason it’s 5B is that, compared with an LLM, the model is very small, but it seems they believe it needs to be around this size for frames to rapidly stream out when we provide input in real time, making interaction possible. So many video models also seem to be around 14B, 5B, or 7B, roughly within that size range.
Next, it says here that it was trained on 10,000 hours of Rocket League, and apparently that information is in the technical report. I had an agent research this, but when I asked them on-site how much data they had trained it on, they said, “We can’t discuss the details, but it’s YouTube scale.” That’s how they answered. They said they had trained the video model on an enormous amount of data, and then there’s their funding. They’re also well known for having raised an enormous amount of funding. They raised a Seed round and, recently, a Series A as well. But I felt that one of their business models could become extremely successful because they have so much data, so I asked whether they sold any of it to frontier labs. They said it was their moat and that they were trying to avoid selling it as much as possible. Having raised funding, they might have had reason to sell it, but I think that’s probably why all those investors funded them. But they said that even after raising funding, they weren’t selling it and currently had no plans to, and that they would build and sell their own world model instead. So I found that rather impressive. When that company first raised funding,
Chester Roh I remember following it with great interest as well. It still isn’t widely known, but the dataset that company has seems very distinctive.
Jonghyun Park I think collecting game key logs together with the footage gives it considerable value in the action domain. So that was General Intuition’s world model, MIRA, and next is ElevenLabs. I’ve personally been using this a great deal lately and was interested in it, so I asked them about it. A representative from ElevenLabs was also there, specifically the person responsible for Scribe, which converts speech into text— what we call STT, ASR, or speech recognition. That person came and told us a lot about how “this is how we’re building it.” We heard quite a lot along those lines. Then, one of the things I was most curious about was the METR booth,
ElevenLabs Scribe and Sketches from the METR Booth 41:17
Jonghyun Park but there was never anyone at the METR booth, so I couldn’t speak with them. That was such a shame.
Chester Roh I heard that one booth cost $30,000, so why didn’t they show up? Yes. That’s what I heard too. I heard it was $40,000,
Jonghyun Park but I think this particular spot was probably $40,000. There were people from the Singapore government next to us, and they said they hadn’t seen anyone there even once throughout the entire event. I suppose something must have come up. And there were also some other companies
conducting various kinds of research. I’ll quickly move past these, and if I were to identify one more category, there were a great many quant and trading firms.
Their booths were enormous too. But as for those firms, even when I visited them personally, I wasn’t knowledgeable enough to hold a conversation with them, so I didn’t take a close look, but in any case, this was another major category. I think they were all there to recruit. That was the general impression.
Chester Roh It sounds like the exhibition area was absolutely enormous. A lot of people were going in. It was so packed with people inside that I couldn’t bring myself to enter.
Jonghyun Park The lines for the booths were also far too long, and at the popular booths, it was difficult to even have a single conversation. So although I was there for more than half a day, I couldn’t see everything either.
The Density of Side Events and Robotics Night On Site 43:00
Chester Roh Exactly. So there were also lots of meetings nearby, and every lunchtime, there was a room where researchers from frontier labs would gather from all over to have lunch, so I went in and joined various lunch gatherings as well, and everyone was young and from Google DeepMind, followed by OpenAI. That was about it. People from OpenAI and Google DeepMind often traveled around together, but the people from Anthropic didn’t seem to travel around together. That was one thing I found rather unusual. Perhaps because not many of them attended, they weren’t very noticeable either. Aside from the official sponsored-booth events,
Jonghyun Park there were many side events, and this is one of the things I photographed while attending a side event. Chester also seems to host a lot of side events and attend them as well. Next, among those events, I attended an event called Robotics Night hosted by RealWorld and saw a demo like this. A huge number of people came to that event too—probably somewhere over a hundred people. So I was able to meet and talk with a great many people who conduct robotics research. Since robotics is a specific domain, I often felt that the field itself was somewhat more concentrated. Among them were also people from industry, such as the Toyota Research Institute, as well as graduate students from academia, and especially a considerable number of researchers actively conducting research. Most people there, as you mentioned earlier, were divided over whether scale is the answer or whether the answer lies in how effectively domain knowledge is used. Opinions were also divided over how RL should be done, with some saying it might be better not to use RL here, so there was an enormous range of views, and people were having very active discussions with one another.
So I attended events like those.
Then Chester’s event overlapped with my schedule, so I was only able to stop by briefly and couldn’t stay very long, but what I personally felt there was that the concentration of expertise was extremely high. There were also a great many people from San Francisco, and there has been a lot of discussion lately about whether we need to do token maxxing, whether it’s pointless, or whether efficiency is what’s important. People have been saying a lot of things like that, but when I listened to those people talking, someone said that at one startup, they spend $10,000 worth of tokens per person per week. Then someone else said, “Why do you spend so little? Why have you limited it like that?” That’s how they responded. “I sometimes burn through $20,000 worth of tokens in a single day.” When I heard that, I wondered whether we were really living in the same world, and that was what went through my mind. How can they burn through that many tokens and still generate an ROI? I realized there must be ways to create value by burning tokens in order to run a business more effectively. Those were some of the things I came away feeling. We should also ask exactly what they made.
A Story of Witnessing $20,000-a-Day Token Maxxing 45:02
Chester Roh token maxxing and the quality of the resulting output, along with its evaluation, all need to operate as a pair for it to be assessable. We shouldn’t evaluate something based solely on someone saying, “I spent this many tokens.” That’s the perspective I hold. I also remember many people saying that perhaps the era of token maxxing has already begun to pass. I was curious about that too, so I asked,
Jonghyun Park but it seemed they couldn’t discuss it in detail. They only said that they had to spend that much because they handle countless figures and aggregate data, and after hearing only that, I thought perhaps that kind of work could require spending that much.
Chester Roh I suppose it could.
Seungjoon Choi Right. So we’re talking about roughly $20,000 a day?
Chester Roh That’s right. Did they calculate that backward from their own subscription plan? Or did they actually spend that much?
Jonghyun Park Yes, I asked about that, and they said they use it all through enterprise. At the enterprise level, using a subscription plan apparently violates the Terms of Service. So I then asked whether, from the perspective of a frontier lab, or a company like Anthropic, individuals use more through subscriptions, or more tokens are used by enterprise customers, and they said the volume of tokens consumed by enterprise customers is overwhelmingly greater. For now, that’s all I have to share, and it’s Saturday morning today,
with two more days of the event on Saturday and Sunday, so if I get the chance, I’ll continue next week and tell you more about it.
Seungjoon Choi By the way, Jonghyun, the material you showed us earlier appeared to be on the AI Frontier website.
Jonghyun Park Yes, this time, we created the material to be added to the AI Frontier website, We made it available as an addition there. Going forward, whenever we have materials to share, we’ll keep adding them there whenever possible. Thank you. I think that’s the right approach.
Networking and Parties Becoming the Real Main Event of ICML 47:50
Chester Roh The main ICML event is important, of course, but you’re so busy going back and forth between nearby restaurants and coffee shops, and then the parties held in the evenings, that your head is spinning. So I wonder if that isn’t actually the main event rather than the main conference. Also, for people from frontier labs, rather than coming to read papers or study, more than half their reason for coming is probably to have fun, so most of them have either already visited Jeju Island, or afterward, they go to Jeju Island, or they’ve visited Busan or go to Busan. Korea itself is also highly popular, so I heard this year’s ICML attendance was nearly the highest ever. Some say there were 10,000 people, while others say 14,000 or 15,000, and I spend every day around COEX, since that’s my main base of operations, but I don’t think I’ve ever seen that many people around COEX. And the people there weren’t stylish foreigners, but crowds full of engineers like us, walking back and forth, so I thought, “Whoa, this feels a little strange.” It felt like I was back in San Francisco, and that was the impression I had. The proportion of women wasn’t particularly high, while the proportion of engineering nerds was high— it seems to have been a week with an overwhelming density of engineering nerds. So we also hosted several side events,
and I attended others as well. From Monday this week through almost tonight, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, and Sunday evenings, I’ve been partying every single night. Starting with Tuesday evening, Tuesday’s event involved Delta Institute, Striker VC, Recursive Intelligence, and a company called Illorian, which are all companies connected through the same portfolio. They’re companies Striker VC invested in, and Striker is the venture capital firm where Nikhil, whom we recorded with recently, is a partner. This company was also very interesting, and there was a great concentration of engineers there. What particularly stuck with me was talking with a researcher from Mistral about various things. What they said wasn’t substantially different, and they consider it somewhat important to establish some kind of presence in the U.S. market. And then, when you ask about sensitive internal data, naturally, they wave their hands like this and refuse to tell you, which also happened. Among the people I met was someone from a company called Flapping Airplanes.
Flapping Airplanes and Bets on the Next-Generation Model Paradigm 49:26
Chester Roh When I heard Flapping Airplanes, I thought, “Flapping, like a bird flapping its wings—what do they do?” I wondered if it was some kind of aviation simulation company, and because there were so many people and it was so noisy, I leaned in and listened closely. As I mentioned in a previous episode, there are companies trying to build a new kind of model. At the level of SSI, they argue that something about transformerism is wrong. They say we need a model that is much more data-efficient and can learn much faster through few-shot learning. There are groups making that argument, and these groups tend to consist of extremely smart people, with teams of 10 to 20, or 30 at most. Their starting valuation is already in the billions of dollars, and they raise hundreds of millions of dollars. I don’t think Flapping Airplanes has even existed for a full year. It held its first seed round a few months ago, and with a 1.5B valuation, that’s a valuation of approximately $1.64 billion, with around $143 million to $214 million raised in seed financing. I met that guy again over lunch yesterday, because I could sense the unmistakable aura of genius about him. I said, “Let’s meet just one more time,” and took him to lunch yesterday while asking him about various things.
You know those things going around on Twitter these days. The recipe for building a frontier model is actually nothing unusual. It’s all public knowledge, and everyone knows one another’s recipes. It’s just that there are only a few companies capable of doing it at that scale while taking on the financial risk, or so people openly say. He said that part was only natural. It’s true that they all know one another’s recipes, and I remember discussing which algorithms are advancing, and which points they are establishing footholds at as they move forward, along with things like that.
Seungjoon Choi Is he in his twenties or thirties? He’s probably in his twenties. He also graduated from Stanford
Chester Roh and worked at a finance company called Citadel. He was there before moving over to this company.
Jonghyun Park How were they able to secure such a high valuation?
Chester Roh I’m not sure. I didn’t ask him that, since he isn’t the CEO, but presumably, the market places an extremely high value on it. The venture capital firms capable of investing that much at that valuation are extremely well-known firms. They’re all firms like Sequoia or a16z, or Index Ventures. They’re not small boutique firms even by Silicon Valley standards, but enormous firms. So these enormous firms are doing what SoftBank’s Masayoshi Son used to do when selecting new companies: when everyone else was raising $7.1 million or $14.3 million, he would invest hundreds of millions at billion-dollar valuations, giving the company an enormous financial advantage, positioning it as a market leader, and using that strategy to give it an advantage in a particular market.
But now, I think the leading firms in the Bay Area have benchmarked that strategy. Those firms are effectively saying, “These are highly reputable and promising companies, and whether in terms of talent or the research they’re conducting, “It’s worth this much.” They’re effectively making a declaration. It seems like they’re making a declaration, and the frontier companies in the Bay Area have such enormous amounts of capital that if the venture capital firms backing them recognize that level of valuation, there’s an atmosphere where everyone else accepts it as well. They have people willing to back them later, and that confidence seems to be why they set the valuation that high. And they probably know more than we do. Of course, that trend carries enormous risk, but if they can create something other than the transformer, a true candidate to succeed the transformer, the expectation of an enormous return seems to be reflected in that valuation. It sounds like they do have something.
Seungjoon Choi I mean, Jerry Tworek also left OpenAI and founded a startup. I don’t know whether it’s still in stealth. Given that we occasionally hear that he’s working on something new, it sounds like he does have something. Right. But even when asked, they never tell you anything at all. They only vaguely say, “This is a problem.
Chester Roh It’s a very big problem,” and talk around it with this and that, but what they intend to do is build a model for the paradigm that comes after Transformers. That person stood out to me, and on Wednesday, AI Frontier Korea—our podcast— and SemiAnalysis in the US co-hosted AI Industry Night. We had originally planned to invite only about 20 people to Lablup for a casual meal and conversation, but so many great people applied that we forfeited our deposit to cancel the reservation and hurriedly moved to a larger venue the day before. Fortunately, Lablup CEO Jeongkyu let us use his company’s space, so in the lobby, we hosted nearly 40 to 50 people, who came and talked with one another. Jonghyun, as you mentioned earlier,
AI Industry Night and Conversations Around the Data Center Value Chain 55:35
Chester Roh venture capitalists from San Francisco, OpenAI researchers, neocloud companies, and TensorWave, which works with AMD, were among the many fascinating people who attended. And from Korea, people from SK Square, Jinwon Lee from HyperAccel, and Dr. Dongsoo Lee also attended, along with many people from LG Electronics and Hyosung Heavy Industries, and we mainly brought together people interested in everything from data centers to the chip side and primarily discussed those topics. Most of the discussion was about where the current bottlenecks are in the data center value chain and which companies are doing what, followed by what opportunities Korea has, because surprisingly, they don’t know much about Korea. SK hynix, Samsung Electronics, and BTS are pretty much all that foreigners know about Korea. So they asked about the parts of the value chain preceding memory: how is your data center business structured, what is the power situation like, and are there participants other than major operators such as SK? Are there smaller operators, and how are they preparing? I think they went through questions like those. So an incredibly diverse range of discussions took place, and there were also people working on OpenAI robotics and post-training. Then an engineer from GLM came as well, so we talked about GLM. I said, “You’re extremely popular in Korea. Did you know that?” and the engineer said, “No, I had no idea.” I remember having conversations like that as well. On Thursday, Matthew Kim and I co-hosted Yacht Night
The Bio AI Meetup and the Virtual Cell Debate 58:01
Chester Roh with a company called AfterQuery, but heavy rain that day caused traffic jams, among other things, so it was a bit chaotic. But we went and spoke with the researchers who attended. And on Friday, since I’ve recently been trying to shift my business toward AI and the bio side, I attended workshops on Friday, Saturday, and today. After this, I’m going back to the workshops again. The ICML main tutorial sessions—what do you call them? After the main conference ends, they hold workshop sessions, and there are many interesting workshops. So I’ve mainly been attending sessions on bio or AI for Science, and there was a GenBio session yesterday. We spread the word informally among the participants who had come to GenBio and held a meetup exclusively for people working at the intersection of bio and AI. So we talked about various things over dinner, and people from Arc Institute and NVIDIA, as well as Anthropic, attended. They were mainly people working on the bio and AI side, and someone from Harvard attended as well. These people and others were also the ones involved in the argument I mentioned earlier. The scaling maximalists
and the people saying, “Hey, you shouldn’t do it that way,” were somewhat divided. Here’s the funny part: I was sitting at our table with someone from NVIDIA, someone from Arc Institute, and an interesting person who had dropped out of a PhD program at Harvard to start a company. One person was sitting at our table but fled to another table. And when I heard about it later, the person had gone to the other table and said, “That table is dangerous. It’s full of scalist idealists, and they’re approaching bio in a very dangerous way. This isn’t a problem that can be solved like that.” But the person I spoke with the most was working on the Evo 3 model at Arc Institute, and we discussed the current problems with DNA foundation models and how, when it comes to creating a virtual cell, every lab talks about a virtual cell, but every lab means something different by “virtual cell.” And this is something I’m gradually learning about biology as well, but first of all, there isn’t as much data as there is for LLMs. It’s not like there is text scattered all over the internet, or Video data. The data is highly fragmented across labs, and just because there is a lot of that data doesn’t mean all of it is decisive data. For example, suppose you’re conducting an experiment, and after going through the whole process of preparing the experiment, you reach the moment at the end when you trigger a reaction, the reaction occurs, and you get data in chronological order. The segments containing valuable data are all concentrated in the few minutes or hours at the end, while the data that comes before could be compressed into a single unit. But when people talk about biological data today, including all that earlier data, a pharmaceutical company, for example, may possess a tremendous amount of data, and we may say that we have data from millions of experiments, but the uncomfortable truth that many people point out is that the portions of that data that can actually be used for training, the portions that have value as data, are often only very brief moments. So there are still practical limitations overall.
Even if the compute can be made ready to pursue this scale, there is a recognition that the datasets needed to train it at that scale are indeed still insufficient, but I think everyone has at least some optimism that goes beyond the current reality. Maybe we could solve it this way. So, as I mentioned last time, people who come from traditional biology pipelines start from that domain, and because there are prerequisites that must be met to make it work, the problems are defined very concretely in a truly domain-specific way, while people who start out in computer science are super generalists. So, to borrow a somewhat disparaging expression used by people on this side, some computer geeks who never learned things properly anywhere come along and think it’ll work if they just try that a few times, but this field doesn’t work that way— I remember it being said with that sort of nuance. But I think that’s what’s good about this conference. People at the forefront of their respective fields come together, and because they have to communicate in a short amount of time, they dive right in on the assumption that they know something about each other’s fields and raise fundamental points. Then, if they can’t go back and forth on the topic, they realize, “This isn’t going to work,” and quietly part ways, but if they do click, they establish common ground and exchange a few key questions. That’s when a lot of truly valuable conversations happen, and through those conversations, I also form a clear picture in my head of the distribution of problem difficulty as perceived by people at the frontier. I think that’s really good, but one thing that’s clear is that on the LLM side, as I mentioned earlier, everyone seems to have become AGI-pilled. They see this as a game that can be won, and I think I mentioned that many papers are now focusing on how to measure that success. Because they believe it will work, the question is how to measure it properly so they can measure even better things. Right now, whenever a model comes out, whether it’s a good model or a lousy one, all the benchmarks are maxed out, and this issue has become a subject of investigation, so that side seems to be following the trajectory of the trends we already knew, while AI for Science is clearly trying to establish itself as the next trend. Apart from a small number of highly forward-looking venture capitalists, this trend hasn’t yet reached venture capitalists in general. It hasn’t reached them yet. And it goes without saying that it hasn’t yet reached the capital markets or the general public either, but I definitely got the sense that by around this fall or next spring, it will firmly establish itself as the next trend and take over from Physical AI. That’s about it. Whew, Wednesday, Thursday, and Friday were incredibly intense.
Confidence in AGI and AI for Science Emerging as the Next Trend 63:07
Seungjoon Choi Yes. You’ve really been going nonstop. My voice is cracking right now. I talked way too much,
Chester Roh and when people gather and talk somewhere noisy, there’s this research problem too, you know. You have to do source separation. I need to hear only the sound coming to me, but because all of it comes in mixed together, it feels like we’re communicating through context while catching individual words, words, words. That was kind of interesting.
The Compressed Pace of Progress and the Agent Market That Has Barely Even Started 65:45
Seungjoon Choi So how would you wrap up everything we discussed today?
Chester Roh To wrap up everything we discussed today: progress is being squeezed into ever shorter timeframes. Things we would have discussed over several sessions in a month, and things that, in the past, we would have spread across an entire quarter, we’re now recording in week one, week two, and week three, Jonghyun, about three times in total, and although it may seem like we’re discussing similar things each time, we’re talking with a sense of urgency that the pace of change is accelerating even further. I think that may be the core message I’m feeling right now.
And another interesting thing I noticed was the discussions about AI applications. Whether on YouTube, in harness discussions, or at any hackathon, everyone talks about agents. They talk about what they built with agents and how agents can improve their work, but none of that is being discussed here. Most of the people who came to ICML this time do come from research backgrounds, but even the venture capitalists and others here aren’t asking any questions about those things or discussing them at all, which I found refreshing, and when I ask about that, I sense two signals. One is the common view that “Doesn’t Codex just handle all of that?” At the application layer, super apps like Codex or coding harnesses can just use prompts to make everything work in the form of generative UI and generative services—that’s one view. The second side makes me think, “This hasn’t even started yet.” As I listened to those conversations,
I thought, “This field hasn’t even started yet.” My perspective on that is, of course, Codex, Claude, or whatever else will build anything you tell them to, but people won’t even tell them what to do, so companies will emerge to do that on their behalf, and just as B2B SaaS companies emerged, countless AI agent companies will start appearing—but that hasn’t begun yet. That’s because the underlying foundation is shifting so much, and it has yet to become firmly and densely established. But I think that, industry-wide, it will become an enormously large business, so I think it’s something we should keep an eye on. About the latter point—they don’t discuss it, but
Seungjoon Choi at the same time, an enormous market is opening up, so the idea that they are not pursuing it is where I can’t quite connect the dots…
Chester Roh The timing just hasn’t reached the point where it falls within their interests. The market timing isn’t there yet for the venture capitalists moving the money around or the people at the frontier labs… So the reason capital isn’t yet talking about
Seungjoon Choi those agent workflows is that the market timing isn’t right.
Chester Roh Put simply, it may be a characteristic of ICML. That is simply the profile of the people who attended, whereas if you went somewhere like Y Combinator Demo Day, this would probably be all anyone talked about.
Seungjoon Choi The context may have been somewhat different because it was ICML, This week, as I monitored ICML,
A Research Landscape That Has Become One with Industry 69:02
Jonghyun Park we hardly discussed any particular paper or research result today either, and personally, the sponsor booths and things like that were more impressive to me. If you think carefully about what that means, what has changed from the past is that research has become much more closely connected to industry. One reason is that scale is extremely important, and personally, especially when I went to Chester’s event, there were people who actually research models at frontier labs, for example, so I asked them. I went around asking which papers at this ICML were worth noting and what I should look at. I asked everyone. There is so much research that I also wondered, “Which of these should I focus on?” But what most people said was that these days, compared with papers, the things being discussed in industry seem to have more value. That’s because research is also being conducted around those things.
Hearing that made me think, “They’ve become more closely connected. Industry and academia have come together.” Right. In the past, if the layers consisted of
Chester Roh assembly, then C, then Python on top of that, and Django on top of Python, the people building applications at the top could simply say they only needed to know how to use Django well. There used to be that kind of division of roles, but with this, everything from assembly all the way up to Django feels like it has been fused into one. So the stack is quite deep and broad, and it feels as though you need to know the entire stack and become a kind of jack-of-all-trades before you are qualified to talk about anything, whether it’s the application at the top or something else. Otherwise, even when you go to the so-called industry side, you quickly run out of things to say. “Codex is great.
Claude Code is great. In loop engineering, I tried running this harness.” You have to go deeper than that: “Then for this, we do it this way, separate out the model and change it like this, and we use Codex for this one. We have so much proprietary data here that we need to use this model for this one.” Only when the conversation goes that far does it feel like there is a real back-and-forth. Another interesting thing I heard
The Risk of Frontier Lab Dependence and the Outlook for a Ban on Chinese Open-Source Models 71:39
Chester Roh was also covered in the podcast we uploaded last week. We all operate in some line of business. But when we mindlessly throw everything into Codex, saying, “Do this, do this,” aren’t we feeding it our companies’ core assets? And by even using auto-compacting to keep everything neatly organized in a single thread, we feed it the company’s entire workflow, so from the perspective of the frontier labs, an entire company’s business is simply being handed over to them. And then companies like Anthropic and OpenAI will need to justify their valuations, so, as companies like Microsoft did in the past, they need to take over and own many core industries. Microsoft swept up everything with Office, and Google captured a great many things as well. But in the case of AI, it isn’t simply selling storage, selling functions, or selling a distribution channel. It is selling intelligence, which makes it difficult to draw boundaries between layers. There has been a lot of discussion about that. As a result, enterprises are waking up to the fact that handing all of this over to frontier labs is extraordinarily dangerous, so the importance of using open-source models to build their own harnesses and their own farms seems to be rising much faster than expected. The importance of on-premises models. And another interesting point here is that the open-source models that American enterprises currently rely on are almost all Chinese models. They are based on things like Qwen and Kimi, If I say, “a friend from SemiAnalysis,” everyone will know who it is. Anyway, according to what that friend said, by around next year, the U.S. government may ban U.S. enterprises from using Chinese open-source models as well, or prohibit their use— that friend said this was the prediction they had. If that happens, models that can be supplied by countries friendly with the Western world— the so-called near-frontier models, open-source frontier models— will shoot up in value. Hearing that this is what they think, I thought, “That could happen.” It also struck me that it could have major business implications. Right now, there’s no one but China.
Seungjoon Choi I feel like there was a similar point in AI 2040. I haven’t looked closely, but China is an important keyword in any case. Indeed. To ban it outright like that. Open-source models themselves.
Jonghyun Park Yes, the expectation is that even running them on-premises will be banned. That’s the prediction.
Seungjoon Choi I’m not sure whether that would actually be feasible, but they could introduce such a policy, at least. Things are a bit hectic for us.
The Need for Synthesis in an Age of Information Overload and Closing Remarks 74:44
Chester Roh We need time to calmly organize our thoughts, discuss things, and really delve into a single topic, but the world isn’t giving us that breathing room right now. Things got so bad that I remember telling everyone at the company, “Let’s just frantically absorb information through this week, and starting next Monday, we’re getting to work.” GPT-5.6 came out, and I also heard a lot of estimates about the model size, but they told me not to talk about that, and they shared so many things that everyone will find out about later anyway. So, with that, we’ll wrap up today’s recording. Jonghyun, Seungjoon, thank you again today.
Jonghyun Park Great work, everyone.
Seungjoon Choi It was fun.