Event Replay: Building the Future of AI: Emergent Capabilities of AI

[00:00:00] So, let's just jump right in. [00:00:07] Thanks Jason, thanks for the invitation. [00:00:11] So I'll do, I'll jump right in. [00:00:14] And actually, from, from Rick's encouragement, I've been doing some vibe thinking and trying to prove some limit theorems. [00:00:21] Actually, these were a couple things I was working on on the plane over. [00:00:26] So one is trying to draw on information theoretic ideas to understand kind of membrane systems in thermodynamics and the other is trying to understand the limits of flying through forests without crashing into trees. [00:00:41] And this has been like Rick is describing, I mean you get a, like when you do it, it just feels incredible and I'm glad you've been pushing me to do this. [00:00:50] So what I'm going to talk about is AI capabilities and in some broad sense. [00:00:56] And I'll be drawing on some work at Cochri which is a startup company that I lead and Heisei Yu as my co-founder who's here as well, so you can also talk to him about some of these things.

[00:01:09] Right. So as we're thinking about capability overhang, I want to kind of go back in history a little bit. [00:01:16] So Tom Wheeler, who was the head of the FCC at some point, wrote this nice book about Lincoln. [00:01:21] And the central argument that he was making was in terms of national power is that Lincoln, when he was using the telegraph, was the manner in which he grew to see the telegraph as an instrument of leadership. [00:01:36] He applied the telegraph's technology to create advantages for the Northern War effort entirely on his own. [00:01:41] Because no national leader had ever had this technology, there was no guidance the president could rely upon in the experience of historical figures. [00:01:49] There was no textbook on the application of electronic information. [00:01:52] And certainly, there was no tutor. Instinct alone was Lincoln's guide. [00:01:56] And that's exactly I think how many of us feel with this new technology. [00:02:01] We have no tutor, we have no textbook. [00:02:03] And so we're drawing on instinct. [00:02:05] And that's exactly why we're not able to access all the capabilities that might be there.

[00:02:11] Moreover, I think a lot of us, as we're thinking about these new technologies, we think of them in terms of old technologies rather than reimagining them on their own terms. [00:02:22] We think about horseless carriages and wireless telegraphs, plant-based meat and artificial intelligence. [00:02:28] Right. It's always cast in terms of that old technology. [00:02:33] But as we reimagine wireless telegraphy as radio, you know, that capability of broadcast is the central capability. [00:02:39] It's not point-to-point communication. [00:02:41] Likewise, with intelligence, we need to kind of reimagine the capabilities somehow. [00:02:47] And that's how I would argue you get over this capability overhang. [00:02:51] There's an element of kind of this human creativity to even probe for these things and unlock new capabilities rather than just accelerating the standard playbook.

[00:03:01] So this is kind of where I'm starting to think about this capability overhang question. [00:03:09] And I'm going to try to take a little bit of an idiosyncratic view of it. [00:03:13] Like I was talking to James Evans during the break. [00:03:16] I think the power of humans is that we can be in those weird tales because we are very idiosyncratic. [00:03:22] Like we have our weird interests. [00:03:25] And so let me propose a little bit of a straw man.

[00:03:28] So how does one reason about things you've never seen before? [00:03:33] These capabilities, for example. [00:03:35] So one approach is a very statistical one. [00:03:38] This one, in some sense, goes back to Laplace. [00:03:41] And he was concerned about this question of what's the probability that the sun won't rise tomorrow? [00:03:47] You know, there was no empirical data that showed the sun never rose. [00:03:51] But when you're thinking about it, it seems reasonable to assign at least some probability mass to that logical possibility. [00:03:59] And so that's what led to the Laplacian smoothing, adding a little bit of probability to things you've never seen before.

[00:04:06] And this really kind of came to the fore during the war. [00:04:10] So I.J. Good and Alan Turing developed the Good-Turing Estimator for breaking codes. [00:04:17] And there was some probability assigned to the fact that the Nazis might do something you've never seen before. [00:04:24] And that, in some sense, is one way to think about it, right? [00:04:29] Kind of strategic surprise as just some unseen thing that you assign some probability to. [00:04:35] And, you know, if you look at the movie version of Good and Turing, you know, they go into this in some depth as well, if you want to watch The Imitation Game.

[00:04:41] Right. So. But is there anything else that you can do? [00:04:51] Right. So. So as you're thinking about kind of futurism and strategy and kind of policymaking that comes from it, I would argue.

[00:04:58] that comes from it, I would argue that one needs this kind of sociotechnical perspective. And there's been a strong push for sociotechnical policy making, thinking of things as a coherent system, which is to say that it's not possible to understand the social without the technical, nor the technical without the social.

[00:05:20] And a lot of folks have really emphasized the don't forget the social, but I think what we really need to do also is not forget the technical. And at least for me, public policy is more than just applied political economy. It's not just about addressing market failures like information asymmetries and externalities. There's something very fundamental about the thing itself, which should really influence how one thinks about policy. And I'm going to try to talk through that a little bit.

[00:05:51] Of course, one wants to avoid abstraction traps of various kinds and our colleagues from the social sciences, you know, have dug into this a fair bit. But there's also the opposite abstraction trap. If you abstract the technology actually away from the policymaking.

[00:06:11] If you regard, to take the example, the production function as an economist concept, rather than looking at the actual engineering of it, one can argue that the impact of technology is not just about rebalancing labour and capital. There's something more to it. And that's something to dig into as we go forward.

[00:06:29] Suppose you're someone like me, you're serving in the White House when CHAT-CPT comes out, right? When was it? November of 2022. And you get spun up into all these policy processes and you're asked to think about regulatory policy and so on.

[00:06:47] And regulatory policy is interesting because there's things you care about controlling, which are capabilities. On the other hand, there's things you can actually control, which are resources like computer data and so on. And at least at that time we didn't really have a good understanding of how do resources translate into capabilities, which is the thing that's actually a potential concern.

[00:07:11] And so what would be helpful is an engineering theory that relates resources to capabilities to provide kind of a solid footing for such regulatory policy. We did have the scaling laws and there were some numbers that were proposed and certain executive orders captured those and so on. But this kind of theoretical relationship between capabilities and resources was lacking.

[00:07:39] And so when I went back home, that was something I thought one should work on. And to do that, the difficulty is that you don't know what the technology is going to be in the future. There's no concrete technology to analyze as you're thinking about future policy.

[00:08:03] And so that's where engineering systems theories like information theory that was mentioned, the brainchild of Claude Shannon and others really comes to the fore. Engineering systems theories respect the constraints of physical laws, but they're theories of what can be. They abstract from most of the details.

[00:08:18] And the question is, even if all the ingenuity of the engineer is invoked to address this problem, there's fundamental limits of what's possible and what's impossible. And that actually is super helpful for thinking about policymaking. You understand the fundamental limits and then you can go from there.

[00:08:34] All right. So just again to review some examples. So as I noted in the first slide of the talk, right, thermodynamics is an example. So the Carnot limit establishes fundamental limits on the efficiency of engines. And what I've been playing around with is the same thing, but for kind of a multi-piston system essentially.

[00:08:56] There's fundamental limits on the rate of communication in the presence of noise. There's fundamental limits on how fast you can fly a drone through a forest without crashing and so on, right? And so these are the kinds of things that we want to think about here, but maybe for intelligence.

[00:09:13] And what one can argue is, you know, what every engineer needs is a good set of these limit theorems. But you might wonder why. So what do you get from these? So fundamental limit theorems, first of all, there's the fundamental part, right? They establish which resources and performance criteria are fundamental and which are largely unimportant.

[00:09:36] Secondly, the theorems, the limit part, it demarcates what's possible from what's impossible. But also provides design insights into operating at that boundary, which is to say kind of principles are architectures for optimal design. Third, it defines fundamental benchmarks that allow an evaluation of new technologies on an absolute scale.

[00:09:56] on an absolute scale, not just comparing to previous technologies. [00:09:58] So it's not just about being 10x better than the previous guy, it's getting within 0.0045 dB of the Shannon limit. [00:10:06] And that really changes the nature of how you approach things. [00:10:12] And also, it kind of states ideals for pushing people to build technologies that approach those absolute limits. [00:10:19] There was some discussion about a credit, so actually, these kind of limits encode people to work harder and harder to get to those limits as well. [00:10:29] And I think most importantly, for the policy question, helps characterize probabilities and possibilities of what-if scenarios, especially when there's deep uncertainty. [00:10:40] Emerging technology, by its nature, has this very deep uncertainty, and so if you can establish these fundamental limit theorems it helps clarify that. [00:10:48] And just to be clear, there's some NOGO theorems of this type that are statistical, so that would be, for example, the noisy channel coding theorem of Shannon. [00:10:54] There's also algebraic NOGO theorems, for example, the no-cloning theorem in quantum information science. [00:11:06] And this will be important as we go forward.

[00:11:10] Okay, so if we have these fundamental limit theorems, what can we do with them? [00:11:15] So one is net assessment, so if you're comparing, say, Western military capabilities to Chinese ones, that's fairly straightforward if you're counting missiles, though, you know, the missile gap had its own history. [00:11:30] But for domains of competition like emerging technologies, especially when there's a strong private component and so the IC can't collect directly on American capabilities, what one can do is shift from relative scales of comparison to absolute scales. [00:11:45] And that allows deep insight into the capabilities of competitors. [00:11:49] Again, if you're within some fraction of the fundamental limit, then you know the competitor is doing quite well and you can analyze yourself in these ways. [00:11:58] And so that allows this kind of absolute scales to be used for net assessment.

[00:12:03] Another example is architectures for industrial policy. [00:12:07] My colleague, Kurt Campbell, had kind of very nicely described the need for technical architectures that support resilience. [00:12:17] And you can actually think about optimal architectures for these various requirements and then support industrial policy along those technical architectures. [00:12:28] Or you can even think of these theories as levers that can shape the world in their own image, what's often called Bernstein performativity in the sociology of finance. [00:12:41] That the use of a theory can actually make the thing more like the theory. [00:12:43] Theories can be engines, not just cameras. [00:12:49] I think the very classic example of Donald McKenzie is in the pits of Chicago. [00:12:54] The differential equations and the martingales are actually enacted in In Flesh and Blood as people are trading.

[00:13:02] Good, so what I've tried to argue so far is that we want to develop these kind of theories as a way to do futurism and kind of policy making. [00:13:15] And so one way we can do this is for scaling laws like Jason was mentioning. [00:13:20] So we were able to establish fundamental limits on kind of large language models. [00:13:25] And what we were able to prove is the optimality of the Chinchilla scaling law, the actual emergence of these phase transitions, plateauing and so on. [00:13:36] And so this comes out of a very simple model which looks something like this, which is that there's this kind of hierarchical structure of skills and as you put them together, that's where capabilities come from. [00:13:49] And so this is exactly the kind of thing. [00:13:51] Now you can take these kind of plateaus and emergence things and use those for relating capabilities to resources, okay?

[00:14:01] And now coming back to the question of this capability overhang. [00:14:06] So why is there overhang? [00:14:09] Why are there surprising capabilities if we're just assembling ingredients? [00:14:14] My own foray into AI started with in the culinary domain. [00:14:17] And you can put cayenne pepper together with papaya and orange and cream and plantain chips and you make a really great dessert. [00:14:30] But, and it's somewhat predictable based on flavor chemistry and psychophysics and all kinds of things. [00:14:36] But with these larger AI models, why are there surprising emergent capabilities? [00:14:42] And secondly, why are there jagged capabilities? [00:14:46] So as rather than a smooth positive manifold like is seen in psychometrics for human intelligence. [00:14:53] Right, so these are two questions that have kind of struck me.

[00:14:54] these are two questions that have kind of struck me as worth digging into. And so we have the statistical limit theorems like I was showing here, which are already useful, but now I wanna kind of put forth an algebraic limit theorem kind of a no-go conjecture, which is that to really have general intelligence, you need what is called bioalgebraic capability.

[00:15:22] The ability to decompose tasks and ideas and then recompose them into new things. And this is what's needed for a systematic generalization rather than statistical generalization. This ability to take pieces and compose completely new things. And one can show that that implies a positive manifold of general intelligence.

[00:15:43] Secondly, that the network architecture of the human brain actually supports such bioalgebraic capabilities and the transformer architecture, the vanilla one, does not have it. So this can be kind of this other argument that we can take forth for futurism, for policymaking and for technology itself.

[00:16:05] So the NOGO theorem itself indicates a path to general intelligence from bioalgebraic architectures that have the systematic generalization and compositional creativity. From a policy perspective, we did this report with RAND Corporation starting multiple courses to AGI. And the argument was that hyperscaling is not the only path.

[00:16:25] And in fact, as a nation, we should be hedging our bets. We should be pursuing some of these other paths, in particular, the algebraic ones. So information lattice learning, something that Heize and James and I developed a couple of years ago actually has that structure. It allows decomposition going down the lattice and recomposition going back up the lattice.

[00:16:49] And it's also performant in a lot of tasks. So, vision transformers are not doing all that great at this image classification task, whereas we're at nearly the human level. So, significant jump in performance. And Heize spoke about this last week, but one can discover some of the laws of particle physics, not just predicting scattering amplitudes, but actually finding the structural laws that govern amplitudes across loop orders.

[00:17:15] So, classification, optimal semantic compression, knowledge discovery, et cetera, et cetera. Right, so this algebraic idea that comes from kind of these limit theorems seems like it's useful and a good way to think about things for our capability overhead. And I'll close with a policy proposal, which will hopefully enable both innovative capabilities and safety for wide-scale diffusion.

[00:17:45] As we've been discussing through the day, there's a desire for kind of intertwining innovation and diffusion. And then the proposal is that we should require formal verification certificates for AI systems in deployment. And this is kind of a proposal that does not limit the particular technology.

[00:18:05] There's many ways to achieve formal verification. Just like the Longitude Prize allowed a clockmaker to win rather than an astronomer. We want to leave that open for innovation. And kind of, there's many technical ways to try to do this. One is formal verification of neural networks itself. That's an open challenge to do it at scale.

[00:18:29] There's beyond LLM technologies like information lattice learning that's already transparent, white box, human controllable, traceable, and formally verifiable. Or one could completely be AI model-agnostic and construct a harness for agentic AI orchestration, which puts friction points at the interface between the agent and the world that's formally verifiable.

[00:19:01] And so this can handle any unseen model capability. So no matter how much the AI models advance and come up with capabilities, the harness will ensure formal verification. So one can do this for information lattices, as I described, and one can do this with this idea of containment verification for agentic AI that we'll be presenting soon.

[00:19:12] So just like one does engineering safety for, say, nuclear or other settings, where the fail-safes don't depend on the system that is being controlled, you don't want your agentic frame, I'm sorry, your controls to depend on the model itself, it's alignment, and so we have this containment verification idea.

[00:19:35] So it's kind of to summarize, what I'm trying to argue is that we've been drawing a lot on the past as data, and trying to do kind of futurism as kind of statistical generalization, but really what we wanna think about is systematic generalization.

[00:19:52] Generalization. Think of history as a guide, but as building blocks, right? Not as a way to predict the future mechanically,

[00:19:58] but enlarging the strategist's repertoire of patterns and analogies and dangers and possibilities, like a lot of folks in strategy think about.

[00:20:03] And this is not just true about, you know, the future and trying to think about how to develop policy for it,

[00:20:10] but for AI itself. If we can take this more algebraic view, it might provide a guide towards AGI.

[00:20:17] So that's kind of a lot of my current thinking is let's move beyond kind of a statistical description of intelligence to this algebraic one,

[00:20:25] not only for the technology, but also for how we think about the technology in society.

[00:20:31] So I think I'll stop there.