Event Replay: Building the Future of AI: Can Agents Replicate Science?

[00:00:00] How many people here are running long-term agents personally every day? [00:00:13] Okay. That's pretty good. That's good. I've been doing that since February. [00:00:19] And what I'm going to talk about is some of my experimental anecdotes on doing that. [00:00:26] So I've got a bunch of slides that were actually constructed by my agents to report out on this.

[00:00:32] And what I—I want to pause for a minute to talk about why I'm doing this, all right? [00:00:38] So I'm one of the leaders of Genesis. [00:00:42] And Brian and I and Earl and others here from the labs, we've been trying to work out some of the questions even that Tom was asking, like what should the investment structure look like? [00:00:52] How much should we put into people? How much should we put into tokens, machines, maybe experimental labs and so forth? [00:01:00] And data, yeah, we talked about that a little bit. Of course, things will produce data.

[00:01:08] But what we're really trying to understand is to calibrate what the platform needs to look like and what capabilities the platform needs to have. [00:01:17] And so, somewhat similar to the talk that Gabriel gave this morning, I started out with this premise of trying to get my agents to reproduce papers. [00:01:24] Now, we've gathered all the open source papers that have been published by the DOE labs. [00:01:40] DOE has this organization called OSTI. Some of you know what it is. It has this obligation to capture the papers that come out of the public research. [00:01:50] And across the 10 Office of Science labs, over the last, since 2000 to 2026, there's on the order of 250,000 papers, something like that, across the labs, and only of which about 100,000 are actually open-open that you can actually get access to.

[00:02:08] So I started out by pulling all those down. My agents did it. I didn't do it. [00:02:12] They figured out how to use resources of the lab. Pulled down a few hundred gigabytes of these papers. [00:02:19] We parsed them all. We then categorized them into computational science, theory, things that didn't require experiments, and things that require experiments. [00:02:29] They partition about, out of, say, out of the 40,000 scientists in DOE, about 10,000 are computational scientists, and about 30,000 would be viewed as experimentalists. [00:02:43] So the papers kind of reflect that distribution.

[00:02:45] So there's this corpus of papers on computation. [00:02:48] And I started having my agents randomly choose papers out of that. [00:02:52] But I did not do what Gabriel was talking about. [00:02:56] I did not filter them to not require significant computational effort.

[00:03:02] And so let me show you what this looks like, if I can make this work. [00:03:06] Okay, so this is from a couple, I started this back in March. [00:03:12] This was back when we had 67 papers. Some of the data we'll show a bit more. [00:03:15] We now have about 200 that we've done.

[00:03:19] And my AI agent, by the way, is called Ali. It happens to be running OpenClaw. [00:03:25] The orchestration environment can be GPT, it can also be the Arch Enemy, which by the way, you can't get to from the network here, if you haven't tried.

[00:03:39] So basically the mission for the agent was, here's some papers, reproduce them, show all of your work. [00:03:47] So I was not trying to extract the problem statement, I was not trying to hide information from it, exactly the opposite. [00:03:54] I'm saying, here's the paper, it's essentially the same assignment I would give to my graduate students when they're in training. [00:04:00] Here's a bunch of interesting papers. Before you're a qualifier, try to do similar work, or reproduce this work. [00:04:06] So it was totally open source, and so on.

[00:04:09] Why is this interesting? Well, part of this reproducibility is science. [00:04:14] There is no science without reproducibility, so that's the core premise. [00:04:21] Agents now are pretty good at doing end-to-end stuff.

[00:04:25] And when we're talking about end-to-end stuff this year, I just hand them the paper. [00:04:27] Now I have an environment set up where they have access to about 20 different compute resources. [00:04:33] My home lab, which is quite extensive, my resources at the University of Chicago, which are extensive, and my resources at Argonne are extensive. [00:04:43] And I've got a tail-scale network set up so my agents, with proper supervision, can access my big machines, Aurora, Polaris, they can access a bunch of DGXs, they have access to GPUs, different flavors, they have access to databases, they have access to the real network, they can run things, they can build computers.

[00:04:58] Network, they can run things, they can build code, and so on. And so, part of what I was trying to learn was how well can they do that, how well can they actually effectively operate in my environment.

[00:05:10] And the other reason is that when they are effectively working on this, we are in dialogue. Now, I want them to automate that as much as possible. When they get stuck, we have little conversations. And I'll give you an example of one of those in a little bit.

[00:05:29] So, we've done this in Waves, usually collections of papers, maybe ten at a time, which I hand the agent. Now, Ollie is able to create a lot of sub-agents, so in practice there's usually 20 or so actual agents working on things.

[00:05:47] And this collection of papers I mentioned was selected from across the labs, and it was not filtered by domain. So it covers about 20 different domains. There was a couple of Waves and covered this period.

[00:05:59] And what they basically do is they ingest the paper, they parse it, they make a plan, right? They then do setup. Setup is really interesting. They have to figure out what software they need, they have to figure out if they're going to do symbolic calculations, if they've got to find data, they go out on the network and try to pull the data down from any kind of resource, right?

[00:06:19] They setup the environment. They have to build usually custom Python environments because they've got weird tools that need specific dependencies. Sometimes they struggle with that. They have to find code. If they can't find code, we have a built into the skill is they have permission to write code, but they should try to find an open source implementation as close to the one that was used in the paper as possible, right?

[00:06:41] And often there's proprietary codes, and so they try to find an open source code. And if all that fails, they write code, okay? They then implement and execute this. Cherry Road here is my home, my house is on Cherry Road. It's a nexus in my lab, and they can push things out to the machine.

[00:07:02] They have access to the schedulers, my allocations, so they can write code that runs on 60,000 GPUs. And then they work through the paper section by section. They get the results, and they score.

[00:07:14] I have an automated scoring agent, and they get points. And how do they do? So I score them, let me see if I can show, let me go back here. Similar to what, and we haven't talked, so I didn't know how we do this, but we do something similar.

[00:07:28] So I have a coverage estimator. If they perfectly cover the paper, they get ten points, right? And we have an agreement estimator, which is how well do the agreements agree with the figures, the numerical values, and so on in the paper? And they get scored, and there's a little bit of a rubric.

[00:07:53] I didn't come up with the rubric, I just said one to ten agents came up with the rubric, right? So on average what happens? Well coverage, and I'll give you some antidotes in a second, but about 7.5? Agreement about 8, right?

[00:08:09] In some cases we get papers that get replicated extremely well, in some cases perfect, right? They're all theory papers, by the way. They're things where the model just has to do derivations or some kind of analysis, those are relatively easy.

[00:08:27] The cases where they fail often is due to access to codes they can't reproduce, or models they can't build. So here we're talking about models, often what they're doing is pulling down a dataset and having to train an ML model on that dataset and then use it to make predictions.

[00:08:46] And one of the really interesting first examples that I did, the agent had to pull down public data, build a classifier model, a classic ML kind of model, it was a neural network but it was a classifier. It then built a generative model, it generated a whole bunch of data using the classifier to bin it, and then it iterated.

[00:09:06] And this was a paper that was quite interesting, it was a brand new paper that was published like a couple of days before I handed it to the agent. And the first thing the agent realized is that on the website, the authors had not yet published the supplementary information and it wrote an email to the authors saying where the hell is your SI file.

[00:09:28] And I, you know, it was polite, but a day later they sent the tarball, right, and the agent unpacked the tarball and worked on it. We worked on that for a while and what we discovered was that the authors of that paper made an error.

[00:09:42] And I was telling Jason this over dinner, that my agent was working hard on it and could not reproduce one of the results. And eventually I said, well here, try something different. See if we can figure out what mistake they made.

[00:09:56] They made. It went off and tried about a hundred different things and came up with the exact mistake that reproduced their figures exactly. And that was kind of like one of the early examples that made me think this was a really interesting exercise.

[00:10:09] This is what the distributions look like, it's not completely unexpected, right? There's a nice distribution here, some things on the end that we're investigating in terms of is it tool failures, is it agentic handling failures, is it building codes? Like there was a paper in combustion that we were working on, it was a combustion paper on ignition. The initial handful of experiments the agents tried, they built codes but they were in the wrong flow regime and they couldn't get combustion, they couldn't get ignition. Eventually they figured that out, went off and got a different version of the code, did it, that resulted in a couple hundred hours of GPU computation to reproduce that result, and they finally nailed it.

[00:10:57] Here's the agreement. Some papers that just can nail, so again these theory papers, closed form formulas, derivation, proofs, quite good. Things that have, well this graph was rebuilt an entire algebra in a few minutes, some Poisson solvers. One of the things I started doing after the initial tranche of papers, which I don't much have in here, is I have a library of papers on PDEs, about 20,000 PDE papers, and I asked them to create a separate track just working on PDE papers. And I have another one in Bioinformatics papers, and another one in Radiation Biology, to see how the trends differ.

[00:11:51] Every top paper had three kind of aspects of it that's super important. One thing that's really frustrating is over half of the work that's done at the DOE labs is not published in the open literature. And we should somehow fix that problem. The taxpayer is paying for this, and it's not published in the open literature. That seems like a crime.

[00:12:07] The second thing is people publish papers without publishing the code, and that's what creates a huge lift for the agents to do it. Same with the data. The data's public, right? So if you have open source code, public data, and it fits on a workstation or, say, a DGX, the models can almost always reproduce the papers, right?

[00:12:31] And I think this is consistent with Neil's comment about the models papers are using are these old small models. They fit on a workstation that's relatively straightforward to do. Where it breaks is when there's proprietary stuff. Compute scale is really large. At some point I don't let the model burn like a million GPU hours because I don't have that much I can devote to this. Pay walls and where there's actually mistakes. And we have found mistakes, you can think of this as auditing papers. And some fraction of papers are actually wrong.

[00:13:09] And the AI can find that relatively cheaply, which means we might want to think about changing how we do paper reviews where this kind of replication should be built into the review process. I think we should just create a movement to do that. Early on, I didn't let the agents go crazy with creating sub-agents. Eventually, we solved that problem and it really started accelerating.

[00:13:31] And that gave me an idea about how far could we take this. And I just want to go here. So, let me just go to this last slide here.

[00:13:43] We're trying to launch a project to reproduce 1,000 papers in 10 days. It looks like Amazon is going to fund the bill for doing this, all the compute tokens, GPU cycles, and so on. And the reason we're doing this is to try to collect data, like what do these agents have to be able to do? What does a Genesis Mission platform have to provide, such that agents can do this kind of work? That's like the first order goal.

[00:14:07] The second order goal is what is what I call the expansion factor. This is all reproducing existing work. If you're doing de novo work, and not just theory, which we kind of have evidence of how expensive that is, but de novo work where you have to do open-ended simulation, what is that expansion factor? Is it a factor of 10, more effort? A factor of 30? We don't really know. We think it's somewhere in that 10 to 30 range. But getting more experiments, a lot more data is we think possible.

[00:14:42] We think that it's possible that with a 10-day concerted effort, we could probably reproduce about 70 to 85 percent of 1000 papers. If you look at the throughput of papers from the national labs.

[00:14:54] One of the things is that we're trying to estimate the resources to do this. [00:14:59] We think to do a thousand in ten days we need something like 200-300 agents. [00:15:04] It's probably more than 100 billion tokens. I burnt about 20 billion tokens doing 100-ish, so it's probably quite a bit more than that.

[00:15:18] GPU hours, core hours, and you probably sit on around 350 terabytes of data. [00:15:23] Now if we try to scale this up, one of the other reasons that we're doing this, I know I've got to get off the stage here, is we've been doing this analysis.

[00:15:29] We have 40,000 scientists, engineers, and DOE. [00:15:31] If we want to boost their productivity by a factor of two, a factor of five for example, how much compute do we have to have? [00:15:37] How many tokens do we have to have? This is an exercise Brian and Court and Earl and I have been doing.

[00:15:42] We estimate that as pushing 100 trillion kind of tokens per year, but we see a positive correlation between how many tokens the agents have and how much compute they need, based on the kind of work. [00:15:54] So we see GPU hours scaling up.

[00:15:57] This is density functional theory. This is small cases of machine learning. This is high-performance simulation. This is hard machine learning. [00:16:06] The tail of distribution that's leaning into building significant models is going to consume the cycles, and this is data, same thing. [00:16:12] HPCIM produces a huge amount of data.

[00:16:14] The agents have to be able to read and analyze that data in order to reproduce what's happening in the paper, so we're trying to build a parametric formula that would allow us to predict. [00:16:21] Like, if we want to get a 10x acceleration of DOE science, we need this many tokens, this much compute, and so on.

[00:16:28] So I'm going to stop there. Thank you. [00:16:32] Thank you. [00:16:36] Thank you. [00:16:40] Thank you.