Event Replay: A Benchmark for Scientific Reasoning: Frontier AI on Unseen Science

[00:00:00] Thank you for the invitation. It's a great pleasure to be here and sharing all this knowledge with you. My name is Gabriel. I'm a PhD student at MIT in Computer Science. And today I'm gonna be talking a little bit about this joint work with Claudio Verdum and Neil Thompson.

[00:00:26] Alright. It might not be news for people in this room that AI is actually improving very fast. That's what we have been discussing here since we started the event. And together with all these improvements, we have a lot of people saying that AI is doing science. AI is the universal solver or the solution for everything. But the thing is that we don't actually have reliable evidence of that.

[00:00:55] And that's mainly because of two reasons. One is that sometimes AI is being evaluated against things that the model already saw, and also we have benchmarks that are focused only on the final answer instead of evaluating the reasoning that the model actually needed to take in order to reach the correct and final answer. So, to evaluate AI in science, we need to check not just whether AI gets the answer right, but whether it gets there in the right way.

[00:01:25] And for that, we needed an instrument. And that was what we did. And with that instrument, we asked this question: can AI independently, blindly, and reliably solve scientific questions it has never seen? Alright, so I'm going to give you an intuition of all the steps of our system and how we get to the results that I'm going to show in the next slides.

[00:02:05] So, the first thing that we do is basically go into archive, we select the science field that we are interested in, and then we randomly pick a sample of 30 plus papers. These papers must be the first version of the work, they must be released after the model's knowledge cut off, and from a top ranking institution, which is our proxy for trust, because we know, like, on archive, we have a lot of papers that can be wrong, so that was one way we found to, you know, proxy for trust.

[00:02:37] And then we have these papers sent to the first part of our system that we call the screener. The screener is, the first thing that it does is basically to drop if the paper results can't be reached by reasoning alone. So we are not interested in looking at, for example, survey papers, meta-analysis papers, or simulation-based papers. And then we drop these papers, and this part of the system is responsible and in charge of outputting this two package.

[00:03:12] The first one is the respondent, and the judge package. I'm going to get into details of all the components that we have in each of these systems, but let's keep going.

[00:03:24] Alright. We have the respondent. The respondent is the guy responsible to take the question and the material needed to actually try to solve and make this end-to-end attempt to solve the question. And as you can see here, we only have one input, and this input is the respondent package. And then it tries to solve the question and send the question to the judge, the one responsible for the entire evaluation.

[00:03:54] And as you can see, in contrast, here we have two different inputs. One is the question, the solution, and the judge package that the respondent part of the system doesn't actually have access to.

[00:04:10] All right. It is important to mention here that the respondent does the math and the reasoning, and it can use like 22 local tools we developed as verifiers. So along the way, when the model is trying to solve the question, each step, it can use basically tools that tells the model if it is correct to what it did or not, and also inconclusive.

[00:04:40] And for this case that we have inconclusive, we have like a flag that we call a gap, and these gaps and all the trace of the reasoning to get to the final answer is used by the judge to evaluate all the steps, not only the first part, the first...

[00:04:58] the first part, the first, the last solution of the problem. [00:05:05] All right, so then let's go very briefly for the respondent package. [00:05:12] All right, so here one of the things that we have is a question, obviously, and the idea behind this is that this question could be like, have asked it by an expert. Before the paper actually exists with very viewer information about the answer. [00:05:33] So to make things a little bit more concrete, let's use an example. We took this very recent paper, which the motivation was basically this modification of Einstein's gravity by adding one extra term with an unknown strength, this alpha pen, and then they asked this question, how large can this new coefficient alpha pen be? [00:06:00] All right. When we started this project, that was the thing that we were sending to our respondent. But we have some problems here, because this question by itself can be too broad, or even reusable across different domains. So while trying to actually solve this question, the model can basically get lost. [00:06:22] So then we have a second thing that we use in this respondent package, which is what we call the premises, which is basically the minimum setup that makes the question well posed. We are not providing any information for the model who is trying to solve the question that would guide or provide hints of the answer, but only the necessary, just to make the question well posed. [00:06:51] And again, using the same real example, in this case, we could have these three premises that were given together with the question. [00:07:04] All right, and then we have the judge package. And there we have two main components. One is the answer keys, which are the commitments. Any valid solution must satisfy. And also the grading context, which are the required steps to get to the final answer that was taken at a model, except the roots and tempting traps. [00:07:30] I think it's important to mention here that the idea of this project is not actually to make the model to reproduce literally the derivation that we find in the paper. We are interested in the question. And we know that in math, for example, you can get to some conclusion by evaluating two different paths, like you can have more than one path to get to the solution. So that's one thing that I wanted to mention here. [00:07:59] And again, with the same example, the question was how large can this new coefficient be? We have to be considered 100% correct the solution. We have these two commitments that the model needs to achieve. This is the second part that I mentioned, the grading context, which are basically the context enough for the judge to understand possible traps that the model could actually try to take that step because of like strong priors or any other acceptor roles that the model who is in charge of the judging can actually be aware, so then it can actually recognize it. [00:08:49] Okay, so let's talk about some numbers. So we did the experiment across 260 papers and seven science subfields, the ones that you see on the screen. And after doing that, we saw that the models, in this case, we are using GPT 5.5 on X-Hair. The model was able to fully solve about 47% of the papers in quantum physics and let's say 22% in general relativity. [00:09:27] And this is a big gap, but I think having this number, these numbers are good for us to actually understand where you could be relying more on AI and where you would be a little bit more careful. But then we also looked at this mean quality score, which is not taking into account only the ones that the model...

[00:09:56] Speaker: That the model got 100% correct. [00:10:00] But we can see that when AI misses the full solution, [00:10:05] it still recovers most of the results, which is also nice. [00:10:11] And then we moved out a little bit and saw that across models and model generations. [00:10:16] And we used the same 260 papers and focusing on O3 and GPT 5.5. [00:10:26] It was nice to see that the mean quality that I just mentioned almost doubled, actually, in 12 months. [00:10:35] And the same full rate, instead of double, it increased like five times. [00:10:44] Also, still looking at O3 and GPT 5.5, we can see that the most common outcome flipped and didn't shift, which is very interesting to see as well. [00:11:00] Because what I was personally expecting when I ran this analysis was that we would literally have a shift, like an improvement. [00:11:10] But then it actually shifted.

[00:11:14] All right, before the takeaways, I think it's important to mention that checking limits, building on results, and verifying claims are essential scientific work. [00:11:29] But they are not the new idea itself. So for scientists, reasoning work is becoming fast, cheap, and measurable. [00:11:39] And we were able to see that across the seven fields that we studied in this work, a great portion of the problems the AI was actually able to or fully solve them or recovers at a rate of like 70%. [00:12:00] So as I said previously, I think you could use these numbers to decide where to rely more on AI and where to supervise it more closely.

[00:12:11] Another thing is that numbers are moving very, very fast. [00:12:15] And now with this project, for example, we were able to see that we can track them. [00:12:22] And a full attempt to solve a question takes about 20 minutes and roughly $5 in tokens compared with hours or even days or weeks of when we are trying to do this by ourselves.

[00:12:39] And for labs, as new ideas get harder to find, discovery is getting more expensive, and AI should help speed up the scientific work around discovery. [00:12:56] By using our system, for example, if we take all the papers from the same seven categories from 2025, which is about 40,000 papers, and use it with our pipeline, this would be around the price that we, the cost of everything. [00:13:21] And when we compare that with the same, the cost that we would actually have by doing the same things by ourselves, it would be much higher. [00:13:33] And this is a lower bound. So this number can vary a lot.

[00:13:41] And not less important, the map of where science is reasonably accessible is where the strategy lives. [00:13:48] And the budget should point you. Labs already give AI assistance to individual scientists. [00:13:54] We have a lot of partnerships. OpenAI has some partnership with national labs. [00:13:59] But projects like this one turn out to be from a separate budget line, which comes from the API tokens that I've been mentioning. [00:14:12] And I think we should be considering to allocate resources accordingly together with the traditional AI assistance.

[00:14:24] And yeah, that's it. Thank you.