Sign in or Join the community to continue
Event Replay: Using AI at Scale to Solve Math and Science Challenges: Discussion with an OpenAI Researcher
Posted Oct 16, 2026 | Views 244
Share
Speakers
Ven Chandrasekaran
Researcher @ OpenAI
Chris Nicholson
Member of Global Affairs Staff @ OpenAI
Chris V. Nicholson serves on OpenAI’s Global Affairs team, where he uses data and storytelling to document major AI use cases and support the company’s economic research. He co-founded the deep learning company Skymind (Y Combinator W16), which created the open-source AI framework Eclipse Deeplearning4j. He previously reported for the New York Times and Bloomberg News. Born in Montana, he now lives in the San Francisco Bay Area with his family.
+ Read More
SUMMARY
Physicist and OpenAI researcher Ven Chandrasekaran discusses how he uses Codex for theoretical physics research, using his work on the black hole information paradox as an example. In conversation with Chris Nicholson, he explains when he intervenes, how he uses skills and parallel agents, and why researchers still need to choose the questions and judge the results.
For more on Ven's work, see his paper "Smooth horizons from topology change in canonical quantum gravity": https://arxiv.org/pdf/2606.06404
+ Read More
CONTENT & TRANSCRIPT
00:00 Chris Nicholson
Morning, everyone. Welcome to the OpenAI Forum. My name is Chris Nicholson. I'm on the Global Affairs staff at OpenAI. And I'm pleased to be joined today by Ven Chandrasekaran, a physicist and researcher on OpenAI's reasoning team. Ven earned his doctorate at Berkeley and conducted research at the Institute for Advanced Study. Today, he's going to walk us through a deck on why and how he uses Codex to tackle hard scientific problems. You can submit the, you, the community can submit questions through the Q&A throughout and we'll engage with them as the conversation rolls along. Ven, it's great to have you here.
00:50 Ven Chandrasekaran
Hey, yeah. Thank you so much, Chris, for the introduction. I want to, yeah, I'm going to spend some time walking you guys through how I use Codex for scientific problems, why I use Codex for it. And, and I and what I hope to impart is, you know, why this is the right time for a new kind of scientific renaissance by using something like Codex. And I hope to convince you that there is a lot of progress to be made on tackling really, really important and hard problems that you wouldn't have really tried to tackle before because of where we are with AI. And again, the structure is such that, you know, I really would love for people to ask questions along the way in order to, you know, keep it as engaged as possible.
01:43 Ven Chandrasekaran
So, yeah, I'm going to launch right in.
01:45 Ven Chandrasekaran
So I'm going to go through some high-level points first on why we should attempt harder questions now, and like what the intuition is for what kind of questions to attempt right now, like, “harder” is vague. I'm going to try to make that more precise. And I'll walk you through an example of how I use Codex to try and answer a very important fundamental problem in quantum gravity, which is my background. Obviously, the technical details won't matter. It's all meant to be pretty high-level, and it's mostly just to give you a flavor of how I interact with the model and how I best use the model to try and solve problems that would have been extremely hard for people in my community, like the high-energy theory community, to solve before.
02:34 Ven Chandrasekaran
I'll give you examples of prompts, kinds of calculations the model does, how I stay in the loop as a human. An important part of this presentation is this isn't meant to be examples of one-shot problems, where you give the model a super-hard problem. It just goes and does it and comes back to you with an answer. This is really more about months-long research projects that otherwise would have taken decades. And how you, as a human, as a researcher, stay in the loop in order to guide the model and get the most out of it. And then I'll end by just explaining at a more infrastructure level how to best use Codex, from the level of what kind of compute to throw at it, how to make use of things like subagents, how to fork threads, just the nitty-gritty of it, in order to get the most bang for your buck.
03:26 Ven Chandrasekaran
So, let me first convince you, or try to convince you, why this is the right time to really go all in on scientific acceleration. So, you guys may or may not know, or if you've seen the news, OpenAI has had some pretty amazing breakthroughs in the field of mathematics over the last month or so, the first of which was a resolution to the Navier-Stokes Millennium Prize problem in the case of smooth forcing, showing that there's a finite time singularity. And then more recently, we had hundreds of math problems that we found solutions to, some of which had Lean verifications that constituted decades of progress in the area of mathematics, building off of the work of many, many mathematicians. I just picked one of them because it was probably for me the one that was the most mind-blowing and the most impactful, which is a proof of the quasi-Riemann hypothesis.
04:34 Ven Chandrasekaran
And the main point here is, I think this is pretty good evidence that we have some notion of scientific superintelligence. It's arguably untested waters beyond the realm of mathematics and probably theoretical physics. It could be jagged, of course, like it could be extremely good at certain areas and still some work to do in other areas. The main point is that there are clear hints. There are clear signs that we have some kind of alien intelligence that can probably do things that humans would otherwise find impossible to do. And I think OpenAI has given us some good evidence of that so far.
05:18 Ven Chandrasekaran
So the kinds of things the model is extremely good at, so far at least, and the kinds of things that I think, as scientists, we could avail of the model with, is what I would term as superhuman analytical ability. So these are just some examples from the proof of Navier-Stokes blow up that we had. And I would argue, and probably several other people might even agree, this is probably one of the most impressive, technically difficult mathematical proofs ever. It's extremely complicated, super detailed calculations with lots and lots of like subtle cancellations where if you even missed one minus sign or like a factor of two, the whole proof would fall apart. And it's like hundreds of pages of this. And you have to keep track of all of that.
06:14 Ven Chandrasekaran
And at the same time, you need to have a lot of intuition being held in your head about the physics of what you're actually doing, i.e there's some fluids involved, there's some physical properties of the fluids. You need to guess ansatzes for how to even make progress. And I think it's kind of evident why it would have been extremely hard for humans to just land at this. Humans had been trying numerical approaches for a while, What the model did was pure pen and paper. And essentially the model is superhuman at persistence. It can just keep working forever, basically. Long context reasoning I would say is largely solved when it comes to like pure reasoning tasks like physics and math, the model never gives up.
06:55 Ven Chandrasekaran
It'll never get tired. It holds in its head all of mathematics at once. So it has context that's far beyond what any individual human could have. And when you have many copies of models, they can all work kind of like as a hive mind, again in a way that humans typically have a hard time doing. So these are the kinds of things that right now the model's spectacular at, right? And that's amazing because then as a human, you can just go in and start with some idea or problem that you want to tackle and all the really difficult, subtle, and technical stuff the model can just like do for you.
07:36 Ven Chandrasekaran
Okay, so here's a more maybe quote unquote quantitative way to depict what I'm trying to explain. So back in 2020, we had GPT-3. It could do three digit arithmetic and even then sometimes it gets it wrong. Then somewhere in 2023, we had GPT-4 could do SAT math. Then AIME, then research, we had some internal research model that got gold level on the IMO. And now, deep into 2026, we have an internal model that can not only do Navier-Stokes, it can solve many other open problems in math like far beyond the human research frontier. And this is really meant to depict the exponential explosion in model intelligence. And the fact that we're above this dashed line is really the reason we should take very seriously that this is the time to make use of Codex and other AI tools as much as possible to push the boundaries of science.
08:40 Ven Chandrasekaran
Okay, so what is actually tractable now and what is not tractable yet are things that are a bit hazy, of course, but I wanna give you a sense so you know where to spend your compute and your time and where things might still be somewhat far off. So if you have a problem that you have always wanted to solve, a problem that is maybe like one of the most major open problems in your field, and for whatever reason, humans haven't really made much progress on it or very slow progress, but there's existing framework within which that problem is scoped, that's the kind of thing that the model right now is superintelligent at. So things which were tractable before without model assistance are now, I would say, kind of like a weekend Sudoku puzzle.
09:29 Ven Chandrasekaran
The newly tractable things are things where you have some framework, say quantum gravity or quantum field theory or condensed matter, physics, fluid mechanics, whatever, like some structure that we understand various aspects of, and then there are questions within those structures that are like the Holy Grail, and that's the kind of thing the model is spectacular at. You give it that problem, maybe you seed it with some ideas, vague ideas or even totally vague ideas are completely fine. Like, just shower thoughts basically, and then just let it go ham on that.
10:02 Chris Nicholson
Hey, Ven. Hey, Ven. I've got a couple live questions coming in. So you said superintelligence a couple times. Superintelligence. How do we think about that? What's different about it compared to previous terms? That's the first question. And the second is, is there some magic around math and physics that makes our models so good? Is it special training data? Something else? Those are the two questions.
10:23 Ven Chandrasekaran
Okay, great. Those are two really, really good questions. For the first one, yeah. So when I say superintelligence, I think superintelligence is a more precise term than something like general intelligence. So when I say superintelligence, I mean, the model can solve tasks that basically would take humans decades or centuries in order to solve. And the superintelligent aspect of that is it has skills that humans just don't really have. So some of the things that I mentioned are like extreme persistence in solving problems. Like it could just work forever. The ability to work in many copies of itself and delegate and act as a hive mind and kind of like, you know, turn like a thousand agents or 10,000 agents into one like emergent super brain, kind of like an ant colony.
11:21 Ven Chandrasekaran
Again, not really the way humans organize. So those are the aspects that I think of as superintelligence.
11:28 Chris Nicholson
And then, if I can just jump in, is there some unit below that, like that horsepower in kinetics, but is there some unit of like human research hours where we can really say this problem would have taken a decade?
11:40 Ven Chandrasekaran
Totally. Yeah, so for example, the quasi-Riemann hypothesis problem. I think roughly speaking, if it was a human doing it, it would have probably been like a 100 years of human research hours that the model did in a week. So, obviously, the model is spending a lot more compute than a human brain would. And so, we could maybe say one week of compute of the model is like basically a 100 years of human research effort on that problem. Yeah, I don't know if that sufficiently answered that question,
12:19 Chris Nicholson
But it's very inspiring.
12:24 Ven Chandrasekaran
And then the second question about physics and math. Yeah, that's also a great question. Physics and math are, and when I say physics, by the way, I specifically mean theoretical physics, and it'll be clear why. Theoretical physics and math are unique in that you don't need a lab, right? It's basically all pen and paper, and then every now and then there's numerics. These are things the model already has access to and it just lives in its system. And math in particular is a highly verifiable task. It's a set of logically consistent axioms, and then when you get a result, it's like programmatic to check whether it's correct or not. Theoretical physics is much more fuzzy, of course, but it's still in the realm of mathematical consistency checks and conceptual consistency checks.
13:12 Ven Chandrasekaran
So it's, one, much easier to generate data for that kind of thing, and two, much easier to check that the data is actually good. When it comes to fields that go beyond that, like experimental physics, chemistry, biology, there's still a lot of progress being made, but naturally the bottlenecks there, eventually you'll hit bottlenecks related to actual experimental work, like wet lab work, or things of that nature, and remains to be seen exactly how rapid progress will be made when we hit those bottlenecks. I think AI research is a good example of something in between, where it relies a lot on experiments, but the experiments aren't like wet lab experiments, they're just living on a computer, and so there are bottlenecks there that things like physics and math don't have.
14:03 Chris Nicholson
So if I could rephrase, it sounds like this work can be entirely digital. It's not bottlenecked by some wet lab you have to send atoms to, and two, there's verifiability, which is really a question of the harness, so the model can enter into a feedback loop. Is that right?
14:19 Ven Chandrasekaran
Yeah. That's right. And it's a feedback loop that's essentially self-consistency as opposed to probing nature in some direct way and waiting for some feedback from nature, and that's the sort of wet lab aspect of it. That being said, I think there are reasons to be hopeful and excited about things really picking up in like some order of months. If models get really good at simulating experiments. Yeah. Are there any other questions?
14:50 Chris Nicholson
There's gonna be. Why don't we just go back to your slides and we'll keep the conversation going.
14:55 Ven Chandrasekaran
Sounds great, thanks. So just to recap here, if you're asking about sort of extremely novel and profound questions within an existing framework, that's exactly where the sort of superintelligence should be thrown at. Things, and to give you a sense, and contrast it with things that I think are not yet tractable, are if fundamentally, there's a bottleneck in some existing framework that humans have developed, and the model needs to basically come up with something totally out of left field, completely new from the ground up, has to rewrite the way we think about things. I would be personally highly skeptical if, or I'm not particularly convinced that any model in existence right now is at that point yet, but if this plot is any evidence, the point is that everything is just a matter of time.
15:52 Ven Chandrasekaran
I am sure we will get there, but I don't think we're there yet. So as a very, very illustrative and maybe cliched example, Einstein's miracle year involved him taking existing frameworks and then building from them and posing new ideas and then doing some hard calculations to try and verify them, I would say his miracle year stuff would be something I would consider, if we go back in time, tractable for the current models. But if we then look at general relativity, which he developed 10 years later and took him a very long time, I would say that, if we were to ask the models, could you come up with something like that by just seeding it the basic idea of trying to make special relativity gravitational?
16:38 Ven Chandrasekaran
I don't think the models could do it because it required a complete rewrite of our entire understanding of nature from the bottom up. And I don't think that's the kind of thing models can do now. So I'm mainly harping on this point to give you a sense of where your effort and time and compute could best be spent.
17:00 Ven Chandrasekaran
So this is mostly me preaching, but if we think of modern AI as a scientific engine, then I really think, you know, so here's my experience in academia was the following. And I wanna explain why I think AI can really revolutionize academia, and probably the same goes for national labs and other approaches. A lot of academia was, you know, you solve problems because you wanna publish papers and because the really, really difficult problems are, you know, they're very difficult and they would take you years, tens of years, hundreds of years to make progress. And obviously we have to get grants, we have to get citations, we need to get tenure, there's all kinds of things that prevent you from tackling those problems head on.
17:44 Ven Chandrasekaran
So instead, we publish papers on simpler problems or we come up with toy models or we make incremental progress or sometimes we just play around with some topic because it's fun. And those are the kinds of problems where even if it would have taken you a year before, now just become like weekend projects, you know, the kind of thing you might give a grad student in their first project so that they just get used to something. It's like minuscule AI hours to solve those kinds of problems. And so we should just not think about those anymore. I mean, they might be relevant along the way towards the problems we care about but they're kind of the same as like what you would think of as like a one-step calculation or something like that.
18:26 Ven Chandrasekaran
And instead, our ambitions should significantly rise towards breakthroughs, like the really, really big questions should not be thought of as ordinary research projects. I think that's the, if there's one thing I can impart upon you, I think that it's that. So, okay.
18:44 Ven Chandrasekaran
So that's the high-level stuff and me just mostly trying to set the stage. Let me now come to how, okay, how do we actually divide labor between us, the humans, and the AI agents? Like the goal here isn't that agents just like do everything for us literally and we just sit around. It's really to maximally optimize what are the labors that we give to agents. And then what are the labors we keep for ourselves that allows us to make progress the most quickly? So, obviously there's no unique answer to this. This is just what I have found works best in my experience and I tend to use the models a lot for physics. So, as a researcher, I think the most important thing and the most high-value thing that you have right now is choosing the question.
19:35 Ven Chandrasekaran
The models don't really have a good sense of that yet. It requires a good amount of taste, it requires intuition and it requires judgment on what actually matters to humanity or to your field. So, choose the question. Setting the assumptions, I think, is also something agents are not very good at and is the kind of thing that humans are still very, very important for. This is particularly important and it's not so much a thing in math, but it is important in all other areas of science, including physics and chemistry and biology, where, sure the model could maybe solve a problem, but did that problem matter for anything? Is it actually physically relevant? What are the boundaries of the problem that I care about as far as the physical world is concerned?
20:18 Ven Chandrasekaran
You know, like, is gravity gonna be really strong? Is quantum mechanics important? When is it important? Those kinds of physical assumptions are the things that you have to prescribe to the model. It really won't be able to do a great job right now of coming up with those on its own. And then lastly, judging the evidence, which is pretty self-explanatory, especially for fields that aren't mathematics, where things can be Lean verified, you really need human judgment in order to tell whether a result is legit or not, or is relevant or not. And then on the AI agent side, they're kind of the scientific powerhouse, right? And I wanna really emphasize, when I say they're like a scientific powerhouse, I don't mean to liken them to things like Mathematica or whatever your favorite scientific computational packet is.
21:08 Ven Chandrasekaran
They're much more than that, far more than that. So if you're using it like Mathematica, I would say like, you know, the point here is you should be using it in a much deeper way. They're like, it's like a brain. It's like I've given you a brain. It's not exactly a human brain. It's some kind of alien brain, but that alien brain is extremely, extremely, extremely good at the various properties that we just discussed, like deriving and calculating things, exploring a plethora of alternative approaches and checking and revising the work in a very like sharp and tight loop. So it is really, think of it as a brain that's collaborating with you. And it's just that there's a very clean division between what the human brain is really good at and bad at and what the AI brain is really good at okay?
21:58 Ven Chandrasekaran
And so, okay. So now I'll give you an example based on a paper that I published a few months ago of a very important open problem in the field that I worked in, which is high energy theory or quantum gravity and how I was able to use Codex to find a resolution to the problem and why I wouldn't have been able to do it on my own. Okay, and I'll just maybe, are there questions or should I just keep going?
22:25 Chris Nicholson
Please keep going.
22:26 Ven Chandrasekaran
Okay. So, okay. So again, I'm not gonna get too much into technical details. I'll just explain to you what the problem is so you have some sense of context when I show you my like Codex threads and why it's important, how it was resolved, why Codex was needed. So it's a very famous problem, you might've heard of it before, it's called the Black Hole Information Paradox. And it basically puts our two best theories of physics in direct conflict with each other, which is general relativity and quantum mechanics. And what it says is essentially, black holes are super cool. They're kind of insane in many ways. One of the most deep facts about black holes is when you're falling into a black hole, it just looks like empty space to you.
23:14 Ven Chandrasekaran
If you have a big enough black hole, the tidal forces are basically negligible, and you can't even tell you're falling into a black hole. It's literally just empty space. This is called the equivalence principle. And it's essentially just the statement that a black hole is just spacetime that's warped or curved in some super complicated way. But there's no matter there. There's no object there. You just fall through and everything is nice and simple. But on the other hand, Hawking famously showed that when you have a black hole combined with quantum mechanics, that the black hole can actually radiate, there's Hawking radiation. But if you have Hawking radiation, eventually, the black hole will completely evaporate away, right? And you'll just have some gas of Hawking radiation that's collected far away.
24:04 Ven Chandrasekaran
So that means if quantum mechanics is true that all the information inside of the black hole, it can't just disappear, it can't just get destroyed, it has to be represented in some crazy complicated way in the radiation that it produced. That statement is not unique to black holes, of course, if I have like a lump of coal and I burn the coal and the coal eventually completely disappears and just turns into radiation, I should in principle be able to reconstruct everything about the coal from that radiation. The unique thing about black holes is that they are not objects that are sitting there, it's just spacetime moving at the speed of light when you fall towards it. uh, that means it's simultaneously to the outside, looks like a regular object radiating, but when you fall in, just looks like nothing, right?
24:56 Ven Chandrasekaran
And, um, at a fundamental level, this violates a very well-known property of quantum mechanics called the monogamy of information. Um, because in order for something to look like empty space, uh, it needs, um, information to be shared between the inside and outside of, uh, the horizon in a special way. But in order for information to be recovered from the radiation, it needs information to be shared in the same special way between the radiation and the stuff falling inside the black hole. You can't have both of those at the same time. It's a, it's, it's a literal contradiction. So this thing has a name, it's called the firewall paradox. Um, but what it, what it says is either general relativity is wrong at the horizon of a black hole or quantum mechanics is wrong.
25:41 Ven Chandrasekaran
Um, or there's some thing we don't understand something deep about gravity that resolves both of these and we can have our cake and eat it too. And this is a, maybe like a 10 year, 15 year old problem.
25:57 Ven Chandrasekaran
And no one has really found a great solution to it. Only suggestions and toy models. Um, and I, you know, I, this is a problem I spent all my time in my PhD and in my postdoc trying to tackle and was never able to really make progress. It was really hard, but I had developed very good intuition for what I thought the solution might be. And, uh, as soon as Codex became smart enough, I was like, all right, screw it. I'm just going to throw all my compute at this problem. I'm going to give it all the intuitions that I have and just let it go ham on it and see if it can make progress. Right. And it, it, it did.
26:28 Ven Chandrasekaran
Um, but again, it wasn't like I gave it and then it came back overnight and it solved the problem. It was like a back and forth process between me and the model. And it took like three or four months, um, to get to a solution, uh, which otherwise might have taken me literally like 20 years to get to, right. Um, like knowing me and knowing the field, it definitely would have taken me decades instead of a couple of months. Um, and the resolution to the information paradox that the model helped make precise, um, is that essentially like, uh, you know, I'm just going to say it in words and then I'll move on because, you know, the goal here isn't necessarily to go deep into the physics.
27:08 Ven Chandrasekaran
Um, but to kind of give you a sense for why it was so hard to find a solution to this without something like Codex. Um, the resolution is the inside of the black hole and the radiation that it produces that's very far away are one and the same object. Um, it's essentially saying that in gravity, there's some extremely deep type of nonlocality where even if two, two systems are super far away from each other, it could be millions of light years away from each other. And it seems like they're two independent systems. I can probe one, like just, you know, millions of light years away. I don't even know about the other one, but actually fundamentally they are one and the same object.
27:54 Ven Chandrasekaran
Um, there, there is really no way you can distinguish them. Um, and it's essentially because the black hole is so complicated and so has such high entropy that the quantum entanglement between the black hole and the radiation it produces becomes so strong that, uh, essentially they behave like a collective degree of freedom. It's just literally one object. And so what you thought were two different things are really the same thing. So this measurement of, did I get the information back? And this measurement of, am I falling through a black hole? Everything looks normal actually, are the same measurement, and I shouldn't have treated them as two different things. And once you, uh, make this clear and make this precise, uh, there is no paradox and everything gets resolved.
28:39 Ven Chandrasekaran
And the difficulty is in actually doing the calculations and formulating a modification of our understanding of gravity that actually allows for this extreme nonlocality to come about.
28:53 Ven Chandrasekaran
Um, and that involves super fancy like topology and Hamiltonian mechanics and, uh, you know, Schrödinger's equation in like a black hole setting, like all kinds of stuff that the calculations are just super hard to do. Um, just requires persistence, requires a lot of mathematical techniques across different fields that again, I would not have been able to learn about in one lifetime, um, by any means. Um, and it requires doing a lot of consistency checks, um, and having a good idea for when terms can be dropped in a, in an approximation, when things are canceling out versus not. Um, and lastly, and maybe like most importantly, and I didn't mention this before is alignment. The models are extremely aligned towards wanting to tell the truth.
29:42 Ven Chandrasekaran
If you guys had used models, maybe like a year ago, two years ago, three years ago, they would hallucinate quite a lot. But models have come a long way since then. And definitely our, like, Astra and our internal models, they're extremely well-aligned and their goal is to get to the truth. So if they think they find something that seems inconsistent, they will cast it aside and keep going until they get to what you have told them is your understanding of the truth, which actually means it cuts out a lot of directions that would have otherwise led to dead ends.
30:19 Chris Nicholson
So, this is fascinating. Let's get back to that moment when you decided to throw all your compute at this, you're betting the house. You've talked about kind of human judgment, taste, what can you help our audience kind of understand and build their own intuition for when something is likely tractable? When should they bet the house?
30:40 Ven Chandrasekaran
Yeah, that's a good question. So I would say, again, I'm speaking as a physicist here or maybe a pseudo mathematician, but I think the same thing would apply in other fields, the closer they are to fields that are in some sense like numerical computational or theoretical. If you feel that there is a question that ought to have an answer, like it's not a question like some vague, unanswerable, like philosophical question, but it's a concrete question that the scientific method ought to be able to give an answer to. But the bottleneck is we just don't know how to make progress on the actual analysis that's needed. Either the calculations are way too hard or we don't even know what kind of ingredients we need.
31:30 Ven Chandrasekaran
We don't know what is out there in the literature that could help us because it's just not our domain of expertise, but that, you know, there has to be an answer. That's exactly the point at which I would say throw all the compute.
31:44 Chris Nicholson
Okay, one more question. So you've talked about these results. We've seen a lot of amazing results. How much of this is possible with public-facing models and how much is this really possible just with special internal models?
31:58 Ven Chandrasekaran
Great, yeah, so I don't think, for the kind of thing I'm describing, you don't need internal models at all. So like you could definitely just use Astra, even 5.6 Sol. But let's just stick to Astra because it is public and it's our most powerful model. 100% you can do all of this with Astra because the whole point here is you are always there as a human in the loop. The internal models are really good for like, just give them a problem and let them go to town and they come back to you and give you an answer. But I also think that kind of workflow is limited to specific types of problems like say math, open problems in mathematics. But for something like physics or biology, it really requires months of back and forth, human in the loop, iterating on ideas, deploying them, understanding what they mean.
32:46 Ven Chandrasekaran
And for that kind of thing, I actually think something like Astra is far more than sufficient.
32:52 Chris Nicholson
Okay, so you're basically talking about this interaction, your own interventions in the problem solving. Can you talk to us a little bit about what is the nature of that relationship? What are you doing when you're intervening?
33:01 Ven Chandrasekaran
Yeah, so that's what the next few slides will be about.
33:05 Chris Nicholson
Great.
33:06 Ven Chandrasekaran
Okay. So okay.
33:12 Ven Chandrasekaran
So let me first just give you a sense of how I interact with the model and just give you a taste for what my role is. And then I'll explain the actual feedback loop that I have with the model. So what I wanna show you, again, the words here, there's a bunch of jargon here. That doesn't matter too much. What I wanna show you is how vague the prompts I give it are. I'm not coming up with some galaxy brain, super complicated scaffold. There's no scaffolding here at all, by the way. Literally I'm just opening Codex, going to my favorite model, like say Astra, and I'm just giving it what's in my head. And it's quite vague. I'm just like, oh, I'm studying the black hole information problem in this sort of simplified setting.
34:01 Ven Chandrasekaran
I'm kind of confused. I have this vague prior for what I think should be correct, but I don't really know how to make it concrete. There's words that physicists use, but what do I do to make this mathematically precise? And like, what is the concrete calculation here? So I'm just literally, it's like what I would ask a colleague, but I'm just asking it to the model cause I know the model can actually make progress. And then, you know, there was some feedback from the model and then it made it a bit more precise for me. So then I was like, oh, okay, this is pretty promising. This seems like there's something here. Let's try to make it even more precise and turn it into a calculation.
34:44 Ven Chandrasekaran
So it goes from a vague idea to an actual dynamics calculation in physics that I can do. Like solve Schrödinger's equation or do time-dependent perturbation theory in quantum mechanics. Stuff that physicists know how to do they just don't know how to do in this super more complicated, interesting, deep context of black hole information paradox. And so on and so forth. And then it gives me some calculations or responses and then I take a look at them. I pore through them and then I'm like, huh, I think you have to make this more precise. I don't really know if this is what I wanted. Are you sure you filled in the gaps here? And blah, blah, blah. Like why isn't this happening?
35:28 Ven Chandrasekaran
Why is this not working? So you can basically see that I'm interrogating the model and I'm being extremely skeptical of the nitty-gritty. And I'm poking it and trying to get it to revisit various things it did and sharpen them and make them precise. And so it's pretty clear here the role the human is playing. Because if I was like, if this was, I'm not a biologist. If there's some fancy, interesting question in biology, I gave it the problem and it gave me back some response. I have no idea what it means and I won't know how to interact with it or improve upon it. But here, as a physicist, I have the context and the expertise. So as soon as I see the output, I can make sense of what it's done and what needs to be done next and also where there might be potential flaws.
36:20 Ven Chandrasekaran
So that's where my human experience and my human intuition is super important. Otherwise, it's really hard to know if you're going in the right direction and it's hard to tell the difference between AI slop and AI progress. And maybe this is another point I want to emphasize, because there's been a lot of discussion of this externally around how these math proofs are basically AI slop and humans can't understand them. And that's all completely fair. And that's largely also because it's one of these one-shot examples where you give the model a problem, it goes off and does something and comes back. But in this workflow and the workflow I'm encouraging you guys to try out for scientific acceleration is one where the human in the loop is very important.
37:00 Ven Chandrasekaran
And it happens more as a collaboration and more progressively. So there's no point at which the model gives you back something that's like complete slop and unintelligible because it's happening in bite-sized pieces. And so that's why I can actually take a look at its output and be like, okay, these are the places that I think you need to improve. Kind of like if you're an advisor and you have a grad student who's a mathematical genius but doesn't really have a sense for what really matters or doesn't.
37:28 Ven Chandrasekaran
Okay. So more of this, except this time maybe I was a bit more mean to the model saying it's just a bunch of words, like where's the proof, blah, blah, blah. But you can see here, okay, so these are specific screenshots of like the interactions that I had with the model. But if I now take these three examples and turn it into a sort of general rule for a workflow, this is the way I would try and depict it in a very cartoonish sense. So Codex is working on some problems, some derivation. It reads my draft. The draft could be a set of notes. It could be some Overleaf document. Because you're using an agentic model, you're not just using ChatGPT.
38:08 Ven Chandrasekaran
You're using Codex. It can interact with the artifacts that you would normally interact with. So I just give it my Overleaf. And I just have it interact with the Overleaf directly the way a collaborator would. So it reads the draft. It derives some missing steps. It updates the draft. And then I look at it. And I'm like, yo, you totally messed all this up. Or, okay, that looks good. Choose the next calculation. And then I choose the next calculation for it to do. And then it just keeps repeating until I'm happy.
38:40 Ven Chandrasekaran
So in order to make this process as streamlined and as beneficial to me as possible, one thing I do, which is a really, really nice aspect of Codex that you guys might already be aware of, but if you're not, I'm going to give you this little tip, is to create skills for Codex that will essentially act as a scaffold, but a general purpose scaffold. It's not like you have to put it in your prompt each time. You just go on Codex and you hit @ and then type in the name of the skill. And Codex will just know. Or sometimes you don't even need to do that. Based on the problem you give it, it'll just know, oh, I should use this type of skill.
39:20 Ven Chandrasekaran
And the skill basically focuses the model's, like, probability distribution onto the types of reasoning strategies and skill sets and heuristics that you as a researcher in physics or biology or chemistry or whatever would really want the model to have. It's what you would do. So to give you some examples, it took me many, many months to collect enough Codex threads to make this skill from, but I have a theoretical physicist skill that I use all the time with Codex.
39:53 Ven Chandrasekaran
And the way I came up with it is, you know, I took months and months and months of interactions with Codex over various theoretical physics problems where I'm just like cursing at it and getting annoyed at it and telling it, please do this, please do that, think this way. And then I just had a Codex session go through all of those and turn it into essentially a rulebook or a set of guidelines. And in this particular case, I then rewrote the skill and I was like, look, this is how a theoretical physicist does research. You try to find a physical mechanism. You don't just go straight for some complicated calculation. You first look for symmetries, you look for scaling arguments, dimensional analysis, simplifying limits.
40:37 Ven Chandrasekaran
And typically a physicist has some idea of what they're looking for and they can make leaps of logic, you know, spherical cow type of leaps of logic to the final answer they think ought to be true. And then you try to work backwards from there to fill in the gaps and find what the argument requires using symmetries, scaling and limits. And only at the end do you fill in the super specific mathematical details. If you don't do this, the model's natural inclination is to become super myopic. It will take a problem, get into some rabbit hole where there's a specific aspect of it that it keeps trying to refine over and over and over and over. And maybe that's useful for some areas.
41:16 Ven Chandrasekaran
Definitely not useful for most areas of science, I would say. And instead, if I tell it to use this skill, it's infinitely better and it makes my life so much easier. It just starts thinking like a physicist and it can solve problems. Like it actually unblocks problem-solving abilities that, you know, if it was thinking like some super myopic mathematician, wouldn't be able to do. So this is extremely important. And if you guys don't already do this, I would suggest you give it a shot.
41:46 Ven Chandrasekaran
Relatedly, you know, when I'm writing up a paper that I want to publish, it's extremely important to convey the stuff in that paper as well as possible so humans can actually understand it. Models right now, arguably, I would say, are not good at technical writing. So at least for physics, I also have a writing skill that essentially just takes the way I like to write. I gave it all my papers, I gave it all my suggestions over many months of Codex, and it distilled that into a writing skill, and that has also been pretty helpful. But that's sort of secondary, I would say, this skill of how do you think the way I would in my field is a very valuable trick to have.
42:29 Chris Nicholson
And so you see this kind of infinite validation, that's an indication of a failure mode. If somebody's in the middle of a process, what should they be, what signs are gonna tell them I'm on the wrong path, or maybe Codex is doing this thing that I should steer it out of?
42:45 Ven Chandrasekaran
Great, yeah. So usually the signs are, you can see what the model is thinking on Codex. You can see, it gives you partial summaries along the way, and you can probably tell if it's been spending, say, more than, I don't know, 10 minutes on a particular really specific aspect of the problem, a specific calculation, or a validation of an experiment, or some plotting or fitting, and it's just keeps refining the same thing, adding more and more qualifications, adding more and more complexities, that it's definitely gone down the wrong route. My heuristic is it shouldn't take more than a couple of minutes for it to decide, like I've gotten a sense for this is probably okay, and I can move on to the next thing.
43:38 Chris Nicholson
And if you see it failing, what do you do? How do you do that?
43:41 Ven Chandrasekaran
Oh, what do I do? Yeah, so either I remind it of the skill that I've given it, so I'll just be like, @theoretical physicist, and then be like, yo, please use the skill, or I will take the thread. I will take a link for the thread. I'll start a new thread. I'll give the new Codex session the link to that thread. And I'll say, hey, this model seems to have really gotten mode collapse. It's really gotten stuck in some thinking. But take all the useful lessons it managed to get so far and, using your fresh context, pick up from there. So I call this like forking. It's like forking a thread, basically. And it's extremely useful because it essentially resets the model's myopic nature, but you take all the information it learned and dump it into a new context.
44:34 Ven Chandrasekaran
And then it really, it's kind of amazing, if you guys haven't tried this, to try it. It's like, you know, when you've worked on a problem for too long and you're like, okay, man, I really gotta stop thinking about this. I need to take a break. I need to like do something else for like a few days. And then you come back to it and all of a sudden, you're like, oh shit, like that's what I missed. And then you make progress that way. It's literally that, but on the order of seconds instead of days, right? Give it a new context.
45:04 Chris Nicholson
So forking a thread is how models touch grass basically.
45:07 Ven Chandrasekaran
It's how they touch grass, yeah, exactly.
45:10 Chris Nicholson
Okay, live question coming in. Should people still be building their own harnesses or is Codex just gonna do better on its own?
45:17 Ven Chandrasekaran
100% the latter. I would say harnessing, harness engineering or whatever you wanna call it, I would say is not something I would spend time on. As the models get better, the level of detail of a harness definitely diminishes. So at this point, I don't have any kind of harness whatsoever, except for these super generic skill. It's like theoretical physics skill.
45:42 Chris Nicholson
You've got the skills and then you tap it into tools like Lean. Is there anything else that's kind of part of that verifiability feedback loop that people should be aware of outside the model?
45:52 Ven Chandrasekaran
No, not at all. It's really just this one skill. I don't even use Lean for the physics stuff. It's really just this one skill. And the skill is mostly so I don't have to keep reminding it how it should think. And the rest the model will just do. It really has a good sense of like, and I would actually say the less you tell it, the better. And the less you prescribe, the better. The more you prescribe, the more it'll get myopic and stuck, but it has a really good sense of what it means to be a consistent problem solver. It'll try numerical checks on its own. When we were doing the Navier-Stokes stuff, we didn't even ask it to, but it would just do numerical simulations of fluids on its own to sanity check the calculations that it was doing or the proposals it was making.
46:36 Ven Chandrasekaran
It will use SymPy a lot in order to do like integrals or Taylor series or whatever you need. You don't have to ask it to do that. It'll just do that. It'll communicate to other Codex threads if it needs to. So like this is also another thing I forgot to mention is you should really be meta-Codexing. So this is something you don't need to scaffold for, and it's actually something the less you scaffold, the better, where if you have say multiple Codex threads running in parallel, attacking different aspects of a problem, you can have them talk to each other. You can just tell them, hey, there are these other Codex threads that are running on this problem. You can be as vague as you want.
47:17 Ven Chandrasekaran
It'll just figure it out. You don't have to be super specific and they will send each other messages and communicate with each other how to make progress on the problem. And they will share their ideas and they'll create essentially like an organic messaging board. So I would also say like, that's an example where the less you scaffold, the better. Just let them go to town basically.
47:41 Ven Chandrasekaran
Okay, I will skip this. It's more of just me explaining how I interact with the model. I like intervene in the argument. You know, like, I'm like questioning it over and over. And here's the thing I was mentioning just now. I often have separate threads for separate questions, but all aspects of this same problem. So one of them might be doing a derivation, like doing some complicated calculation. And then I have another one, which is trying to interpret that derivation as it's going on. It's like, does this make sense? What does it actually physically mean? Should we be trying something else? And then I might have another one, which is like coming up with figures and diagrams to like help aid the original agents along the way.
48:28 Ven Chandrasekaran
And then, as I was saying, I often fork, I choose multiple branches. So I have one conversation, but let's say it's going down a route and I'm like, maybe this is promising, but my intuition tells me there might be a different way to do it. So then I don't want to kill that first one because who knows, maybe it'll get to the right answer. So I'll just fork another thread and then change the strategy that it's taking, bias it in a different way, and then just let the two of them go to town. And I will allow them to communicate because eventually the communication could find the best of both worlds. And then I will compare the arguments and then decide what it should pursue next.
49:04 Ven Chandrasekaran
So it's like a complicated tree, except that Codex takes care of the complicated nature of the tree for you. Really all you have to do is provide judgment.
49:13 Ven Chandrasekaran
And lastly, I also have, so if you go on Codex, there's many reasoning modes, I'm sure you know that you could use. And there's one called Ultra, which consumes, you know, it consumes tokens more quickly. But so, but if you're like, say you're trying to solve a problem, you wanna throw a scientific compute at it. And you're like, okay, with that, then Ultra mode is amazing because it uses what's called sub-agents. So the main thread that you start with is your, like, parent agent or coordinating agent. And you give it a problem. Again, no scaffolds, you don't need to tell it anything. You just give it a problem. It decides on its own how to decompose that problem into sub-problems.
49:58 Ven Chandrasekaran
And it will create several sub-agent copies of itself. And it will delegate different problems, different aspects of the problem to different agents. One of them might be doing numerical checks. The other one might be doing calculations. The third one might be searching the internet for literature. And it will take results from the sub-agents, consolidate it, communicate back to multiple other sub-agents, and so on and so forth. And it keeps integrating the results in this loop. You can intervene at any time, but it's gonna do all of this for you on its own. And this is obviously amazing for problems where multiple strategies are helpful, where there's some kind of search space, like a hyperparameter search, for example, or a search over ansatzes if it's a physics problem, and takes away a lot of the minutiae that you would have to do by otherwise doing scaffolding.
50:50 Ven Chandrasekaran
And in my experience, if you go to Codex, you can also have it change for you how many sub-agents it's using. Obviously, the more sub-agents you have, the more you'll run out of a quota more quickly. But for example, the default, I think, is four agents. For some tasks, that's more than enough. There are other tasks for which I use 64 agents. So for example, for checking the Navier-Stokes results, I used a 64 sub-agent Astra Ultra session, and it's incredible. 64 agents just working in conjunction on something is quite insane. And I'm currently working on a project, a physics paper, trying to come up with a theory of everything, which it sounds ridiculous, right? Why is this one guy trying to go at this?
51:44 Ven Chandrasekaran
But it's not one guy, I have this whole fleet of agents. And I'm just using Astra Ultra for this, by the way, I'm not using any internal model, it's just Astra Ultra. Can it construct for me a theory of everything that unifies gravity with all the other forces? And it's able to make progress, which is incredible. It's making progress that humans could have never made, and this agentic structure is super, super, super helpful for that.
52:07 Chris Nicholson
What's your intuition about how many agents are appropriate for a problem?
52:12 Ven Chandrasekaran
Yeah, I think the heuristic is essentially, the more vast and dramatic the problem is, the more agents are helpful. But it does cap off, I would think of it as a logarithmic scaling, so there's definitely diminishing returns on log scale. So if you go from four to eight, I think there's a huge benefit. Eight to 16, still pretty good benefit. 16 to 64, probably still a pretty good benefit. But then 64 to 128, diminishing returns. 128 to 256, probably it's not that useful. It's very much logarithmic, and so you should think of your task also on a logarithmic scale. Like if you had 1,000 grad students, would it be actually more helpful than just having 100 for this task? There are tasks for which parallelization is not the most useful, and sometimes you just need to just keep thinking about it in sequential.
53:00 Ven Chandrasekaran
So the trick here is developing, by using Codex a lot, developing an intuition for when something needs sequential compute, which just means let it take its time, versus more and more parallel compute. And I generally think it scales with the vastness of the problem. Coming up with a grand theory of the unified forces is a very vast problem. I need many agents for that. But this black hole information paradox thing from before, four agents was enough. It's a very specific, targeted problem.
53:30 Chris Nicholson
Are there any other thoughts to keep in mind for people who are resource constrained or anxious about tokens?
53:37 Ven Chandrasekaran
Yeah, if you're resource constrained, I would say only use Ultra when you feel like there's a huge search space over parameters, optimization, or just values. And like if you're doing an experiment and there's just many knobs to tune, that's really the only time I would use Ultra if you're resource constrained. Otherwise, just using the usual max thinking mode or high thinking mode is more than enough. I would never use the non-reasoning models, of course. They just aren't gonna be useful for any of this stuff. But as far as you're using reasoning models, just use the highest reasoning mode without Ultra, so no sub agents. And that's already good for most things that are sequential thinking. Yeah. Okay, and that's basically what we just discussed now.
54:31 Ven Chandrasekaran
Okay, and then let me just end. Okay, there's just a couple more things and then I'll end. So let's say you're now like, okay, then I feel kind of convinced, like what should I, you know, how should I try this out? So let's say you're starting a research task, some really like ambitious problem. The way I would go about it is first, you know, come up with a question, solidify the question. And then just take all the files you think would help be helpful for that question and dump it into the context. And then ask it for the main derivation or analysis that would get things going. And then tell it, most importantly, tell it what would falsify the result.
55:07 Ven Chandrasekaran
Basically, you have to kind of handhold it through the scientific method. Tell it what would be, like what would count as a wrong result, so it knows what to look for. And then you just read the argument and then, you know, you're in some loop. So I step in specifically when the claim, I give it a claim and the model has changed the claim along the way, which it does, right? Like if you give a long enough session, it will start veering away from the claim. So then I step in and I say, hey, that's not, you got to step back and think about the big picture. That's not the claim I originally gave you. Or I think this derivation is missing.
55:41 Ven Chandrasekaran
Then I intervene. And then once the calculation is clear and it's checkable and it's on track, I let the agent continue. And this is the like feedback loop that I just keep going over months and months until I'm satisfied and get to a point where I feel the problem is solved. Okay.
55:57 Ven Chandrasekaran
So last slide that I want to leave you with is, you know, once this is done and you have some problems in your field that you're super excited about, like even like high Tc superconductivity, right? Like I, I think it's well within reach if you were to use Astra. But well within reach in the sense of like making significant progress that humanity has not been able to make before. So think about what becomes worth attempting in your field, which problems did you set aside because the work just seems so prohibitive that you wouldn't have been able to make progress in finite time. And think to yourself, if I now had all of this compute, I could run serious attempts in parallel.
56:37 Ven Chandrasekaran
I could use Astra Ultra or whatever, what would change? Like, would you now go back and be like, I would totally just go for it. I would go for that problem. Or do you still think it's prohibitive for some reason? And if you do that, you can probably narrow down to like some of the biggest questions in your field and recognize that actually now it's probably tractable and just go for it, right? So yeah, I'll end there.
57:02 Chris Nicholson
Amazing. Ven, thank you so much. This has been super instructive for me. I think our community's really engaged. We only covered a fraction of the questions that came in. So ideally, this will be the start of deeper conversations with a lot of the folks that are here for this event, where we can really dig deeper into what they're confronting, the questions they're working on, and the way to use our tools. So thank you so much. Hugely valuable. Hope to do this again. Thank you.
57:33 Ven Chandrasekaran
Thank you.
+ Read More
Like
Comments (0)
Popular
Watch More
Exploring the Future of Math & AI with Terence Tao and OpenAI
Posted Oct 09, 2023 | Views 28.4K
# STEM
# Higher Education
# Innovation
Event Replay: Using AI to Protect Children Online: In Conversation with Thorn
Posted Feb 11, 2026 | Views 2.2K
# AI Safety
# Ethical AI
# OpenAI Presentation